Bias (Statistics)
Statistics & ProbabilityBias in statistics is a systematic error that causes results to consistently deviate from the true population value.
Definition
Bias means that a survey or study is unfair because it consistently favors certain results. A biased sample does not truly represent the whole population.
Example
Asking only students in the math club whether they like math will give a biased result because math club members probably like math more than average students.
Key Insight
Bias is like a scale that always reads $2$ pounds too heavy. No matter how many times you weigh something, the answer is always wrong in the same direction.
Definition
Bias is a systematic error in sampling or measurement that causes results to consistently differ from the true population value. Common sources include sampling bias (non-random selection), response bias (dishonest answers), and question-wording bias.
Example
A phone survey conducted only during daytime hours will underrepresent working adults who are unavailable, producing a sample biased toward retired or unemployed respondents.
Key Insight
Bias cannot be fixed by increasing sample size. A biased sample of 1 million people is still biased. Randomization is the primary defense.
Definition
The bias of an estimator $T$ for parameter $\theta$ is defined as $\text{Bias}(T) = E[T] - \theta$. An unbiased estimator has $\text{Bias} = 0$. The mean squared error (MSE) decomposes as $\text{MSE} = \text{Variance} + \text{Bias}^2$, revealing the bias-variance tradeoff: reducing bias may increase variance and vice versa.
Example
The maximum likelihood estimator of population variance $\sigma^2$ uses $n$ in the denominator, yielding a biased estimator (underestimates by a factor of $(n-1)/n$). Replacing $n$ with $n-1$ gives the unbiased sample variance $s^2$.
Key Insight
In regularized regression (e.g., ridge regression), bias is deliberately introduced to reduce variance and improve prediction on new data, exemplifying the bias-variance tradeoff in practice.