Law of Large Numbers
Statistics & ProbabilityThe law of large numbers states that as the number of trials increases, the experimental probability gets closer and closer to the theoretical probability.
Definition
The law of large numbers says that the more times you repeat an experiment, the closer your results will get to what you expected. Small experiments are unpredictable; large ones are reliable.
Example
Flip a coin $10$ times: you might get $7$ heads ($70\%$). Flip $10{,}000$ times: you will almost certainly get very close to $50\%$ heads. More flips, more accurate.
Key Insight
This is why casinos always win in the long run. Each individual gambler is unpredictable, but with thousands of gamblers, the casino's actual results converge to the expected (profitable) outcome.
Definition
The law of large numbers states that as the number of trials $n$ increases, the sample mean $\bar{x}$ converges to the population mean $\mu$. For i.i.d. random variables with finite mean $\mu$: as $n \to \infty$, $\bar{x} \to \mu$. This explains why experimental probability approaches theoretical probability with more trials.
Example
Roll a die repeatedly. After $10$ rolls, the sample mean might be $4.2$. After $100$ rolls, it might be $3.7$. After $10{,}000$ rolls, it will almost certainly be very close to $3.5$ (the theoretical mean). The convergence is gradual and guaranteed.
Key Insight
The law of large numbers is not about short-term runs. If you flip heads $10$ times in a row, the next flip is still $50/50$ (no "due" for tails). The law averages over infinitely many trials, not corrects short-term runs.
Definition
Weak LLN: $\bar{x}$ converges in probability to $\mu$: for any $\epsilon > 0$, $P(|\bar{x}-\mu| > \epsilon) \to 0$ as $n \to \infty$. Strong LLN: $\bar{x}$ converges almost surely to $\mu$: $P(\lim \bar{x} = \mu) = 1$, a stronger statement. Both require i.i.d. data with finite mean. The strong LLN holds under the weaker condition $E[|X|] < \infty$.
Example
The ergodic theorem is a generalization: for a stationary ergodic process, the time average converges to the ensemble average almost surely. This connects the LLN to statistical mechanics and time series analysis, where a single long time series serves as a substitute for many independent realizations.
Key Insight
The LLN does not imply that outcomes "balance out" in finite time (the gambler's fallacy). It implies that random fluctuations become negligible relative to the growing sample size. The precise rate of convergence is given by the central limit theorem (standard error $= \sigma/\sqrt{n}$).