Law of Large Numbers
The law of large numbers states that as the number of trials increases, the experimental probability gets closer and closer to the theoretical probability.
Definition
The law of large numbers says that the more times you repeat an experiment, the closer your results will get to what you expected; small experiments are unpredictable, large ones are reliable. As the number of trials $n$ increases, the sample mean $\bar{x}$ converges to the population mean $\mu$ for i.i.d. random variables with finite mean, explaining why experimental probability approaches theoretical probability with more trials. Formally, the weak LLN states $\bar{x}$ converges in probability to $\mu$: for any $\epsilon > 0$, $P(|\bar{x}-\mu| > \epsilon) \to 0$ as $n \to \infty$; the strong LLN states $\bar{x}$ converges almost surely to $\mu$, $P(\lim \bar{x} = \mu) = 1$, a stronger statement holding under the weaker condition $E[|X|] < \infty$.
Example
Flip a coin $10$ times and you might get $7$ heads ($70\%$); flip $10{,}000$ times and you will almost certainly get very close to $50\%$ heads. Rolling a die repeatedly, the sample mean might be $4.2$ after $10$ rolls, $3.7$ after $100$, and almost certainly very close to $3.5$ (the theoretical mean) after $10{,}000$. The ergodic theorem generalizes this: for a stationary ergodic process, the time average converges to the ensemble average almost surely, connecting the LLN to statistical mechanics and time series analysis, where a single long time series substitutes for many independent realizations.
Key Insight
This is why casinos always win in the long run: each individual gambler is unpredictable, but with thousands of gamblers, the casino's actual results converge to the expected (profitable) outcome. The law of large numbers is not about short-term runs; if you flip heads $10$ times in a row, the next flip is still $50/50$ (the gambler's fallacy is a misunderstanding of this), since the law averages over infinitely many trials, not corrects short-term runs, and the precise rate of convergence is given by the central limit theorem (standard error $= \sigma/\sqrt{n}$).