Probability and statistics
The law of large numbers
Averages of many independent trials settle towards the true value; averages of a few do not, and the law never 'corrects' a streak.
Jacob Bernoulli, whose proof appeared in Ars Conjectandi, printed in Basel in 1713, eight years after his death.
Toss a fair coin 10 times and the share of heads is anybody's guess: the standard deviation of the count is the square root of 10 times 0.25, about 1.58 heads, so the proportion typically lands anywhere from 35 to 65 percent. Toss it 10,000 times and the standard deviation of the count is 50, so the proportion typically sits between 49.5 and 50.5 percent. That is the law of large numbers: the average of independent draws converges on the expected value as the number of draws grows, and the spread shrinks with the square root of the count. Bernoulli proved it and saw what it was for. It is the reason frequencies observed in the world can be read as probabilities, and the reason an insurer or a casino can count on an edge that no single customer feels.
The failure mode is to expect the law to work on small samples. Tversky and Kahneman called this the belief in the law of small numbers in 1971, after asking 84 psychologists at two conferences what chance a result significant with 20 subjects had of coming out significant again with 10 more; the median answer was 0.85 and a reasonable one was about 0.48. The same intuition produces the gambler's fallacy, the sense that after a run of reds a black is 'due'. It is not. The law works by swamping, not by compensating: an early excess of five heads is never cancelled, it is divided by a larger and larger number of tosses. Croson and Sundali found the fallacy on casino video from 1998: after six or more of the same even-money outcome, 85 percent of bets went against the streak.
The law has conditions, and when they fail it fails. The draws must be independent, or nearly so: a thousand samples from one bad day are not a thousand samples. The expected value must exist and the variance must be finite, or convergence is slow enough to be useless; under fat tails the average of a million draws can still be dominated by the next one. And 'large' is relative to the noise. A metric with a standard deviation of 30 percent needs 900 observations to pin its mean to within 1 percentage point, which is more than most A/B tests, pilots and quarterly comparisons have.
Example
Rachel Croson and James Sundali coded 18 hours of security video from a single roulette table in a Reno casino, recorded over three days in July 1998, and reported in the Journal of Risk and Uncertainty in 2005: 904 spins and 24,131 bets by 139 players. The expected loss on every bet was the same, 5.26 percent. On even-money bets after a single outcome, 48 percent of bets went against it, close to the 50 percent that indifference predicts. After a streak of five, significantly more bets went against the streak, and after six or more, 85 percent did. The players also showed the 'hot hand': 80 percent left the table straight after a losing spin and only 20 percent after a win.
Where it applies
Fix the sample size before the test and do not stop when the number looks good, because a small sample is as likely to flatter as to disappoint. When comparing two model checkpoints on 50 prompts the difference is mostly noise; on 5,000 it starts to mean something, provided the prompts are independent.
Think about it
Which conclusion are you holding on the strength of a sample of five, and what would it take to get to five hundred?
Applied it to something real? Log it.
Test yourselfAfter seven reds in a row on a fair roulette wheel, is black more likely on the next spin, and what does the law of large numbers actually promise?
No; each spin is independent. The law promises that the share of reds over many spins approaches its expected value because early streaks are swamped by later spins, never because they are cancelled out.
Related models
- Regression to the meanAfter an unusually high or low result, the next one tends to be closer to average, with no cause needed beyond chance.
- Probabilistic thinkingThink in odds rather than certainties: estimate how likely each outcome is, and judge a decision by those odds rather than by the single result that happened.
- Fat tailsIn some domains the rare extreme event is so large that it dominates everything else, and the bell curve cannot see it coming.
- Expected valueWeigh each outcome by its probability and add them up; that is what a repeated bet is worth on average, and only on average.