Probability and statistics
Fat tails
In some domains the rare extreme event is so large that it dominates everything else, and the bell curve cannot see it coming.
Benoit Mandelbrot, who showed in 1963 that cotton price changes did not fit the Gaussian model; Nassim Taleb made it a theory of practical risk.
In a normal distribution the tails die off so fast that a move of five standard deviations is a once-in-thousands-of-years event, and the largest observation in a sample is never far from the rest. Adult heights behave like that: Galton found what he called a beautiful regularity in his measurements, and nobody is ten times taller than average. Wealth, city sizes and daily market moves do not. Their tails follow something closer to a power law, so the biggest case in a sample can exceed the sum of all the others, and the next sample can contain a bigger one. Mandelbrot made the case in 1963 with cotton prices from 1880 to 1958: a stable distribution with exponent 1.7 fitted the data, and the sample variance never settled down as more data came in.
Javier Estrada's figures for the Dow Jones Industrial Average make the gap concrete. Over 29,190 trading days from 1900 to 2006 the daily mean return was 0.02 percent and the standard deviation 1.07 percent. If returns were normal, 79 days should have fallen outside three standard deviations; 461 did. The fall of 22.61 percent on 19 October 1987 was about 21 standard deviations from the mean, which under a normal model should not happen in the life of the universe. The practical consequence is that the average and variance of past data are unreliable guides under fat tails. Taleb's point is that the law of large numbers still holds but works so slowly that the sample mean of a lifetime of data can sit far from the true one, and the next observation can move it.
Which model to use depends on the domain, not on taste. Where quantities are sums of many small independent effects, heights, measurement errors, the noise in a well-run A/B test, the normal is right and fat-tail talk is a distraction. Where quantities multiply, concentrate or cascade, incomes, outages, viral spread, losses in a portfolio bought with borrowed money, assume fat tails until shown otherwise. Under fat tails the average is a poor summary, the largest past event is a floor for the largest future one rather than a ceiling, and the useful question is not 'how likely is a crash' but 'can I survive one'. That shifts the work from forecasting the tail to limiting exposure to it.
Example
Javier Estrada of IESE Business School analysed the daily returns of the Dow Jones Industrial Average from the start of 1900 to the end of 2006, 29,190 trading days, in a 2008 paper in the Journal of Investing. The mean daily return was 0.02 percent and the standard deviation 1.07 percent. A normal distribution with those parameters predicts 79 days outside the three-standard-deviation band; the Dow had 461. The smallest of its ten best days, a gain of 9.19 percent, sits 8.6 standard deviations above the mean, an event a normal model expects about once every 250 quadrillion trading days, and ten such days occurred in 107 years.
Where it applies
Capacity, incident cost and latency all have fat tails, so plan for the p99.9 and for the biggest outage you have seen plus some, not for the mean. In AI systems, token usage per request and agent run lengths are fat-tailed too: a handful of runaway runs can dominate a month's bill, which is what per-run caps are for.
Think about it
In the system you run, what is the largest single event in your data, and would you be fine if the next one were three times as big?
Applied it to something real? Log it.
Test yourselfThe Dow's 1987 crash was about 21 standard deviations under a normal model. What does that tell you about the model rather than the event?
That the normal model is wrong for daily market moves: with 461 days beyond three sigma against 79 expected, the tails are far heavier, and the past standard deviation says little about the next extreme.
Related models
- Power laws and the 80/20 ruleIn many distributions a few items hold most of the total, and doubling the size of an event divides its frequency by a fixed factor.
- Margin of safetyLeave a wide gap between what you expect to need and what you build or pay, so that your errors and bad luck do not sink you.
- Asymmetric betsTake bets where the most you can lose is small and known and the most you can win is large, even if you expect to lose most of them.
- NonlinearityCause and effect are not proportional: a push can do nothing or everything depending on where the system sits relative to a threshold.
- The law of large numbersAverages of many independent trials settle towards the true value; averages of a few do not, and the law never 'corrects' a streak.