Math intuitions
Power laws and the 80/20 rule
In many distributions a few items hold most of the total, and doubling the size of an event divides its frequency by a fixed factor.
Vilfredo Pareto's 1896 work on the distribution of income in Italy; George Zipf on word frequencies (1949); the 80/20 label came later.
A power law is a distribution in which the probability of a value falls as a fixed power of that value: double the size of a city, an earthquake or a fortune and the number of cases drops by the same factor every time, whatever size you start from. On logarithmic axes it is a straight line. Pareto found it in Italian income data in 1896, Zipf in how often words appear in English text, and Mark Newman's 2005 review lists citations, web hits, book sales, solar flares, wars and moon craters beside them. The 80/20 rule follows from the exponent: if wealth follows a power law with exponent 2.1, Newman's figure for the United States, the share held by the richest fraction P is P to the power (2.1 minus 2) over (2.1 minus 1). With P equal to 0.2 that is 0.2 to the power 0.091, which is 0.86: the richest fifth holds about 86 percent.
Power laws come from mechanisms in which size feeds growth: cities draw migrants in proportion to their size, papers get cited because they are already cited, wealth earns returns on wealth. Newman's review covers the two best-known generators, preferential attachment (the rich get richer) and critical phenomena, and both produce tails far heavier than a bell curve. The practical consequence is that averages mislead and the largest few cases dominate the total. In Newman's figures the top 20 percent of websites take about two-thirds of all hits and the largest 10 percent of US cities house about 60 percent of the population. If your incident log, your customers or your model's errors have such a tail, the biggest handful are not outliers to be trimmed; they are where most of the mass is.
Many claimed power laws are not. Aaron Clauset, Cosma Shalizi and Mark Newman tested 24 data sets that had been called power laws and found 17 consistent with one, but the power law was clearly the best fit in only one, word frequencies, and a log-normal could not be ruled out for any of the rest except one. A roughly straight line on a log-log plot over two decades of data is weak evidence; log-normal and stretched-exponential tails look the same over realistic ranges. The honest claim is usually heavy-tailed, and the honest method is a likelihood fit with a goodness-of-fit test, which is what their paper supplies. The 80/20 ratio itself is not fixed either: 70/30 and 90/10 are as common, and the stable fact is the imbalance, not the numbers.
Example
In 2009 Aaron Clauset, Cosma Shalizi and Mark Newman published Power-law distributions in empirical data in SIAM Review, applying maximum-likelihood fitting and a Kolmogorov-Smirnov goodness-of-fit test to 24 data sets that had each been conjectured to follow a power law. For 17 the power law was consistent with the data; for seven, among them earthquake sizes, forest fires, web links, wealth and HTTP connection sizes, it was firmly ruled out. Only one set, the frequency of words in English text, fit a power law so well that no alternative carried any weight, and a log-normal could not be ruled out for any set except the HTTP connections.
Where it applies
Before sizing a system for typical load, look at the tail: one customer, one query or one prompt can be most of the bill. In data engineering, sort items by contribution and see how far down the list you get before reaching half the total; in a model's error analysis, the few failure types that dominate deserve the first fix.
Think about it
In the thing you are measuring this week, what share of the total comes from the top 20 percent of items, and did you check or assume it?
Applied it to something real? Log it.
Test yourselfWhy is a straight line on a log-log plot not enough to call a distribution a power law?
Log-normal and stretched-exponential tails look just as straight over realistic ranges. Clauset, Shalizi and Newman found only one of 24 claimed power laws, word frequencies, convincing against the alternatives.
Related models
- Fat tailsIn some domains the rare extreme event is so large that it dominates everything else, and the bell curve cannot see it coming.
- Network effectsA product becomes more valuable to each user as more people use it, which makes leaders grow faster and makes leaving hard.
- NonlinearityCause and effect are not proportional: a push can do nothing or everything depending on where the system sits relative to a threshold.
- Averages hide the spreadA mean describes a group, not any member of it; without the spread you cannot tell whether anyone is near it.