Probability and statistics
Regression to the mean
After an unusually high or low result, the next one tends to be closer to average, with no cause needed beyond chance.
Francis Galton, 'Regression towards mediocrity in hereditary stature' (1886), from his seed experiments and his family height records.
A team has its worst quarter on record, the manager brings in a consultant, and the next quarter is merely bad; the consultant gets the credit. That is regression to the mean with a cause invented for the return. Any measured result is partly the thing itself and partly noise, and whoever scores highest this time is on average both good and lucky. Next time the luck is redrawn, so the extreme group moves back towards the average even though nothing changed. Galton found the pattern in heights: the adult children of tall parents were tall, but on average only about two-thirds as far above the mean as their parents had been, and the children of short parents were likewise less short.
How far the extreme result moves back depends on how much of it was noise. If a score is a true value plus error, the expected follow-up for someone who was k standard deviations above the mean is r times k, where r is the correlation between the two measurements. With r of 0.5, a team two standard deviations above the mean is expected one standard deviation above next time: half the gap closes on its own. Barnett, van der Pols and Dobson's 2005 review shows the effect grows with measurement error and whenever follow-up is examined only in a subgroup picked by its first value, which is exactly how worst-performer interventions, 'before and after' pilots and remedial programmes are usually selected.
Real changes happen too, so the question is never whether the result moved but whether it moved more than regression predicts. The only clean answer is a control group selected the same way: if the untreated extreme group recovers just as much, the treatment did nothing. Repeating the baseline measurement before selecting also helps, because it averages away part of the noise. The failure mode runs both ways. Punishment after a bad result looks effective and praise after a good one looks harmful, because both are followed by a return to the mean, which is a reason to distrust what your own experience teaches you about feedback.
Example
In 1886 Francis Galton published, in the Journal of the Anthropological Institute, the heights of 928 adult children from 205 sets of parents, with the mothers' heights scaled by 1.08 to put them on the men's scale. The population average was about 68.25 inches. Parents whose average height was 71.5 inches had children whose median height was 69.9 inches, and parents averaging 64.5 inches had children with a median of 65.8 inches. The children sat between half and two-thirds of the parents' distance from the mean, and across the whole table Galton fitted the ratio at two-thirds, which is why he titled the paper 'Regression towards mediocrity in hereditary stature'.
Where it applies
Before crediting a fix for a drop in error rate, check whether the week you intervened was chosen because it was the worst. In machine learning, the model that tops a noisy benchmark by a hair is the one most likely to have been lucky, so re-evaluate the winner on fresh data before announcing it.
Think about it
Where did you last see an extreme result followed by an ordinary one, and what cause did you (or someone) invent for the return?
Applied it to something real? Log it.
Test yourselfA team's worst quarter is followed by an ordinary one after a consultant is hired. Why is that weak evidence for the consultant?
The worst quarter was partly bad luck, and luck is redrawn: an extreme result tends to be followed by one nearer the mean whether or not anything changed. Only a control group selected the same way separates the two.
Related models
- The law of large numbersAverages of many independent trials settle towards the true value; averages of a few do not, and the law never 'corrects' a streak.
- Probabilistic thinkingThink in odds rather than certainties: estimate how likely each outcome is, and judge a decision by those odds rather than by the single result that happened.
- The planning fallacyPeople predict their own projects will finish sooner and cost less than almost every comparable past project did.
- Selection biasIf the way cases got into your sample is related to what you are measuring, the sample will mislead you however large it is.