Probability and statistics
Survivorship bias
When you only see the cases that passed a filter, the pattern in them is partly a pattern of the filter.
Abraham Wald's 1943 memoranda for the Statistical Research Group at Columbia on damage to returning aircraft; the name came later, and Carhart and colleagues measured the bias in mutual fund data (2002).
You read interviews with twenty successful founders, notice that most dropped out of university, and conclude that dropping out is a sign of drive. The sample reached you through a filter, success, and the dropouts who failed never got interviewed. That is survivorship bias: a selection bias where the filter is survival. It needs no bad intent and no careless reasoning after the fact; the data were missing their failures before you opened them. Wald's wartime memoranda made the same point about aircraft. The damage you can measure is on planes that came back, so hits that look harmless are harmless, and the parts that appear untouched on survivors may be the ones whose hits brought the other planes down.
Wald's method, written in 1943 for the Statistical Research Group at Columbia and never published in his lifetime, estimates the chance that a hit on each part of an aircraft downs it from the pattern of hits on survivors, by reasoning about the hits the lost planes must have carried. The Center for Naval Analyses reprinted the eight memoranda in 1980 because the method was still in use. The same bias is measurable in finance. Carhart, Carpenter, Lynch and Musto tracked all 2,071 diversified US equity funds from 1962 to 1995 and found that funds disappeared at 3.6 percent a year, mostly by merger; a database holding only the 1,346 funds still alive at the end overstated average annual returns by around 1 percentage point in samples longer than 15 years.
The cure is to ask what the filter was and go looking for the cases it removed: funds that merged, services that were rewritten, companies that closed, experiments that were never written up. Where the failures are genuinely unrecoverable the bias can at least be bounded, which is what Wald did. Two cautions. Survivors are not worthless evidence; they show what is compatible with surviving, which is a real constraint. And the diagnosis is sometimes applied too freely. Not every success story is survivorship, only one where the sample was gathered by the outcome; if the dropouts who failed were also in the sample, the inference stands or falls on its own merits.
Example
Mark Carhart, Jennifer Carpenter, Anthony Lynch and David Musto assembled every known diversified US equity mutual fund from January 1962 to December 1995: 2,071 funds, of which 1,346 were still operating on 31 December 1995. The 725 that disappeared had performed worse than the survivors. Attrition averaged 3.6 percent of funds a year, 2.2 percent by merger and 1.0 percent by liquidation. In the Review of Financial Studies in 2002 they showed that the overstatement from looking only at survivors grows with the length of the sample, from 0.07 percent a year for one-year samples to about 1 percent a year for samples longer than 15 years, because survival depends on several years of results.
Where it applies
Any analysis of 'our long-running services', 'successful customers' or 'models that made it to production' is built on survivors, so pull the retired, churned and abandoned cases before drawing a rule. In machine learning, a training set scraped from what is still online, or built from the sessions users completed, has already been filtered by survival.
Think about it
For the dataset you trust most, what had to survive for a record to be in it, and where are the records that did not?
Applied it to something real? Log it.
Test yourselfWhy does a database of currently operating mutual funds overstate average returns, and why does the overstatement grow with the sample length?
Funds that did badly were merged or closed and are missing. Survival depends on several years of results, so the longer the window, the more poor performers it has removed: about 1 percent a year beyond 15 years in Carhart et al (2002).
Related models
- Selection biasIf the way cases got into your sample is related to what you are measuring, the sample will mislead you however large it is.
- Confirmation biasPeople look for, notice and remember evidence that fits what they already believe, and rarely test what would prove them wrong.
- The availability heuristicWe judge how likely something is by how easily examples come to mind, so vivid and recent things look more common than they are.
- Base ratesHow common something is to begin with matters as much as the evidence in front of you, and people routinely leave it out.
Sources and further reading
- Abraham Wald, A Method of Estimating Plane Vulnerability Based on Damage of Survivors (1943), reprinted by the Center for Naval Analyses (1980)
- Mark Carhart, Jennifer Carpenter, Anthony Lynch and David Musto, Mutual Fund Survivorship, Review of Financial Studies 15(5) (2002)
- Farnam Street, Survivorship Bias: The Tale of Forgotten Failures