Probability and statistics
Selection bias
If the way cases got into your sample is related to what you are measuring, the sample will mislead you however large it is.
A statistician's term with no single author; the 1936 Literary Digest poll is the standard cautionary case, and Peverill Squire's 1988 reanalysis showed how it went wrong.
A sample tells you about a population only if the way cases entered it is unrelated to what you want to know. Selection bias is what happens when it is related: the respondents are the people who answered, the patients are the ones who came to hospital, the logs are the sessions that finished. Survivorship bias is the special case where the filter is survival. The bias does not shrink as the sample grows, which is the counter-intuitive part. In 1936 the Literary Digest mailed 10 million ballots, received 2.4 million back and predicted that Alf Landon would beat Franklin Roosevelt with 57 percent of the vote. Roosevelt won with 62 percent. A sample that large has almost no sampling error; all of its error was selection.
Squire's 1988 reanalysis, using a 1937 Gallup survey, found two filters stacked on each other. The mailing list came from telephone directories, car registrations and club memberships, which in 1936 skewed towards the better off and towards Landon. Then only about a quarter of recipients replied, and those who disliked the sitting president were keener to say so. Squire's finding was that the list and the response were both biased and together produced the error, and that had everyone contacted replied, the poll would still have called the winner. Berkson's paradox is the subtler version: in hospital data two unrelated diseases can look negatively correlated, because admission requires at least one of them, so among patients having one makes the other rarer than in the population.
The defence is to describe the selection step in writing before looking at the results. Who could have been in this data, who was removed, and is either related to the outcome? Random sampling solves it when you control the sampling, which is why a properly drawn sample of a few thousand can beat millions of self-selected replies. When you do not control it, you can sometimes correct for known differences, as pollsters now weight by demographics, but only for differences you measured; the Digest's non-response effect became visible only through a later survey that asked people about their ballots. The uncorrectable case is when selection depends on the very thing you are measuring.
Example
For the 1936 US presidential election the Literary Digest mailed about 10 million straw ballots to names taken from telephone directories, car registrations and country club memberships. About 2.4 million came back, and the magazine forecast Landon at 57 percent. On election day Roosevelt took 62 percent of the vote. Peverill Squire's 1988 paper in Public Opinion Quarterly used a 1937 Gallup survey to show that both the list and the pattern of who replied leaned towards Landon, and that if everyone contacted had replied the poll would have called the winner despite the skewed list.
Where it applies
Every feedback form, app rating and 'how did we do' survey is answered by the people who chose to answer. In data work, training on logged conversations that users finished, or evaluating a recommender only on items it chose to show, bakes the selection into the model; log the exposures and the refusals too.
Think about it
For the data behind your last decision, who chose whether to be in it, and did that choice depend on the thing you were measuring?
Applied it to something real? Log it.
Test yourselfThe Literary Digest polled 2.4 million people in 1936 and was off by 19 points. Why did the size of the sample not help?
Size reduces sampling error only. The list (phone and car owners) and the people who chose to reply both leaned to Landon, so the error was built into who was in the sample, and more of the same sample just repeated it.
Related models
- Survivorship biasWhen you only see the cases that passed a filter, the pattern in them is partly a pattern of the filter.
- Base ratesHow common something is to begin with matters as much as the evidence in front of you, and people routinely leave it out.
- Confirmation biasPeople look for, notice and remember evidence that fits what they already believe, and rarely test what would prove them wrong.
- Correlation is not causationTwo things moving together is a fact about the data, not a verdict on what causes what, until the other explanations are ruled out.