Psychology and biases
Confirmation bias
People look for, notice and remember evidence that fits what they already believe, and rarely test what would prove them wrong.
Peter Wason, who demonstrated it with the 2-4-6 task (1960); Raymond Nickerson's 1998 review collected the evidence.
You have a theory about why the service is slow, and every graph you open seems to agree. You never ran the query that would show the theory to be wrong, and it did not occur to you to. That is confirmation bias from the inside: it does not feel like bias, it feels like being right. Nickerson's definition is seeking or interpreting evidence in ways that are partial to an existing belief or to the hypothesis in hand. It has three parts. You search where confirmation is likely. You weigh confirming evidence more heavily than disconfirming evidence of equal quality. And you remember the hits and forget the misses. Wason showed the search part in 1960 with a task so simple that intelligent adults failed it.
Part of the mechanism is that testing positive cases is often a reasonable strategy: if your hypothesis is narrow, examples that fit it are informative. The failure is in what you make of the feedback. Nickerson points out that a 'yes' to a case your hypothesis predicted is rarely strong confirmation, because many other hypotheses would have predicted it too, and people read it as strong anyway. The other part is motivational. Nickerson separates deliberate case-building, where an advocate knowingly gathers one side, from the unwitting version, where someone who sincerely wants the truth still ends up with the same lopsided file. The second is the dangerous one, because it comes with a feeling of having been fair.
The bias is well established, but its boundaries matter. Klayman and Ha argued in 1987 that Wason's subjects were using a positive test strategy that works well in most real settings, so the 2-4-6 result shows a sensible habit misfiring on an unusually broad rule rather than a general refusal to seek disconfirmation. People also do update on strong contrary evidence; what they resist is weak or ambiguous evidence. So the remedy is specific. Before concluding, write down what observation would change your mind, then go and look for that observation. If you cannot name one, you do not have a hypothesis, you have a commitment.
Example
In 1960 Peter Wason gave 29 psychology undergraduates the numbers 2, 4, 6 and told them the triple obeyed a rule he had in mind. They could propose as many further triples as they liked, hear whether each fitted, and announce the rule when sure. The rule was simply 'three numbers in increasing order of magnitude', yet most subjects tested only triples that fitted their own narrower guess, such as even numbers rising by two. Six of the 29 announced the correct rule without a wrong announcement first; 13 announced one wrong rule, nine announced two or more, and one never reached a conclusion.
Where it applies
When debugging, write the hypothesis and the query that would falsify it before opening dashboards. When evaluating a model or a vendor, decide what a failure would look like before running the demo, because a demo chosen by the seller is a positive-test strategy by design.
Think about it
What is one belief you act on at work, and what single observation this week would make you drop it?
Applied it to something real? Log it.
Test yourselfWhy did most of Wason's subjects fail to find the rule behind 2, 4, 6?
They tested only triples that fitted their own narrower hypothesis, so every test said yes and nothing could show them they were wrong.
Related models
- SteelmanningBefore you argue against a view, state it in its strongest form, well enough that the people who hold it would agree with your version.
- Strong opinions, weakly heldReach a definite conclusion early so you have something to test, then try hard to prove it wrong and let it go when the evidence turns.
- InversionWork a problem backwards by asking what would guarantee failure, then make sure you avoid it.
- The availability heuristicWe judge how likely something is by how easily examples come to mind, so vivid and recent things look more common than they are.
- Survivorship biasWhen you only see the cases that passed a filter, the pattern in them is partly a pattern of the filter.