Psychology and biases
Incentive-caused bias
An incentive bends your own judgement towards the conclusion that pays, and you will not feel it happening.
Charlie Munger, The Psychology of Human Misjudgment, a talk at Harvard in June 1995, revised in 2005 for Poor Charlie's Almanack.
A consultant paid by the hour sincerely believes the project needs another phase. A salesperson on commission sincerely believes the expensive plan is right for you. An engineer whose bonus depends on shipping sincerely believes the known bug is minor. None of them is lying; that is the point. Munger put incentive-caused bias near the top of his list of the tendencies that cause misjudgement, and he meant a bias in the person's own mind, not a calculation. His early example was a surgeon in Lincoln, Nebraska, who removed healthy gallbladders for years. When Munger asked an older colleague whether the man had known, the answer was no: he thought the gallbladder was the source of all medical evil, and that a doctor who loved his patients could not get it out fast enough.
The mechanism is that we reason from conclusions we already prefer, and an incentive quietly sets the preference. Munger's rule for advice is to assume the bias and work around it: take the advisor's reasoning but discount the conclusion, or learn enough of the field to check the advice yourself. He also held that an incentive problem is best solved by changing the incentive rather than by exhortation. His stock case is Federal Express, which could not get its night shift to finish sorting on time until it paid by the shift instead of by the hour. That story exists only in Munger's telling, with no primary record, so take it as a parable rather than as evidence.
The idea sits next to two economics entries and should not be confused with them. The incentives model is about designing a system so that the thing you reward is the thing you want. The principal-agent problem is about an agent who knowingly acts for themselves rather than for you. Incentive-caused bias is the piece in between: the agent who does not know. That makes it harder to catch with monitoring, because the person being monitored passes a sincerity test. It also means the bias runs in you. The honest check is to ask, for any conclusion you hold with conviction, what you are paid to believe, and whether you would still hold it if the money went the other way.
Example
In 2016 Colette DeJong and colleagues at the University of California, San Francisco matched the federal Open Payments records for August to December 2013 against the Medicare Part D prescriptions of 279,669 physicians. The 63,524 industry payments tied to four promoted brand-name drugs were 95 percent meals, with a mean value under $20. Doctors who had received a single meal promoting a drug prescribed it more often than the alternatives in its class: the odds ratio was 1.18 for rosuvastatin over other statins, 1.70 for nebivolol over other beta-blockers, 1.52 for olmesartan over other angiotensin-converting enzyme (ACE) inhibitors and angiotensin-receptor blockers (ARBs), and 2.18 for desvenlafaxine over other antidepressants in its class. More meals, and meals costing over $20, went with still higher rates. The authors call it an association rather than proof of cause, but no doctor in the sample would say a sandwich changed their prescribing.
Where it applies
When you are sure a project should continue, check whether your pay, status or headcount depends on it continuing. When an AI system's evaluation is run by the team whose bonus depends on the score, assume the measurement bent before you assume the model improved.
Think about it
Name one professional opinion you hold firmly: who pays you to hold it, and would you still hold it if they stopped?
Applied it to something real? Log it.
Test yourselfHow does incentive-caused bias differ from the principal-agent problem?
The principal-agent problem is an agent who knowingly serves themselves. Incentive-caused bias is an agent whose judgement has bent towards what pays without their noticing, so they pass a sincerity test.
Related models
- Incentives drive behaviourPeople do more of what a system rewards and less of what it punishes, so to predict behaviour, look at what gets paid for.
- The principal-agent problemWhen you pay someone to act for you, their interests differ from yours and you cannot see all they do, so expect them to serve themselves at the margin.
- Goodhart's lawWhen a measure becomes a target it stops being a good measure, because people optimise the number instead of the thing it stood for.
- Hanlon's razorNever attribute to malice that which is adequately explained by stupidity, or more kindly, by error, haste or ignorance.
- Famous blunders and their causesThe great historical mistakes share a shape: the warning existed, an alternative existed, and the people in charge had reasons not to hear it.
Sources and further reading
- Charlie Munger, The Psychology of Human Misjudgment, Harvard, June 1995 (transcript by Whitney Tilson)
- Farnam Street, The Psychology of Human Misjudgment (the 2005 revised text)
- Colette DeJong et al., Pharmaceutical Industry-Sponsored Meals and Physician Prescribing Patterns for Medicare Beneficiaries, JAMA Internal Medicine 176(8) (2016)