Systems and complexity
Unintended consequences
Purposeful actions produce outcomes nobody planned, because the people and systems acted on respond in ways the plan never modelled.
Robert K. Merton, "The Unanticipated Consequences of Purposive Social Action", American Sociological Review (1936).
Merton's 1936 paper gave the idea its name and its anatomy. He listed five sources of consequences nobody anticipated: ignorance, since the state of knowledge limits what can be foreseen; error, in judging the situation or the means; the imperious immediacy of interest, where concern with the immediate payoff crowds out any thought of what else the act will do; basic values, where an action is taken because a value demands it regardless of what follows; and self-defeating prediction, where a public forecast changes the behaviour it forecast. The consequences can be welcome or not. What defines them is that they were not in the plan.
The mechanism is that actions land in systems that respond. People adapt to incentives, and they react to the act of intervening as much as to its content. Trying to suppress a piece of information tells everyone it is worth seeing; rewarding a proxy for the thing you want gets you the proxy; a public warning of a crash prompts the precautions that prevent it. Merton's point is that these responses are not random noise around a plan but a predictable product of the plan meeting a system with its own feedback. The practical move is to ask, before acting, who will respond, what the action itself signals, and what you would see first if the response ran the wrong way.
Unintended is not the same as unforeseeable, and the phrase is often used afterwards to excuse consequences that a little thought would have predicted; Merton himself was careful to separate unanticipated from undesirable, since plenty of surprises are welcome. The popular cobra effect story, of a British bounty in India that led people to breed cobras, rests on Horst Siebert's 2001 book and has no primary record, so the pattern is real but that example is folklore. Nor is the idea an argument against acting, which has consequences of its own; it is an argument for small reversible steps, for naming in advance the signal that would show the response running the wrong way, and for going back to look.
Example
In May 2003 Barbra Streisand sued Kenneth Adelman, whose California Coastal Records Project had photographed the whole California coast from a helicopter, over one frame of its 12,200 images, Image 3850, which showed her Malibu house. The Los Angeles Superior Court's statement of decision, filed on 31 December 2003, records that between 14 February and 30 May 2003 the site had 14,418 downloads, of which Image 3850 accounted for six, two of them to her own lawyers. The lawsuit, not the photograph, put the picture in the news. The court threw the case out under California's law against strategic lawsuits against public participation (anti-SLAPP) and ordered her to pay $177,107.54 in fees and costs, and the episode gave its name to the Streisand effect.
Where it applies
A moderation rule, a leaderboard or a rate limit is an intervention people will respond to, so ask what it teaches them before you ship it. In AI, a reward written to stop one bad behaviour often teaches the model a different one you did not think to measure.
Think about it
What did your last fix teach the people or systems around it to do, and have you gone back to check?
Applied it to something real? Log it.
Test yourselfWhat is Merton's imperious immediacy of interest, and how does it differ from ignorance?
Being so fixed on the immediate payoff that you choose not to look at further consequences. Ignorance is not being able to foresee them; immediacy of interest is not bothering to.
Related models
- Second-order thinkingKeep asking "and then what?" until you have looked at the consequences of the consequences.
- Chesterton's fenceBefore you remove a rule, a process or a piece of code, find out why someone put it there.
- Feedback loopsA system's output feeds back into its input: balancing loops hold a level steady, reinforcing loops make change grow on itself.
- Goodhart's lawWhen a measure becomes a target it stops being a good measure, because people optimise the number instead of the thing it stood for.
- Postel's lawBe conservative in what you send and liberal in what you accept: it got the early internet to interoperate, and it lets errors harden into the standard.
Sources and further reading
- Robert K. Merton, The Unanticipated Consequences of Purposive Social Action, American Sociological Review 1(6), 1936, pp. 894-904
- Superior Court of California, Streisand v. Adelman, case SC 077 257, Statement of Decision (31 December 2003)
- California Coastal Records Project, The Streisand Lawsuit (case page with press coverage and fee award)