Skip to content
Rabin Poudyal
Page 6-1
6-1 Models

6.98The Shirky principle

Economics and incentives

Institutions tend to preserve the problem they exist to solve, because their budget, staff and sense of purpose depend on it continuing.

Clay Shirky, in a talk shortly before April 2010 drawn from his then forthcoming book Cognitive Surplus; Kevin Kelly wrote the line down and named it in "The Shirky Principle" on his blog The Technium, 2 April 2010.

  1. 1Read

    Kelly quoted Shirky as saying: "Institutions will try to preserve the problem to which they are the solution." Kelly's own gloss is that a complex solution, such as a company or a whole industry, can become so devoted to its problem that it ends up keeping the problem alive, often without meaning to. His example was trade unions: built to protect workers from management, they grew complex enough to need management of their own, until unions and management became one system, each justifying the other. The prediction you can check is this: when an organisation's money and standing grow with the size of a problem, expect it to resist the fixes that would shrink the problem, and to resist hardest the cheap, simple fixes that come from outside.

    It usually happens through incentives and identity, not plotting. Budgets are set by caseload, headcount by backlog, careers by expertise in the problem as it is. Shirky's blog post of 1 April 2010, The Collapse of Complex Business Models, tells the identity half. In the mid-1990s some friends at AT&T asked him how anyone could sell web hosting for 20 dollars a month. The answer was hosting that was not very good, and a company that had spent a century hiring for reliability could not take that answer seriously; as Shirky put it, it "could not be good at the thing it was good at and good at the opposite thing at the same time." The people inside are sincere. The useful test is a plain question: who would lose money, staff or status if this problem were solved tomorrow?

    It is a tendency, not a law. Plenty of organisations do solve their problem, especially when they are paid for results rather than for effort, and some close down or move on when they win. The misuse is cynicism: reading every charity, agency or vendor as a racket. Hanlon's razor is the check, since inertia and habit explain most cases better than malice. Keep it distinct from its neighbours, too. Goodhart's law is about a measure being gamed, and incentive-caused bias is about one person's judgement bending toward their pay; the Shirky principle is about a whole organisation's stake in the problem itself. Kelly drew the hopeful reading: because we are defined by the problems we solve, progress sometimes means letting a problem go.

  2. 2Note

    Example

    The US Internal Revenue Service (IRS) Free File program began in 2003. Tax software companies, in an alliance that Intuit, the maker of TurboTax, helped organise, agreed to offer free federal filing to most taxpayers, and in exchange the IRS pledged not to build a government-run system. Eligibility was eventually set at about 70 percent of taxpayers, some 93 million households, yet use peaked at 5.1 million in 2005 and had fallen to 2.8 million by 2019. In "Inside TurboTax's 20-Year Fight to Stop Americans From Filing Their Taxes for Free" (17 October 2019), ProPublica's Justin Elliott and Paul Kiel reported that Intuit called any government move to make filing easier "encroachment", and that a 2007 presentation to its board noted that a decade of proposals for IRS tax software or a return-free system had all been stopped.

  3. 3Apply

    Where it applies

    When a team, vendor or internal platform proposes the next phase of a project, ask what its future looks like if the problem disappears; a monitoring team rewarded by incident count will not hunt for the change that ends the incidents. In machine learning work the same pull is real: a team that owns a large hand-labelled dataset has reasons to doubt the model that makes labelling unnecessary, so ask someone without that stake to judge it.

  4. 4Think

    Think about it

    Which problem in your work would cost you something if it were solved tomorrow, and how hard are you really trying to solve it?

  5. 5Log

    Applied it to something real? Log it.

  6. 6Check
    Test yourselfWhat does the Shirky principle predict about an organisation whose budget grows with the size of a problem, and what question tests for it?

    It will tend to resist fixes that would shrink the problem, usually through incentives and identity rather than intent. Ask who would lose money, staff or status if the problem were solved tomorrow.

  7. 7Go

    Related models

    • Incentives drive behaviourpage 6-42

      People do more of what a system rewards and less of what it punishes, so to predict behaviour, look at what gets paid for.

    • Incentive-caused biaspage 6-41

      An incentive bends your own judgement towards the conclusion that pays, and you will not feel it happening.

    • Goodhart's lawpage 6-36

      When a measure becomes a target it stops being a good measure, because people optimise the number instead of the thing it stood for.

    • Hanlon's razorpage 6-38

      Never attribute to malice that which is adequately explained by stupidity, or more kindly, by error, haste or ignorance.

    • The principal-agent problempage 6-96

      When you pay someone to act for you, their interests differ from yours and you cannot see all they do, so expect them to serve themselves at the margin.

  8. 8Source

    Sources and further reading