Systems and complexity
Redundancy and slack
Spare parts and spare capacity look like waste until the day they are the only thing between a fault and a failure.
Reliability engineering; Tom DeMarco's Slack (2001) made the case for organisations.
Redundancy means more than one way to do a job that must not fail: two generators, two network paths, a replica of the database. Slack means capacity that is not fully used: headroom on a server, hours in a week that are not booked. Both buy the ability to absorb a shock or a change without breaking. DeMarco defines slack as the degree of freedom required to effect change, and his point is that an organisation run at 100% utilisation can be efficient and still unable to respond to anything. Meadows lists the size of buffers relative to their flows among her places to intervene in a system.
It holds because demand and failure are both variable. Near full utilisation a small bump in variance produces a long queue, so a buffer turns a spike into a delay instead of an outage. Redundancy only works if the copies fail independently: two of the same thing in the same basement fail together, and the arithmetic of two independent one-in-a-hundred failures (one in ten thousand) does not apply to them. In practice that means N+1 capacity, separate failure domains, headroom you do not plan to use, and some time on the calendar that belongs to no project. DeMarco's version for people is the same: a team with no slack cannot learn, cannot change, and cannot absorb the colleague who leaves.
Redundancy that shares a failure mode is theatre, and worse than none because it buys confidence. Redundant components also add complexity and can cause the failure they were meant to prevent, so each one needs a tested path to switch over and a date on which someone last tested it. Slack is the first thing cut because its cost is visible every month and its benefit shows up only on the bad day, which is the same argument investors make for a margin of safety. The honest version of the trade-off is to decide how much you are paying for insurance and against what, rather than to let the budget decide by default.
Example
At Fukushima Daiichi on 11 March 2011 the earthquake at 14:46 cut all off-site power, and the emergency diesel generators started as designed. Forty-one minutes later the first of seven tsunami waves arrived; at least one reached 14 to 15 metres, against a design basis of 5.7 metres and a site grade of 10 metres. By 15:41 units 1 to 5 had lost all alternating-current power: the generator rooms were in turbine-building basements never designed for flooding, and the seawater pumps that cooled the generators were destroyed. The tsunami disabled 12 of the site's 13 backup generators. The one survivor was an air-cooled generator on Unit 6 that sat above grade and needed no seawater, and it kept units 5 and 6 cool.
Where it applies
Replicas in one region share that region's failure, and a backup that has never been restored is not a backup. Keep headroom on the services you cannot afford to queue, and keep some unscheduled time in the team's week for the problem nobody planned.
Think about it
Which of your backups shares a failure mode with the thing it is backing up?
Applied it to something real? Log it.
Test yourselfFukushima Daiichi had 13 emergency generators and lost 12 of them. What does that say about redundancy?
Redundancy counts only if the copies fail independently. Twelve generators sat in the same floodable basements with the same seawater cooling; the survivor was the air-cooled one above grade.
Related models
- Margin of safetyLeave a wide gap between what you expect to need and what you build or pay, so that your errors and bad luck do not sink you.
- OptionalityPrefer positions that give you the right, but not the duty, to act later, so you can take the upside when it appears and walk away when it does not.
- Bottlenecks and the theory of constraintsA system's throughput is set by one constraint; improving anything else is wasted effort until that constraint is fixed.
- NonlinearityCause and effect are not proportional: a push can do nothing or everything depending on where the system sits relative to a threshold.
- The CAP theoremDuring a network partition each operation must give up either availability or consistency; the rest of the time the trade is latency against consistency.