6.48Latent failures

Systems and complexity

Most accidents are set up long before they happen, by decisions made far from the scene that lie dormant until an ordinary slip sets them off.

James Reason, The contribution of latent human failures to the breakdown of complex systems, Philosophical Transactions of the Royal Society of London B 327 (1990), where he likens them to resident pathogens in the body.

1Read
Reason split the human part in an accident into two kinds. Active failures are errors and violations with an immediate effect, made by the people at the front line: pilots, control room staff, ships' crews, signallers. Latent failures are, in his words, "decisions or actions, the damaging consequences of which may lie dormant for a long time", and they only show when they meet a local trigger such as a slip, a technical fault or an unusual condition. Their defining feature is that they were in the system well before the accident began, and they mostly come from people far from the scene in time and place: designers, managers, regulators and maintenance staff. The front-line operators, he wrote, are often "the inheritors of system defects". The prediction you can check is that any serious accident, traced back, will turn up conditions that were known, or could have been known, for years.
His metaphor was resident pathogens. Every system carries some, and the more it carries, the more likely it is that a set of them meets just the trigger needed to complete an accident. The triggers are close to impossible to foresee, but the pathogens can be found and assessed in advance, so he argued that safety effort pays more spent hunting latent failures than trying to stop every slip. His evidence included an Institute of Nuclear Power Operations (INPO) analysis of 180 significant event reports from 1983 and 1984: of 387 root causes, at least 92 percent were man-made, and only about 8 percent were started by the operators. Richard Cook's How Complex Systems Fail makes the same point for any complex system: it always runs with a changing mix of flaws latent within it. Software has its own versions: the restart nobody has done in years, the backup nobody has restored, the race condition that needs unusual timing.
The Swiss cheese model is the picture of the layers and their holes; this is the part that tells you where to dig and when, which is before anything has happened. Two cautions. Hindsight makes every latent condition look obvious, and a live system holds hundreds of them, so you cannot fix them all and should rank them by which one would turn the most likely slip into the worst outcome. And the idea is not a new place to put the blame. Moving the fault from the operator to a manager misses the point, which is that the conditions were built and tolerated by the organisation as a whole.
2Note
Example
On 6 March 1987 the ferry Herald of Free Enterprise capsized off Zeebrugge, Belgium, because she went to sea with her inner and outer bow doors open; 193 people died. The assistant bosun, whose job was to close them, had fallen asleep in his cabin. Mr Justice Sheen's inquiry found that this had happened before: in October 1983 the assistant bosun of a sister ship, the Pride of Free Enterprise, had also fallen asleep and sailed with both doors open. In June 1985 the Pride's captain asked for indicator lights on the bridge to show whether the doors were closed. Managers' replies included "Do they need an indicator to tell them whether the deck storekeeper is awake and sober? My goodness!!" and "Nice but don't we already pay someone!" The court found that the underlying faults lay higher up in the company, which "from top to bottom" was "infected with the disease of sloppiness", and that had the request been taken seriously, the disaster might well have been prevented. Reason cites this inquiry as one of two that turned investigators' attention from the front line to latent failures.
3Apply
Where it applies
In a postmortem, make a separate list of everything that was already true before the day: the backup never restored, the alert everyone mutes, the tool with no limit, the request for a safeguard that was turned down. Fix those, not only the slip. Then exercise the rare paths on purpose, by restoring from backup, restarting cold and failing over, because a path you never take is where latent failures live. For a machine learning system, an evaluation set that has not been refreshed since the data changed is a latent failure waiting for a bad release.
4Think
Think about it
Which request for a safeguard in your organisation was turned down because someone was already paid to get it right?
5Log
Applied it to something real? Log it.
6Check
Test yourselfIn Reason's 1990 account, what makes a failure latent rather than active, and why did he say safety effort should go to the latent ones?
A latent failure was created well before the accident, usually by designers, managers or maintainers, and lies dormant until a trigger meets it; an active failure is a front-line slip with an immediate effect. The triggers are nearly impossible to foresee, but latent failures can be found and assessed in advance.
7Go
Related models
- Swiss cheese modelpage 6-86
  Accidents happen when the holes in several layers of defence line up at once, so look for the holes, not the last person to touch the system.
- Pre-mortempage 6-66
  Before you start, imagine the project has already failed and ask everyone to explain why.
- The fundamental attribution errorpage 6-94
  We explain other people's behaviour by who they are and underrate the situation they were in.
- Inertiapage 6-46
  A body keeps doing what it is doing until a force acts on it, and so, by a looser analogy, do habits, defaults and organisations.
8See
Seen in an incident
- AWS DynamoDB DNS race in us-east-1page 9-3
  Two copies of the automation that manages DynamoDB's DNS (Domain Name System) records raced, the regional endpoint lost every address, and the services built on DynamoDB took most of a day to recover.
- Amazon S3 outage in us-east-1page 9-6
  One mistyped input to an established playbook command removed the servers behind the index of Amazon S3 (Simple Storage Service) in a large region, and restarting them took hours.
- Cloudbleedpage 9-9
  An HTML (Hypertext Markup Language) parser at the edge of Cloudflare's network read past the end of its buffer and wrote other customers' memory into web pages, some of which search engines cached.
- WannaCry and the accidental kill switchpage 9-11
  Ransomware spread itself between Windows machines through a flaw Microsoft had patched two months earlier, until a researcher registered the one domain every copy checked before it ran.
- South Korea's government data centre firepage 9-13
  A fire during battery relocation work at the government's main data centre in Daejeon stopped 709 public systems, and destroyed a cloud drive for civil servants that had no copy anywhere else.
9Source
Sources and further reading

6.48Latent failures

Example

Where it applies

Think about it

Related models

Seen in an incident

Sources and further reading