Skip to content
Rabin Poudyal
Page 6-1
6-1 Models

6.77Single point of failure

Systems and complexity

A part whose failure alone stops the whole system; the dangerous ones are those nobody drew on the diagram, such as the login service, the DNS provider or the tool you would use to fix things.

Reliability engineering. The US military standard for failure mode, effects and criticality analysis, MIL-STD-1629A (1980), defines a single failure point and requires a list of every one that could not be designed out.

  1. 1Read

    MIL-STD-1629A defines a single failure point as "the failure of an item which would result in failure of the system and is not compensated for by redundancy or alternative operational procedure", and its final report must list the single failure points that could not be eliminated from the design. The habit behind it is a walk through the system, part by part, asking one question: if this alone fails, does everything stop? The prediction is blunt. A system can be no more available than its least redundant dependency, however many copies it keeps of everything else.

    This is the diagnosis; redundancy and slack is the remedy, and its catch is copies that share a failure mode. Single points of failure mostly appear by sharing, because sharing is efficient. Many services come to rely on one identity service, one DNS (Domain Name System) provider, one configuration pipeline or one cloud region. Each team sees only its own dependency, and nobody sees that they all point to the same place. The worst ones sit in the recovery path: the status page, the remote access network or the deployment tool that runs on the very system that broke, so the failure also takes away the means to fix it.

    You cannot remove them all. At some level there is one company, one codebase and one planet, and every extra copy adds complexity and new ways to fail. The standard's own wording helps here: an alternative operational procedure counts as compensation, so a rehearsed manual fallback can be enough. The working version is to list them, decide which you accept, and for each accepted one make its failure quick to detect and its recovery practised, with at least one way in that does not depend on it.

  2. 2Note

    Example

    On 14 December 2020, from 03:46 to 04:33 US Pacific time, Google services that needed a Google account sign-in stopped working. The User ID Service, which handles the credentials behind OAuth (Open Authorization) tokens and cookies for Google accounts, sat underneath all of them. During a migration to a new quota system, parts of the old system left in place reported the service's usage as zero; when a grace period ran out, automated quota systems cut its quota, the leader of its account database could no longer write, and reads soon went stale. For 47 minutes every authenticated Google Workspace app was down, and BigQuery saw about three quarters of streaming requests fail. One of Google's follow-ups was to make its external communication tools more reliable during internal outages.

  3. 3Apply

    Where it applies

    Draw your dependency graph including the dull layers, DNS, identity, certificates, configuration, the status page and the paging system, and the path you would use to recover. Mark every node that has one instance or one provider, and decide for each whether you accept it. In an artificial intelligence (AI) product, one model provider behind every feature, one API (application programming interface) key or one vector database is a single point of failure in the same sense.

  4. 4Think

    Think about it

    If your identity provider, DNS provider or model provider went down for an hour, which of your recovery tools would still work?

  5. 5Log

    Applied it to something real? Log it.

  6. 6Check
    Test yourselfHow does MIL-STD-1629A define a single failure point, and what does it accept as compensation?

    The failure of an item that would cause the system to fail and is not compensated for by redundancy or an alternative operational procedure. A rehearsed manual fallback counts, as well as a second copy.

  7. 7Go

    Related models

  8. 8See

    Seen in an incident

    • Facebook disappears from the internetpage 9-5

      A maintenance command took down Facebook's backbone, and its DNS (Domain Name System) servers, working as designed, took Facebook off the internet's map.

    • Amazon S3 outage in us-east-1page 9-6

      One mistyped input to an established playbook command removed the servers behind the index of Amazon S3 (Simple Storage Service) in a large region, and restarting them took hours.

    • Dyn DNS attack and the Mirai botnetpage 9-12

      A botnet of hijacked cameras, video recorders and routers flooded Dyn's DNS (Domain Name System) service, and users could not look up the addresses of some of the internet's biggest sites.

  9. 9Source

    Sources and further reading