Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.6Amazon S3 outage in us-east-1

One mistyped input to an established playbook command removed the servers behind the index of Amazon S3 (Simple Storage Service) in a large region, and restarting them took hours.

Amazon Web Services · 28 February 2017 · Change and rollout

  1. 1Read
    Blast radius
    S3 in the Northern Virginia region, us-east-1, and the AWS (Amazon Web Services) services there that rely on it, including new EC2 (Elastic Compute Cloud) instance launches and AWS Lambda.
    Duration
    4 hours 17 minutes by AWS's own times, from the command at 9:37 AM to S3 operating normally at 1:54 PM Pacific Standard Time (PST). Some dependent services took longer to clear their backlogs.
  2. 2Follow

    What happened, in order

    1. 9:37 AM PST

      While debugging a billing system that was running slowly, an authorised team member uses an established playbook to remove a small number of servers from a subsystem the billing process uses. One input is entered incorrectly, and a much larger set of servers is removed.

    2. 9:37 AM PST

      The removed servers include those of the index subsystem and the placement subsystem. Both need a full restart, and while they restart S3 cannot serve requests. Neither can the services in the region that rely on it: the S3 console, new EC2 launches, EBS (Elastic Block Store) volumes that need data from an S3 snapshot, and Lambda.

    3. 9:45 AM PST

      "Ask HN: Is S3 down?" appears on Hacker News, about eight minutes in.

    4. Until 11:37 AM PST

      AWS cannot update individual service status on its Service Health Dashboard, because the dashboard's admin console depends on S3. It posts updates through its Twitter account and a banner instead.

    5. 12:26 PM PST

      The index subsystem has enough capacity to serve GET, LIST and DELETE requests again.

    6. 1:18 PM PST

      The index subsystem is fully recovered.

    7. 1:54 PM PST

      The placement subsystem finishes recovering and PUT requests work again. S3 is operating normally.

  3. 3Find

    Root cause

    Note

    Two S3 subsystems sit under every request in a region. The index subsystem holds the metadata and location of every object, and every GET, LIST, PUT and DELETE needs it. The placement subsystem allocates storage for new objects, so PUT needs it too, and it can only work while the index works.

    The playbook command was meant to take a few servers out of a billing subsystem. With one input wrong, it took a significant share of the index and placement subsystems' capacity as well, enough that both needed a full restart. AWS had not completely restarted either subsystem in its larger regions for many years. S3 had grown enormously in that time, and the restart, with the safety checks that validate the integrity of the metadata, took much longer than anyone expected.

    One input, two subsystems, everything above them A playbook command meant to remove a few billing servers had one input mistyped and also removed servers from the index subsystem, which holds the metadata and location of every object, and the placement subsystem, which needs the index. GET, LIST and DELETE need the index; PUT needs both. EC2 launches, EBS snapshot reads, Lambda, the S3 console and the Service Health Dashboard's admin console all sat on S3 in the region. The command Servers removed S3 in us-east-1 Built on S3 Playbook command one input mistyped Billing servers the intended few Index subsystem metadata, location Placement subsystem needs the index GET, LIST, DELETE need the index PUT needs both subsystems EC2, EBS, Lambda launches, snapshots S3 console and other services Health Dashboard admin console on S3 both subsystems needed a full restart, and neither had been completely restarted in the larger regions for many years
    The command meant for a few billing servers also took the index and placement subsystems, and everything in the region sat on top of them, including the dashboard.
  4. 4Trace

    Why it got through

    The tool trusted its input. In AWS's words it "allowed too much capacity to be removed too quickly", and it had no floor: nothing stopped it removing capacity a subsystem needed to keep running. The operator was authorised and following an established playbook, so no step in the process was skipped; the process itself had no guard against a typo.

    The slow part was never rehearsed at scale. A cold restart of the index in a large region was a path the system had not taken in years, so nobody knew how long it would take until it was the critical path of an outage. And the tool that should have told customers what was happening ran on the thing that had failed.

  5. 5Check

    The lesson

    Make dangerous operations slow and bounded. A tool that removes capacity should do it gradually and refuse to go below a safe minimum, and any restart you never perform is a restart you cannot predict.

  6. 6Change

    What changed afterwards

    AWS changed the tool to remove capacity more slowly and to refuse any removal that would take a subsystem below its minimum required capacity, and began auditing its other operational tools for the same kind of check.

    It moved forward work it had already planned to split the index subsystem into smaller partitions, which AWS calls cells, so that a failure or a restart touches less at once. It also changed the Service Health Dashboard's admin console to run across multiple AWS regions.

  7. 7Verify

    Where the sources differ

    • AWS's summary gives times, not a total. Its own times make 4 hours 17 minutes to S3 operating normally, so the often repeated "about four hours" slightly undercounts, and dependent services ran later still.
  8. 8Source

    Sources

  9. 9Go