Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.3AWS DynamoDB DNS race in us-east-1

Two copies of the automation that manages DynamoDB's DNS (Domain Name System) records raced, the regional endpoint lost every address, and the services built on DynamoDB took most of a day to recover.

Amazon Web Services · 19 October 2025 · Change and rollout

  1. 1Read
    Blast radius
    Amazon DynamoDB in the Northern Virginia region, us-east-1, and the AWS (Amazon Web Services) services that depend on it there: new EC2 (Elastic Compute Cloud) instance launches, Network Load Balancer, Lambda, the container services, the Security Token Service, console sign-in, Amazon Connect and Redshift, along with the customers built on them.
    Duration
    14 hours 32 minutes by AWS's own window, from 11:48 PM Pacific Daylight Time (PDT) on 19 October to 2:20 PM PDT on 20 October. DynamoDB itself was back by 2:40 AM; the rest was the services built on it. Some Redshift clusters took until 4:05 AM on 21 October.
  2. 2Follow

    What happened, in order

    1. 11:48 PM PDT, 19 Oct

      A delayed DNS Enactor applies an old plan to dynamodb.us-east-1.amazonaws.com just as a second Enactor's clean-up deletes that plan as obsolete. Every IP (Internet Protocol) address for the regional endpoint disappears, and requests to DynamoDB start failing.

    2. 12:38 AM PDT, 20 Oct

      Engineers identify DynamoDB's DNS state as the source of the outage.

    3. 1:15 AM PDT

      Temporary mitigations let some internal services reach DynamoDB again and repair key internal tooling.

    4. 2:25 AM PDT

      All DNS information is restored by hand. Customers recover by 2:40 AM as cached DNS records expire.

    5. 2:25 AM PDT

      EC2's DropletWorkflow Manager (DWFM) starts re-establishing the leases it lost on its physical servers. The work takes so long that leases time out again before it finishes, and it falls into what AWS calls congestive collapse. New instance launches fail.

    6. 4:14 AM PDT

      Engineers throttle incoming work and restart DWFM hosts selectively. By 5:28 AM it holds leases on every server and launches start to succeed.

    7. 5:30 AM PDT

      Network Manager is working through a backlog of network state for the new instances. Network Load Balancer health checks fail instances whose networking has not arrived yet, remove them, and trigger automatic failover between Availability Zones that takes more capacity out of service.

    8. 9:36 AM PDT

      Engineers disable automatic health check failover for Network Load Balancer. Network propagation is back to normal at 10:36 AM.

    9. 2:20 PM PDT

      With EC2 launches normal since 1:50 PM and failover re-enabled at 2:09 PM, the last services recover and AWS's event window closes.

  3. 3Find

    Root cause

    Note

    DynamoDB's DNS management has two parts. The DNS Planner watches the health and capacity of the load balancers and periodically writes a new plan for each endpoint, a set of load balancers and weights. The DNS Enactor applies plans through Amazon Route 53, and for resilience three Enactors run fully independently, one in each of three Availability Zones. Before it starts, an Enactor checks once that its plan is newer than the one already applied. When it finishes, it runs a clean-up that deletes plans much older than the one it just applied.

    On 19 October one Enactor hit unusually long delays retrying its updates. While it was stuck, the Planner produced many newer plans, and a second Enactor applied one of them to every endpoint and started its clean-up. At that moment the first Enactor, whose one-time check was now stale, overwrote the regional endpoint with its much older plan. The clean-up then deleted that plan for being many generations old, which removed every IP address from the endpoint. With the active plan gone, the system was in a state no Enactor could update, so the automation could not repair it and operators had to.

    Everything that resolved dynamodb.us-east-1.amazonaws.com failed, including AWS's own systems. EC2's DWFM keeps a lease on each physical server, which AWS calls a droplet, and checks lease state in DynamoDB, so the leases expired. When DynamoDB came back, the backlog of leases to re-establish was too large to finish before they timed out again. The launches that did succeed queued work in Network Manager, and Network Load Balancer health checks that ran before an instance's networking was ready marked working capacity as failed.

    A race between two Enactors The DNS Planner keeps producing new plans for DynamoDB's endpoints, and three DNS Enactors apply them independently. One Enactor, badly delayed, applied a much older plan to the regional endpoint just after another Enactor had applied the newest one. The second Enactor's clean-up then deleted that older plan as stale, removing every IP address for dynamodb.us-east-1.amazonaws.com. Callers could not resolve DynamoDB, EC2's DropletWorkflow Manager lost its droplet leases, and the network state backlog that followed reached Network Load Balancer health checks. DNS automation Three Enactors The endpoint What depended on it DNS Planner new plan generations Enactor, delayed applies an old plan Enactor, on time applies newest plan Plan clean-up deletes old plans DynamoDB endpoint record empty, no IPs DynamoDB callers DNS lookups fail EC2 DWFM droplet leases expire Network Manager backlog, then NLB overwrites then deleted the inconsistent state stopped every Enactor from repairing it: a manual fix. DynamoDB back by 2:40 AM PDT; the rest by 2:20 PM
    A delayed Enactor wrote an old plan over a new one, and the other Enactor's clean-up deleted it. The endpoint was left empty, and the services that resolved it failed in turn.
  4. 4Trace

    Why it got through

    The redundancy made the race possible. Three independent Enactors meant losing one did not matter, and the same independence let two of them act on one endpoint with different views of the world. The guard against applying an old plan was a check at the start of a long operation, true when it ran and false by the time it mattered, and the clean-up assumed that a plan old enough to delete could not be the live one. AWS calls the race condition latent: it had been in the system, waiting for timing like this.

    The recovery paths had not been exercised against this failure either. AWS says DWFM's situation had no established operational recovery procedure, so engineers worked one out while the outage ran. And Network Load Balancer's health checks, built to remove broken capacity, removed working capacity because the signal they read lagged behind the instances, and nothing limited how much one load balancer could take out at once.

  5. 5Check

    The lesson

    Redundant automation that writes to the same record needs coordination, not only a check before it starts. Guard destructive steps such as clean-up against the live state they are about to change, and put a rate limit on any safety mechanism that can remove capacity.

  6. 6Change

    What changed afterwards

    AWS disabled the DynamoDB DNS Planner and Enactor automation worldwide, and said it would fix the race and add protections against applying incorrect plans before turning it back on.

    Around it: a velocity control on Network Load Balancer to limit the capacity one load balancer can remove when health check failures cause failover, a new EC2 test suite that exercises DWFM's recovery workflow at scale, and throttling in EC2's data propagation systems that limits incoming work by the size of the waiting queue.

  7. 7Verify

    Where the sources differ

    • The outage is often dated 20 October. AWS's window starts at 11:48 PM PDT on 19 October, which was 06:48 UTC on the 20th, so both dates are right in their own time zone.
    • A common account says recovery was slowed by a DynamoDB backlog. In AWS's summary DynamoDB was back by 2:40 AM; the following eleven and a half hours were EC2's lease recovery, Network Manager's backlog and Network Load Balancer's health checks.
    • AWS's event end of 2:20 PM PDT is not the last recovery: its own summary has Redshift clusters restored at 4:05 AM PDT on 21 October. The summary page carries no publication date.
  8. 8Source

    Sources

  9. 9Go