Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.5Facebook disappears from the internet

A maintenance command took down Facebook's backbone, and its DNS (Domain Name System) servers, working as designed, took Facebook off the internet's map.

Facebook (now Meta) · 4 October 2021 · Change and rollout

  1. 1Read
    Blast radius
    Facebook, Instagram and WhatsApp worldwide, and the internal tools Facebook's own engineers would normally use to fix them.
    Duration
    About five and a half hours by Cloudflare's resolver: facebook.com stopped resolving at about 15:50 Coordinated Universal Time (UTC) and resolved again at 21:20 UTC. The apps took longer to recover fully.
  2. 2Follow

    What happened, in order

    1. ~15:40 UTC

      During routine maintenance, a command meant to assess the availability of global backbone capacity takes down every connection in the backbone. Cloudflare sees a spike of routing changes from Facebook.

    2. ~15:40 UTC

      Facebook's DNS servers can no longer reach its data centres, treat that as an unhealthy network, and withdraw their BGP (Border Gateway Protocol) advertisements, as designed.

    3. ~15:50 UTC

      facebook.com stops resolving. Resolvers worldwide retry, and their query load rises about 30 times.

    4. 15:58 UTC

      Cloudflare confirms Facebook has stopped announcing the routes to its DNS prefixes.

    5. Then

      Remote access fails, including the out-of-band network, and internal tools are broken by the loss of DNS. Engineers go to the data centres in person, then wait on the security protocols that let them work on hardened routers.

    6. ~21:00 UTC

      The backbone comes back. Cloudflare sees renewed routing activity from Facebook, peaking at 21:17 UTC.

    7. 21:20 UTC

      facebook.com resolves again. Facebook restores services in stages, because individual data centres had dropped tens of megawatts of power and switching everything back at once risked a new round of failures.

  3. 3Find

    Root cause

    Note

    Facebook's backbone is the fibre network between its data centres. Smaller edge facilities answer DNS queries, and their name servers are advertised to the rest of the internet over BGP. A command issued during routine maintenance, meant only to assess backbone capacity, took down all of the backbone's connections and disconnected the data centres from each other and from the internet. An audit tool existed to stop commands like this one; a bug in it let the command through.

    The second failure was a safety feature. Facebook's DNS servers withdraw their BGP advertisements when they cannot reach the data centres, because a server that cannot reach the data centres should not attract traffic. With the whole backbone down, every location declared itself unhealthy at once and withdrew. The DNS servers were still running, but no route to them existed, so the rest of the internet could not look up any Facebook address.

    How a backbone outage became a disappearance A maintenance command meant to assess backbone capacity passed an audit tool with a bug and took down every backbone link. The edge DNS servers, unable to reach the data centres, withdrew their BGP routes as designed, so the rest of the internet could not find Facebook's servers, and internal tools broke with DNS. With out-of-band access also down, recovery ran through engineers on site at the data centres. The change The backbone What followed Maintenance job command: assess capacity Audit tool bug: did not stop it Backbone links all of them down Internal tools broken with DNS DNS servers cannot reach data centres BGP routes withdrawn, by design The internet cannot find the servers Engineers on site out-of-band down too recovery path: physical access, through hardened facilities
    The backbone went down, and the DNS servers' health check withdrew every route at once. The way back in ran through the same network.
  4. 4Trace

    Why it got through

    There were two layers between the command and the outage, and both had a hole. The audit tool had a bug. The DNS health check did exactly what it was built to do: withdrawing is the right answer when one location loses its connection, and nothing in Facebook's account describes a different rule for every location failing the same check at once. A rule that helps when one node is sick can remove every node when the cause is shared.

    Recovery was slow because the ways in shared fate with the thing that broke. Normal access and out-of-band access were both down, and the loss of DNS broke many of the tools engineers would use to investigate. The data centres are built to be hard to get into and the routers hard to modify, even for someone standing next to them. Facebook's engineering post judged that trade, strong everyday security against slower recovery from a rare event, worth making.

  5. 5Check

    The lesson

    A safety check that acts locally can fail globally. When every node fails the same check at once, the cause is shared, and the check should hold rather than act. Keep at least one way into your systems that does not depend on them.

  6. 6Change

    What changed afterwards

    Facebook's account says its "storm" drills, which take a service, data centre or whole region offline, are what let it bring load back without further failures. It had never run one that simulated the global backbone going down, and said it would look for ways to simulate events like this.

    The stated change was to strengthen testing, drills and overall resilience. The post does not describe a specific fix to the audit tool or to the DNS withdrawal rule.

  7. 7Verify

    Where the sources differ

    • Neither Facebook post gives a duration. The common "about six hours" is not in either; Cloudflare's times put the DNS outage at about five and a half hours, and the apps took longer to recover fully.
    • Facebook's posts name no apps. Cloudflare names Facebook, WhatsApp and Instagram; Messenger, often listed in reporting, is not named in any source used here.
  8. 8Source

    Sources

  9. 9Go