9.1Incidents
Famous production failures, read as a practitioner reads a postmortem: what happened in order, the technical root cause, the safeguards that should have caught it and why they did not, and the one lesson worth carrying into your own systems. Every figure comes from the organisation's own account first, and where sources disagree the case study says so.
- 1Find
Change and rollout
- Cloudflare's oversized feature file2025page 9-2
A database permissions change doubled a file Cloudflare's own systems generate, and the proxy that read it crashed on a limit it had never come near.
Blast radiusMost of the core traffic through Cloudflare's network, in what Cloudflare called its worst outage since 2019: HTTP (Hypertext Transfer Protocol) 5xx errors from its CDN (content delivery network) and security services, failures in Workers KV, Access and Turnstile, and a dashboard most users could not log in to.
- AWS DynamoDB DNS race in us-east-12025page 9-3
Two copies of the automation that manages DynamoDB's DNS (Domain Name System) records raced, the regional endpoint lost every address, and the services built on DynamoDB took most of a day to recover.
Blast radiusAmazon DynamoDB in the Northern Virginia region, us-east-1, and the AWS (Amazon Web Services) services that depend on it there: new EC2 (Elastic Compute Cloud) instance launches, Network Load Balancer, Lambda, the container services, the Security Token Service, console sign-in, Amazon Connect and Redshift, along with the customers built on them.
- CrowdStrike Channel File 2912024page 9-4
A content update to a security sensor crashed Windows machines worldwide, because data the kernel driver read was shipped to everyone at once.
Blast radiusAn estimated 8.5 million Windows devices, by Microsoft's count less than one percent of all Windows machines, many of them in enterprises that run critical services. Mac and Linux hosts were not affected.
- Facebook disappears from the internet2021page 9-5
A maintenance command took down Facebook's backbone, and its DNS (Domain Name System) servers, working as designed, took Facebook off the internet's map.
Blast radiusFacebook, Instagram and WhatsApp worldwide, and the internal tools Facebook's own engineers would normally use to fix them.
- Amazon S3 outage in us-east-12017page 9-6
One mistyped input to an established playbook command removed the servers behind the index of Amazon S3 (Simple Storage Service) in a large region, and restarting them took hours.
Blast radiusS3 in the Northern Virginia region, us-east-1, and the AWS (Amazon Web Services) services there that rely on it, including new EC2 (Elastic Compute Cloud) instance launches and AWS Lambda.
Dependencies and supply chain
- The xz/liblzma backdoor2024page 9-7
An attacker spent over two years becoming a maintainer of a compression library, then shipped a backdoor aimed at SSH (Secure Shell) logins on Linux servers. One engineer noticed logins were half a second slow.
Blast radiusxz and liblzma versions 5.6.0 and 5.6.1, which reached rolling and pre-release Linux distributions: Debian unstable and testing, Fedora Rawhide and the Fedora 40 beta, openSUSE Tumbleweed and MicroOS, Kali and Arch Linux. No major stable release shipped it, and on some of these the payload never activated. Rated 10.0, critical, by the National Vulnerability Database.
- left-pad unpublished from npm2016page 9-8
A developer took all his packages off npm after a dispute over a name, and one of them, a few lines that pad a string, broke installs across JavaScript for two and a half hours.
Blast radiusEvery npm install that needed left-pad 0.0.3, directly or through a chain of dependencies. npm says it impacted many thousands of projects, and names babel and atom, which pulled it in through a package called line-numbers.
Memory safety and bounds
- Cloudbleed2017page 9-9
An HTML (Hypertext Markup Language) parser at the edge of Cloudflare's network read past the end of its buffer and wrote other customers' memory into web pages, some of which search engines cached.
Blast radiusPages served through Cloudflare for any customer, since a leak could carry data from any site behind the same server, not only sites with the affected features. Leaked memory held internal Cloudflare headers, cookies, authorisation headers and tokens; Cloudflare found it in more than 80,000 cached pages across search engines, covering about 150 customers' data. Customers' SSL (Secure Sockets Layer) private keys did not leak.
- Heartbleed2014page 9-10
A missing bounds check in OpenSSL's TLS (Transport Layer Security) heartbeat let anyone ask a server for up to 64 kilobytes of its memory, again and again, with nothing abnormal in the logs.
Blast radiusEvery server and client running OpenSSL 1.0.1 through 1.0.1f with heartbeats on. Netcraft found the heartbeat extension enabled on 17.5 percent of SSL (Secure Sockets Layer) sites with trusted certificates, around half a million certificates, as an upper bound on how many were exposed. Codenomicon, attacking itself, recovered its own private keys, user names and passwords, instant messages and emails.
Attacks at scale
- WannaCry and the accidental kill switch2017page 9-11
Ransomware spread itself between Windows machines through a flaw Microsoft had patched two months earlier, until a researcher registered the one domain every copy checked before it ran.
Blast radiusMore than 200,000 computers in at least 100 countries, by the UK National Audit Office (NAO). In England's National Health Service (NHS), at least 80 of 236 trusts were affected, 34 of them infected and locked out of devices, along with 595 GP (general practitioner) practices; 6,912 appointments were known to be cancelled, and more than 19,000 by NHS England's estimate.
- Dyn DNS attack and the Mirai botnet2016page 9-12
A botnet of hijacked cameras, video recorders and routers flooded Dyn's DNS (Domain Name System) service, and users could not look up the addresses of some of the internet's biggest sites.
Blast radiusDyn's Managed DNS customers, and the people trying to reach their sites. In the first wave mainly users whose lookups went to Dyn's points of presence on the US East Coast; the second wave was more global. Dyn names no customers; the Mirai paper, citing press reports, lists Amazon, GitHub, Netflix, PayPal, Reddit and Twitter.
Resilience and recovery
- South Korea's government data centre fire2025page 9-13
A fire during battery relocation work at the government's main data centre in Daejeon stopped 709 public systems, and destroyed a cloud drive for civil servants that had no copy anywhere else.
Blast radius709 government information systems by the final official count, 96 of them burned in one server room, including the standard document system and G-Drive, the cloud storage central-government officials used for their work files: 858 terabytes, used by about 125,000 officials across 74 agencies. Its data was declared lost.
Performance
- GTA Online's six-minute load2021page 9-14
A player profiled GTA Online's released binary from outside and found two quadratic loops in its loading code, a JSON (JavaScript Object Notation) parse and a duplicate check, that together cost most of a six-minute load.
Blast radiusPlayers of GTA Online on PC, every time they loaded into the online game. On the author's machine the online load took about six minutes, with one processor core maxed out for about four of them.
AI in production
- Cursor's support bot invents a policy2025page 9-15
A bug logged Cursor users out when they moved between machines, and an AI support agent signing as Sam told one of them it was a new one-device policy. There was no such policy.
Blast radiusUsers of the Cursor code editor caught by a session bug, and the company's standing with its own customers: the support reply was posted publicly, users said in the thread that they were cancelling, and the story reached the front page of Hacker News.
- Cloudflare's oversized feature file2025page 9-2