9.2Cloudflare's oversized feature file
A database permissions change doubled a file Cloudflare's own systems generate, and the proxy that read it crashed on a limit it had never come near.
- 1Read
- Blast radius
- Most of the core traffic through Cloudflare's network, in what Cloudflare called its worst outage since 2019: HTTP (Hypertext Transfer Protocol) 5xx errors from its CDN (content delivery network) and security services, failures in Workers KV, Access and Turnstile, and a dashboard most users could not log in to.
- Duration
- About three hours of core impact by Cloudflare's own times, from 11:20 Coordinated Universal Time (UTC) to traffic largely flowing as normal by 14:30. Every system was working again at 17:06 UTC, a little under six hours after the start.
- 2Follow
What happened, in order
- 11:05 UTC
A database access control change is deployed to a ClickHouse cluster. It makes users' existing implicit access to the underlying tables explicit, part of work to run distributed queries under each user's own account instead of a shared system account. It rolls out across the cluster gradually.
- 11:20 UTC
Cloudflare's network begins failing to deliver core traffic. The errors come and go: the Bot Management feature file is rebuilt every five minutes, and each run produces a good file or a bad one depending on which part of the cluster the query lands on.
- 11:31 UTC
The first automated test detects the problem. Manual investigation starts at 11:32 and an incident call opens at 11:35.
- 11:32 to 13:05 UTC
The team chases the first symptom, degraded Workers KV, with traffic manipulation and account limits. The network keeps recovering for short spells, and Cloudflare's status page, hosted entirely off Cloudflare, goes down by coincidence. Some of the team suspect a very large DDoS (distributed denial of service) attack.
- 13:05 UTC
Workers KV and Access are switched to bypass the failing proxy, falling back to an earlier version, and their impact drops.
- 14:24 UTC
Cloudflare stops creating and propagating new feature files, and a test of a known good file completes.
- 14:30 UTC
A correct feature file is deployed globally and core traffic is largely flowing as normal. Downstream services start to recover.
- 17:06 UTC
All services are resolved and the impact ends. The dashboard had a second dip from 14:40 to 15:30 as a backlog of logins caught up.
- 11:05 UTC
- 3Find
Root cause
NoteCloudflare's Bot Management scores every request with a machine learning model. The model reads a feature file, the list of traits it uses to judge whether a request is automated, which a query against a ClickHouse database cluster rebuilds every five minutes and publishes to the whole network. The query read column names from ClickHouse's system.columns table for one table, with no filter on the database name. Before 11:05 users only saw the tables in the default database. After the change they could also see the underlying tables in a database called r0, where each shard stores the data, so the query returned every column twice and the file more than doubled.
The proxy preallocates memory for features, with a limit of 200, well above the roughly 60 in use. The bad file had more than 200. In FL2, Cloudflare's newer proxy written in Rust, the code that loads the features called unwrap() on the error that limit returned, and the worker thread panicked, so requests failed with HTTP 5xx errors. The older proxy, FL, raised no error but gave all traffic a bot score of zero, so customers with rules that block bots saw false positives.
The flapping came from two things at once. The permissions change was reaching the cluster node by node, so a run of the query on an updated node made a bad file and a run elsewhere made a good one, and each new file went out to every machine within minutes. Once every node had the change, every file was bad and the network settled into failing.
A permissions change made a metadata query count every column twice. The file built from it crossed the proxy's limit of 200, and because the change reached the cluster node by node, each five-minute rebuild could go either way. - 4Trace
Why it got through
The feature file came from Cloudflare's own systems, so the proxy treated it as trusted. Nothing between the query and the proxy checked the file's size or shape before it reached every machine. The limit itself was sound, a way to preallocate memory, but crossing it ended the process instead of rejecting the file and carrying on with the last good one.
The change that set it off was to permissions, not to the query or the file, and it was correct on its own terms. The fault lay in an assumption somewhere else: a query written when it could only ever see one database. Then the symptoms pointed away from the cause. An internal error does not usually recover by itself, but this one did every few minutes; Workers KV looked like the source; and the status page failing at the same moment made an attack look more likely than a configuration file.
- 5Check
The lesson
Configuration your own systems generate is input like any other. Validate it before it ships, keep the last good version when a new one fails the check, and make a hard limit reject bad data rather than crash the process reading it.
- 6Change
What changed afterwards
Cloudflare recovered by stopping the generation of the bad file, putting a known good file into the distribution queue by hand and forcing a restart of the core proxy.
Its stated follow-up work, which it called the beginning rather than the end: harden the ingestion of Cloudflare-generated configuration files in the same way as user-generated input, add more global kill switches for features, stop core dumps and other error reports from overwhelming system resources, and review the failure modes of error conditions across every core proxy module.
- 7Verify
Where the sources differ
- Cloudflare's post gives two start times: 11:20 UTC in its introduction and in the caption of its error chart, and 11:28 UTC ("first errors observed on customer HTTP traffic") in its timeline table. This page uses 11:20.
- The post gives no total duration. The three hours and the just under six hours here are worked out from its own times.
- The post says the file "doubled in size", that the query result was "more than doubling the rows", that about 60 features were in use and that the bad file had "more than 200 features". Doubling 60 does not reach 200, and the post does not reconcile the figures, so this page repeats them without doing the arithmetic.
- A common telling has the proxy failing outright. Only the newer FL2 proxy returned errors; the older FL proxy kept serving traffic with every bot score set to zero.
- 8Source
- 9Go