Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.9Cloudbleed

An HTML (Hypertext Markup Language) parser at the edge of Cloudflare's network read past the end of its buffer and wrote other customers' memory into web pages, some of which search engines cached.

Cloudflare · 23 February 2017 · Memory safety and bounds

  1. 1Read
    Blast radius
    Pages served through Cloudflare for any customer, since a leak could carry data from any site behind the same server, not only sites with the affected features. Leaked memory held internal Cloudflare headers, cookies, authorisation headers and tokens; Cloudflare found it in more than 80,000 cached pages across search engines, covering about 150 customers' data. Customers' SSL (Secure Sockets Layer) private keys did not leak.
    Duration
    Leaks were possible from 22 September 2016 to 18 February 2017, worst from 13 to 18 February. Once Google reported it, Cloudflare had an initial mitigation in 47 minutes and was finished globally in under 7 hours, by its own account. Purging search engine caches went on for weeks.
  2. 2Follow

    What happened, in order

    1. 22 Sep 2016

      Cloudflare rolls out a new streaming HTML parser, cf-html, and moves Automatic HTTPS Rewrites onto it. The older parser, generated with Ragel, keeps running in the same chain. This is the earliest date memory could have leaked.

    2. 30 Jan 2017

      Server-Side Excludes moves to the new parser.

    3. 13 Feb 2017

      Email Obfuscation is partly migrated to the new parser. The number of sites whose pages can trigger the bug goes from fewer than 180 to 6,457.

    4. 17 Feb 2017

      Tavis Ormandy of Google Project Zero, working on a fuzzing corpus, finds data that does not match what he expected: fragments of other sites' HTTP (Hypertext Transfer Protocol) requests inside Cloudflare responses. It is Friday evening, so he asks for an urgent contact on Twitter.

    5. 18 Feb, 00:32 UTC

      Cloudflare receives the details from Google. Email Obfuscation is switched off worldwide at 01:19 Coordinated Universal Time (UTC), 47 minutes later.

    6. 18 Feb, 04:24 UTC

      Automatic HTTPS Rewrites is switched off worldwide. At 07:22 a kill switch for the cf-html parser is deployed, which covers Server-Side Excludes, a feature so old it had no global kill switch of its own.

    7. 18 to 23 Feb

      Cloudflare and Project Zero work with Google, Bing, Yahoo and others to find and purge cached pages that contain leaked memory.

    8. 20 Feb, 21:59 UTC

      A fix that checks every pointer access in the generated parser is deployed globally. The three features are switched back on at 18:03 UTC on 21 February.

    9. 23 Feb 2017

      Project Zero opens its report and Cloudflare publishes its incident report. On 1 March Cloudflare follows with its estimate of the impact.

  3. 3Find

    Root cause

    Note

    Cloudflare rewrote HTML as it passed through its edge servers, for features such as Email Obfuscation, using parsers that run as NGINX modules. The old parser was generated by Ragel, a tool that compiles a description of the grammar into C. Generated code moves a pointer p through the buffer and stops when p reaches pe, the end, with the check if (++p == pe). In one of Cloudflare's own rules, the error handler for a broken attribute inside a script tag jumped to the next state without first stepping p back by one, the operation Ragel calls fhold. When the broken tag sat at the very end of the page, that left p one byte past the end. The equality check never saw p equal pe, so the parser kept reading, and kept copying, whatever memory came next.

    The rule had been wrong for years without doing harm. With only the old parser in the chain, the last buffer of a page was never marked as the last, so the error handler could not run there. Adding cf-html changed how NGINX passed buffers between modules, the last buffer was marked as last, and the dormant error path became reachable. There was nothing wrong in cf-html itself. A leak also needed a final buffer under 4 kilobytes ending in a malformed script or img tag, which Cloudflare says occurs on about 0.06 percent of websites.

    What leaked was not the page's own data. The bytes after the buffer belonged to whatever else that server was handling, so a broken page on one site could carry headers, cookies and tokens from requests to entirely different customers. Cloudflare estimates the bug was triggered 1,242,071 times between 22 September and 18 February, at worst about 1 in every 3,300,000 requests, and that more than half of those requests came from search engine crawlers, which stored the pages.

    One byte past the end A page ending in a broken HTML tag sent Cloudflare's Ragel-generated parser into an error handler that skipped the step back Ragel calls fhold, leaving the pointer one byte past the end of the buffer. The generated check, ++p == pe, tests for equality, so it was jumped over and the parser read on into adjacent memory holding other customers' data. The new cf-html parser marked the last buffer differently, which made the old error path reachable. The extra bytes went out in HTTP responses and search engine crawlers cached them; more than 80,000 cached pages were purged. The page The old parser The pointer Server memory Origin page ends in a broken tag Ragel parser error path, no fhold End-of-buffer check ++p == pe, jumped over Adjacent memory other customers' data cf-html, new parser marks the last buffer HTTP response carries the extra bytes Search caches 80,000+ pages purged wakes the path a leak needed a final buffer under 4 KB ending in a malformed tag; at worst about 1 in every 3,300,000 requests, 13 to 18 February 2017
    A broken tag at the end of a page sent the parser's pointer one byte past the end, the equality check never fired, and other customers' memory went out in the response, where crawlers cached it.
  4. 4Trace

    Why it got through

    The bug and the change that woke it were in different places. Nobody changed the faulty rule; they added a new parser that was itself correct, and its effect on buffering was invisible from either module. Review of cf-html would not find a bug that was not in cf-html, and the old parser had passed years of production traffic, because the input that triggered it could not reach the broken path until the buffering changed.

    The last line of defence was a check the team did not write. Cloudflare points out that the generated equality test was produced by Ragel, and that a >= test would have caught the jump over the end. In C, reading past a buffer raises no error and returns ordinary bytes, so the first sign of trouble came from outside, from a researcher whose fuzzing corpus happened to contain Cloudflare responses. And because the output was public web pages, the leaked data was copied into caches Cloudflare did not control before anyone knew.

  5. 5Check

    The lesson

    A latent bug waits for an unrelated change to wake it, so a correct change can still be the one that breaks you. Bound every read in code that parses untrusted input, and remember that a leak into public caches outlives the fix.

  6. 6Change

    What changed afterwards

    Cloudflare added an explicit bounds check to every pointer access in the generated parser code, which logs and stops instead of reading on. Its security team fuzzed the generated code, built test cases from malformed pages found in the wild and inspected the code by hand, then began fuzzing older software, and Cloudflare brought in the code auditing firm Veracode for a full review of the parser.

    The cleanup lasted longer than the fix. Working with Google, Bing, Yahoo, Baidu, Yandex, DuckDuckGo and others, Cloudflare removed more than 80,000 cached pages by 1 March, and advised customers to invalidate session cookies and roll their internal authorisation tokens.

  7. 7Verify

    Where the sources differ

    • Cloudflare's first report counted 770 cached URIs (Uniform Resource Identifiers) on 161 domains, and said it was "satisfied that search engine caches have now been cleared". That sentence was later removed from the post, and Cloudflare's own follow-up a week later reported more than 80,000 cached pages removed.
    • The two accounts differ on severity. Ormandy listed private messages, password manager data, cookies, passwords and keys among what he saw, and wrote that Cloudflare's draft "severely downplays the risk to customers". Cloudflare's sample of leaks found cookies, authorisation headers and tokens but no passwords or credit card numbers.
    • The bug is often summed up as the == check. Cloudflare's report puts the root cause on its own grammar rule, the missing fhold in an error handler; the generated == check is why stepping past the end was not caught. Ormandy's first guess at the cause, an unchecked return value, was also wrong, as his report shows.
    • Ormandy found it in data from a fuzzing project, not by browsing ordinary pages. The 6,457 is sites whose pages could trigger the bug, not sites with the features on; customers whose data leaked "did not need to have flawed HTML or any particular Cloudflare features enabled". The page dates the case to public disclosure, 23 February 2017.
  8. 8Source

    Sources

  9. 9Go