Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.10Heartbleed

A missing bounds check in OpenSSL's TLS (Transport Layer Security) heartbeat let anyone ask a server for up to 64 kilobytes of its memory, again and again, with nothing abnormal in the logs.

OpenSSL · 7 April 2014 · Memory safety and bounds

  1. 1Read
    Blast radius
    Every server and client running OpenSSL 1.0.1 through 1.0.1f with heartbeats on. Netcraft found the heartbeat extension enabled on 17.5 percent of SSL (Secure Sockets Layer) sites with trusted certificates, around half a million certificates, as an upper bound on how many were exposed. Codenomicon, attacking itself, recovered its own private keys, user names and passwords, instant messages and emails.
    Duration
    Just over two years in released code, from OpenSSL 1.0.1 on 14 March 2012 to the fixed 1.0.1g on 7 April 2014. Fixing the code was the quick part: every operator then had to revoke and reissue keys and certificates and invalidate sessions that might have leaked.
  2. 2Follow

    What happened, in order

    1. 31 Dec 2011

      Support for TLS and DTLS (Datagram TLS) heartbeats is committed to OpenSSL. It was written by a co-author of the heartbeat specification and reviewed by the OpenSSL team.

    2. Feb 2012

      RFC 6520 (Request for Comments) defines the heartbeat extension. It says a heartbeat whose payload_length is too large must be discarded silently.

    3. 14 Mar 2012

      OpenSSL 1.0.1 is released with the heartbeat code, and the bug is in the wild.

    4. Then

      Neel Mehta of Google Security finds the bug and is the first to report it to the OpenSSL team.

    5. 3 Apr 2014

      Engineers at Codenomicon find the same bug independently while improving their security testing tools, and report it to NCSC-FI, Finland's national cyber security centre, to coordinate disclosure.

    6. 6 Apr 2014

      The fix is committed: "Add heartbeat extension bounds check."

    7. 7 Apr, ~17:30 UTC

      OpenSSL publishes its advisory and 1.0.1g. heartbleed.com follows at about 19:00 Coordinated Universal Time (UTC), before NCSC-FI has finished reaching affected vendors.

    8. 12 Apr 2014

      Steve Marquess of the OpenSSL Software Foundation sets out publicly how little money and staff OpenSSL has.

  3. 3Find

    Root cause

    Note

    Heartbeat is a keep-alive for TLS and DTLS. One side sends a HeartbeatRequest carrying a payload and a 16-bit payload_length field, and the other must reply with an exact copy of the payload. In OpenSSL 1.0.1 the handler, tls1_process_heartbeat, read payload_length from the message, allocated a reply of that size and ran memcpy(bp, pl, payload): copy that many bytes, starting at the payload. Nothing compared the claimed length with the length of the record that had actually arrived.

    Send one byte and claim 65,535, and the reply carries the one byte and then whatever lay after it in the process's memory, up to 64 kilobytes. There was no total limit: an attacker could reconnect, or keep asking on the same connection, and collect another 64 kilobytes each time. It needed no man-in-the-middle position and worked in both directions, so a malicious server could read a vulnerable client the same way. heartbleed.com says exploitation leaves no trace of anything abnormal in the logs.

    The fix was a single comparison. If 1 + 2 + payload + 16 is larger than the record received, return without replying, which the commit comments as "silently discard per RFC 6520 sec. 4".

    A length nobody measured A TLS (Transport Layer Security) heartbeat request carries a payload and a payload_length field. The attacker sends 1 byte of payload and claims 65,535. OpenSSL's heartbeat handler never checks the claim against the record it received, so it copies the claimed length into the response: the 1 real byte, then whatever sits next to it in the server's memory, which can include private keys, passwords and mail. Each request returns up to 64 KB, an attacker can keep asking, and nothing abnormal shows in the logs. The request The handler Server memory The reply Heartbeat payload 1 byte actually sent payload_length claims 65,535 bytes Heartbeat handler never checks the claim The real payload 1 byte Adjacent memory keys, passwords, mail Heartbeat response up to 64 KB copied back memcpy(bp, pl, payload): copies the claimed length, not the received one ask again for another 64 KB; nothing abnormal shows in the logs
    The request carried one byte and claimed 65,535. The handler copied the claim, so the reply carried the byte and whatever memory sat next to it.
  4. 4Trace

    Why it got through

    The specification had the check and the code did not. The implementation came from a co-author of the RFC and was reviewed by the OpenSSL team, and in Steve Marquess's words, no one spotted a problem. Reading code finds what the reader is looking for, and a length taken straight off the wire was not it. The language made the mistake quiet: in C an out-of-bounds read raises no error and returns ordinary bytes.

    Behind the code sat very little capacity. Marquess wrote that the OpenSSL Software Foundation typically received about US$2,000 a year in outright donations, that it had never taken in more than $1 million a year in total, and that one team member worked on OpenSSL without conventional full-time employment elsewhere. There should be at least half a dozen, he argued, and "the mystery is not that a few overworked volunteers missed this bug; the mystery is why it hasn't happened more often."

  5. 5Check

    The lesson

    Never trust a length you did not measure. And code that much of the internet depends on needs funding to match, or its safety rests on a few people's spare time.

  6. 6Change

    What changed afterwards

    OpenSSL 1.0.1g added the bounds check, and anyone who could not upgrade at once could rebuild with heartbeats switched off. The larger job fell on every operator: heartbleed.com advised revoking the compromised keys and reissuing new ones, and treating all session keys and session cookies as compromised.

    The funding gap became public. Donations to OpenSSL in the week after came to about US$9,000, by Marquess's count. The Linux Foundation set up the Core Infrastructure Initiative to fund critical open source projects, and on 29 May 2014 named Network Time Protocol, OpenSSH and OpenSSL as the first to be funded, with money for two full-time OpenSSL core developers and a security audit of OpenSSL by the Open Crypto Audit Project.

  7. 7Verify

    Where the sources differ

    • The advisory says "up to 64k" of memory per request. The 16-bit length field allows at most 65,535 bytes, which is where the 64 kilobytes comes from.
    • Heartbleed is often summed up as a share of all secure web servers. The only figure in the sources used is Netcraft's 17.5 percent of trusted-certificate SSL sites with heartbeats enabled, which Netcraft itself says overstates the vulnerable number. heartbleed.com's 66 percent is the market share of Apache and nginx, not the share that was vulnerable.
    • Session cookies are in heartbleed.com's advice on what to treat as compromised, not in the list of what Codenomicon actually recovered from its own servers.
    • OpenSSL's advisory credits only Neel Mehta. heartbleed.com says Codenomicon found the bug independently and that others went public before the coordinated disclosure finished, which left Codenomicon's own CVE (Common Vulnerabilities and Exposures) identifier, CVE-2014-0346, as a duplicate.
    • "Very little funding" holds for donations; the foundation also earned money from support contracts and consulting, under $1 million a year. The Core Infrastructure Initiative's founding announcement in April 2014 could not be fetched, so its pledges are not given here.
  8. 8Source

    Sources

  9. 9Go