Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.7The xz/liblzma backdoor

An attacker spent over two years becoming a maintainer of a compression library, then shipped a backdoor aimed at SSH (Secure Shell) logins on Linux servers. One engineer noticed logins were half a second slow.

XZ Utils (the Tukaani Project) · 29 March 2024 · Dependencies and supply chain

  1. 1Read
    Blast radius
    xz and liblzma versions 5.6.0 and 5.6.1, which reached rolling and pre-release Linux distributions: Debian unstable and testing, Fedora Rawhide and the Fedora 40 beta, openSUSE Tumbleweed and MicroOS, Kali and Arch Linux. No major stable release shipped it, and on some of these the payload never activated. Rated 10.0, critical, by the National Vulnerability Database.
    Duration
    About five weeks in the wild: 5.6.0 was tagged on 24 February 2024 and the backdoor was disclosed publicly on 29 March. The work that led to it ran for over two years, from a first patch in October 2021.
  2. 2Follow

    What happened, in order

    1. 29 Oct 2021

      A contributor using the name "Jia Tan" sends a first patch to the xz-devel mailing list. The first commit under that name is merged in February 2022.

    2. Apr to Jun 2022

      Accounts named "Jigar Kumar" and "Dennis Ens" complain on the list about slow progress and press for a new maintainer. The maintainer, Lasse Collin, replies that his ability to care has been limited by long-term mental health issues, that Jia Tan may have a bigger role in the future, and that this is "an unpaid hobby project".

    3. Oct 2022 to Mar 2023

      Jia Tan joins the project's GitHub organisation, is named as a maintainer in the README, gets commit access and makes a first release, 5.4.2, on 18 March 2023.

    4. 23 to 24 Feb 2024

      Two binary test files carrying the obfuscated payload are committed. 5.6.0 is tagged, and its release tarball contains a modified build-to-host.m4 that is not in the repository.

    5. 26 Feb to 28 Mar 2024

      Debian unstable picks up 5.6.0. Version 5.6.1 follows on 9 March with an updated payload, and accounts including Jia Tan's ask Debian and Ubuntu to ship it.

    6. 28 Mar 2024

      Andres Freund, a PostgreSQL developer working at Microsoft, traces unexpected CPU (processor) use in sshd to liblzma and reports it privately to Debian and the distributions' security list. Red Hat assigns CVE-2024-3094 (Common Vulnerabilities and Exposures). Debian rolls back.

    7. 29 Mar 2024

      Freund posts his analysis to the oss-security list, and Red Hat says Fedora Rawhide and the Fedora 40 beta received the affected versions. Distributions revert to 5.4.x.

  3. 3Find

    Root cause

    Note

    The payload lived in the git repository in plain sight, as two binary test files (bad-3-corrupt_lzma2.xz and good-large_compressed.lzma) that no test in 5.6.0 even used. The trigger did not live there at all. A modified build-to-host.m4, present only in the release tarballs, ran an obfuscated script at the end of configure that extracted a prebuilt object file from the test data and spliced it into liblzma. It only did so on x86-64 Linux, building with gcc and the GNU linker, inside a Debian or RPM package build.

    OpenSSH does not use liblzma. But Debian and several other distributions patch sshd to support systemd notification, libsystemd depends on liblzma, and so the library loaded into the SSH daemon. The backdoor replaced two of liblzma's ifunc resolvers, which run early in loading, used them to install an audit hook in the dynamic linker, and waited for RSA_public_decrypt to be resolved so it could point it at its own code. That code runs before authentication; Freund judged it "likely to allow some form of access or other form of remote code execution".

    It was caught by its cost. Freund was micro-benchmarking on Debian unstable and noticed sshd using a surprising amount of CPU on logins that failed at once. An ssh connection to a user that did not exist took 0.807 seconds instead of 0.299, and he remembered odd valgrind errors from a few weeks earlier, after a package update.

    From a test file to the SSH daemon Two binary test files committed to the xz git repository held the obfuscated payload. A modified build-to-host.m4, present only in the 5.6.0 and 5.6.1 release tarballs created and signed by the maintainer account Jia Tan, ran after configure and spliced an object file into liblzma, but only for x86-64 Linux builds with gcc and the GNU linker inside a deb or rpm package build. libsystemd depends on liblzma, and several distributions patch sshd to link libsystemd, so the backdoor could redirect RSA_public_decrypt in sshd before authentication. In git Only in the tarball The built library The way in Two test files obfuscated payload build-to-host.m4 runs after configure liblzma 5.6.x object file spliced in libsystemd depends on liblzma sshd patched for systemd RSA_public_decrypt redirected, pre-auth Release tarballs signed by Jia Tan injected only on x86-64 Linux, built with gcc and the GNU linker, inside a deb or rpm package build
    The payload sat in git as test data, the trigger only in the release tarball, and a distribution patch connected liblzma to the SSH daemon.
  4. 4Trace

    Why it got through

    Every layer of review looked where the attack was not. Anyone reading the repository saw opaque binary test data and no trigger. Distributions built from the signed release tarballs, which they trusted to match the repository, and those tarballs were created and signed by the maintainer account the attacker controlled. The injection fired only in Debian and RPM package builds, which kept it off a developer's own build.

    The trust was earned, on the face of it. Jia Tan did over two years of useful work. The project's long-time maintainer, by his own account short of time and running an unpaid hobby project, was pressed in public by accounts that Russ Cox's timeline judges were likely fakes created to push him to hand over control. Sharing the load with a helpful contributor was the reasonable response. A library that sits under SSH on many Linux servers had one long-standing maintainer and nobody paid to help him.

  5. 5Check

    The lesson

    Build what you reviewed. If the artefact you ship can differ from the source you read, your review covers less than you think, and the health and funding of the people who maintain your dependencies is part of your own security.

  6. 6Change

    What changed afterwards

    Distributions reverted to 5.4.x and Debian rebuilt its build machines. GitHub suspended the accounts of both Jia Tan and Lasse Collin (his was back on 2 April), and the repositories came back on 9 April 2024 with their history intact. Lasse Collin published a review of the backdoor and, on 29 May 2024, new clean releases. Separately, a systemd change merged on 5 March 2024, before the disclosure, stopped linking liblzma into libsystemd.

    The wider change was attention to how maintainers are recruited. On 15 April 2024 the OpenSSF (Open Source Security Foundation) and the OpenJS Foundation warned of social-engineering takeovers, after OpenJS intercepted a similar credible attempt and saw the same pattern around two other popular JavaScript projects. They listed the signs: unknown accounts pressing for maintainer access, sock puppets, and pull requests carrying binary blobs.

  7. 7Verify

    Where the sources differ

    • The payload is often said to have been absent from git. It was in git, as binary test files; only the trigger, the modified build-to-host.m4, was in the release tarballs alone.
    • The half second is client wall-clock time for an ssh connection, 0.299 to 0.807 seconds, with the client's own CPU time unchanged. Freund's post does not name his employer; his own profile says he works at Microsoft. He was benchmarking on Debian unstable machines, not a dedicated test machine.
    • Russ Cox calls the accounts pressing for a new maintainer "likely" fakes. That is an inference from their having no other presence online, not a proven fact.
    • Fedora 40 beta and Arch Linux shipped the 5.6.x packages, but Red Hat says the beta does not appear to be affected by the exploit, and Arch says the payload never reached its binaries because its build was not a Debian or RPM one.
    • The date here is the public disclosure. The backdoored 5.6.0 was tagged on 24 February 2024.
  8. 8Source

    Sources

  9. 9Go