Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.8left-pad unpublished from npm

A developer took all his packages off npm after a dispute over a name, and one of them, a few lines that pad a string, broke installs across JavaScript for two and a half hours.

npm · 22 March 2016 · Dependencies and supply chain

  1. 1Read
    Blast radius
    Every npm install that needed left-pad 0.0.3, directly or through a chain of dependencies. npm says it impacted many thousands of projects, and names babel and atom, which pulled it in through a package called line-numbers.
    Duration
    2.5 hours by npm's account, from shortly after 2:30 PM Pacific Daylight Time (PDT) on Tuesday 22 March 2016 until the original version was back by 4:55 PM.
  2. 2Follow

    What happened, in order

    1. Mar 2016

      Kik, the messaging company, and Azer Koçulu correspond over his npm package called kik and cannot agree. Kik asks npm for help.

    2. 18 Mar 2016

      Under its dispute policy, npm decides the package name kik should be maintained by Kik and tells both parties. Under that policy the existing package would normally have stayed on the registry.

    3. 22 Mar, ~2:30 PM PDT

      Koçulu has unpublished kik and 272 other packages, one of them left-pad. Shortly after 2:30 PM npm starts seeing hundreds of failures a minute as dependent projects, and their dependents, fail to fetch it.

    4. Within 10 minutes

      Another developer publishes a functionally identical left-pad as version 1.0.0. Errors continue, because line-numbers asks for exactly 0.0.3.

    5. 4:05 PM PDT

      npm announces that, at the request of the new owner, it will re-publish the original 0.0.3, which needs a restore from backup because re-publishing a version is otherwise impossible.

    6. 4:55 PM PDT

      left-pad 0.0.3 is back on the registry, and npm asks users to clear their cache before installing again.

    7. 23 and 29 Mar

      npm publishes its account of the incident, then a new unpublish policy.

  3. 3Find

    Root cause

    Note

    The registry let an author remove any version of any package at any time, however many other packages depended on it. Unpublishing left-pad took version 0.0.3 out of the registry, so every install that resolved it failed, and with it every package above it in the tree.

    left-pad was tiny. The 0.0.3 file is 17 lines, 11 of them not blank: a loop that adds a character to the front of a string until it reaches a length. It sat deep in the tree. line-numbers depended on exactly version 0.0.3, and babel and atom depended on line-numbers, so most of the people whose builds broke had never chosen left-pad.

    The quick fix could not work. A functionally identical 1.0.0 appeared within ten minutes, but a dependency pinned to exactly 0.0.3 does not accept 1.0.0, and the registry did not let anyone publish a version number again once it had existed. Only npm could put the original back, from a backup.

    One leaf, one exact pin left-pad 0.0.3 was unpublished along with 272 other packages. line-numbers asked for exactly version 0.0.3, and babel and atom brought left-pad in through line-numbers, so dependent installs failed by the hundreds per minute. A functionally identical left-pad 1.0.0 was published within ten minutes but did not satisfy the exact pin. The failures lasted 2.5 hours, until npm restored the original 0.0.3 from backup by 4:55 PM Pacific Daylight Time on 22 March 2016. The leaf Pinned it Brought it in Downstream left-pad 0.0.3 unpublished, with 272 line-numbers asks for exactly 0.0.3 babel dependency chain atom dependency chain Dependent installs hundreds failing a minute left-pad 1.0.0 same code, in 10 minutes npm restores 0.0.3 from backup, by 4:55 PM 1.0.0 does not satisfy a pin on 0.0.3, so chains through line-numbers kept failing until the original version came back 2.5 hours of failures, from shortly after 2:30 PM PDT, 22 March 2016
    Removing one small leaf failed every install above it, and an exact pin on 0.0.3 refused the identical 1.0.0 published ten minutes later.
  4. 4Trace

    Why it got through

    Nothing between an author and the registry asked whether a removal would break anyone. npm's dispute decision would have left the old kik package where it was, and npm's own account says it was the abrupt unpublishing, not the dispute resolution, that broke the builds. The missing safeguard was a check on dependents before removal, which is what npm added a week later.

    On the consuming side, builds fetched every dependency from the registry at install time and trusted it to be there. An exact pin protected an install from new versions but not from an old one disappearing, and a tool that pulled a few lines of code through a chain of other packages carried the risk of every author in that chain.

  5. 5Check

    The lesson

    Every dependency is a promise someone else has to keep, and a transitive one is a promise you never agreed to. Know what your tree pulls in, keep your own copy of what you build with, and weigh a tiny dependency against copying a few lines.

  6. 6Change

    What changed afterwards

    On 29 March 2016 npm changed its policy. A version less than 24 hours old could still be unpublished; anything older needed npm support, who would refuse if removing it would break other installs. A package with every version removed would be replaced by a security placeholder package.

    The policy was tightened again on 30 January 2020. Today a package can be unpublished within 72 hours if nothing else in the public registry depends on it, and after that only if nothing depends on it, it had fewer than 300 downloads in the last week and it has a single owner. left-pad itself is now deprecated, with a notice pointing to String.prototype.padStart().

  7. 7Verify

    Where the sources differ

    • The often quoted 273 packages is arithmetic: npm says kik "and 272 other packages".
    • Many accounts say left-pad broke React. npm's post names babel and atom, and Kik's own post says its builds broke through JSCS; no source used here names React.
    • left-pad is usually called 11 lines. The 0.0.3 file has 17 lines, 11 of them not blank.
    • Koçulu's post says a patent lawyer contacted him. Kik's post, which publishes the email thread, says the sender was its patent agent, not a lawyer.
  8. 8Source

    Sources

  9. 9Go