Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.4CrowdStrike Channel File 291

A content update to a security sensor crashed Windows machines worldwide, because data the kernel driver read was shipped to everyone at once.

CrowdStrike · 19 July 2024 · Change and rollout

  1. 1Read
    Blast radius
    An estimated 8.5 million Windows devices, by Microsoft's count less than one percent of all Windows machines, many of them in enterprises that run critical services. Mac and Linux hosts were not affected.
    Duration
    The faulty file was live for 78 minutes, 04:09 to 05:27 Coordinated Universal Time (UTC). Recovery took far longer: on 29 July, ten days later, CrowdStrike reported about 99 percent of Windows sensors back online.
  2. 2Follow

    What happened, in order

    1. 28 Feb 2024

      Sensor 7.11 ships to customers with a new IPC (interprocess communication) Template Type, built to detect attacks that abuse Windows named pipes. Sensor code goes through CrowdStrike's staged rollout.

    2. 5 Mar 2024

      The new template type passes a stress test in staging, and the first IPC Template Instance goes to production as a content configuration update.

    3. 8 to 24 Apr 2024

      Three more IPC Template Instances deploy and work as expected.

    4. 19 Jul, 04:09 UTC

      Two new Template Instances ship in Channel File 291. One passes the Content Validator despite carrying problematic content, and goes to every online Windows host on 7.11 or later.

    5. 19 Jul, 04:09 UTC

      Hosts that load the file crash with a Windows stop error (the blue screen). The sensor's kernel driver loads early in boot, so a crashed host has to be brought back without the file.

    6. 19 Jul, 05:27 UTC

      CrowdStrike reverts the channel file. Hosts that come online after this are not affected; hosts already crashed need repair.

    7. 20 Jul

      Microsoft publishes its estimate of 8.5 million affected devices and works with CrowdStrike on recovery tooling.

    8. 6 Aug

      CrowdStrike publishes its root cause analysis with six findings and a mitigation for each.

  3. 3Find

    Root cause

    Note

    Falcon detects new attack techniques with Rapid Response Content: configuration data, delivered as channel files, that a Content Interpreter inside the sensor's kernel driver matches against system activity. Each piece of content is a Template Instance of a Template Type compiled into the sensor. Changing content does not change sensor code, which is the point: it lets detection move faster than releases.

    The IPC Template Type defined 21 input fields. The sensor code that called the Content Interpreter for Channel File 291 supplied only 20 values. Every instance until 19 July used a wildcard for the 21st field, so nothing ever read it. One of the two new instances used a real matching criterion in field 21. The interpreter tried to read the 21st entry of a 20-entry array, an out-of-bounds read in kernel mode, and Windows stopped the machine. The crash dump shows the fault in csagent.sys, at index 0x14 of a 20-pointer array.

    Fixing a machine meant getting it to run without the bad file. CrowdStrike's guidance was to restart on a wired network so the host could pick up the reverted file; failing that, to boot from recovery media or follow Microsoft's manual steps and delete the files matching C-00000291*.sys. BitLocker-encrypted hosts could need a recovery key first. Cloud machines needed the file deleted from their disks or a rollback to a snapshot taken before 04:09 UTC.

    Two paths into the same kernel driver Two rows. Sensor code moves through a staged rollout: release, CrowdStrike's own hosts, early adopters, then general release with customers choosing N, N-1 or N-2. Rapid Response Content moves in one push: a Template Instance passes the Content Validator wrongly, reaches every online Windows host on sensor 7.11 or later, and the Content Interpreter in the kernel driver reads field 21 of a 20-value array, which crashes the system. Sensor code: staged rollout Sensor release code, certified Dogfood CrowdStrike's own hosts Early adopters a slice of customers General release N, N-1 or N-2 by policy Rapid Response Content: one push Template Instance configuration data Content Validator one passed, wrongly Every online host Windows, sensor 7.11+ Content Interpreter in the kernel driver Out-of-bounds read field 21 of 20 System crash csagent.sys bugcheck no canary, no rings: 04:09 to 05:27 UTC, 19 July 2024
    Sensor code moved through rings. The content that drove it went to every online host in one push, and the kernel driver read a field that was never supplied.
  4. 4Trace

    Why it got through

    Several separate defences each had a hole, and on 19 July the holes lined up. The field count was not checked when the sensor was compiled. The interpreter had no runtime bounds check on its inputs. Testing of the template type used a static set of 12 test cases whose test channel file had a wildcard in the 21st field, so the 20-versus-21 mismatch never ran. The Content Validator assumed 21 inputs, so a 21st field looked valid to it.

    The last defence was missing outright. Sensor code moved through internal dogfooding, then early adopters, then general availability, and customers could hold their fleet one or two versions back. Rapid Response Content had no equivalent: the instances deployed on the strength of the March stress test, trust in the validator and the earlier successful deployments. CrowdStrike's own analysis describes the content as configuration data, not code. It still decided what the kernel driver did.

  5. 5Check

    The lesson

    Data that steers code is code. If a change can crash the machine, it needs the same canary, staged rings and rollback as a code release, however small and however routine it looks.

  6. 6Change

    What changed afterwards

    CrowdStrike fixed each hole its analysis found: the compiler now checks the field count (in production from 27 July), the interpreter checks array bounds (added 25 July), the template type now supplies 21 inputs, the validator gained new checks and only allows a wildcard in field 21, and every new template type is tested with non-wildcard criteria in every field.

    The wider change was to rollout. Rapid Response Content now goes to a canary first and then to successively wider rings, with bake-in time and telemetry at each step, and customers can choose where and when it is deployed. CrowdStrike also brought in two independent software security vendors to review the sensor code and its quality process.

  7. 7Verify

    Where the sources differ

    • The 8.5 million figure is Microsoft's estimate ("we currently estimate"); CrowdStrike reports the share of sensors online, not a device count.
    • The preliminary review calls the trigger "a bug in the Content Validator". The later root cause analysis treats the validator as one of several conditions that all had to hold for the crash to happen.
  8. 8Source

    Sources

  9. 9Go