Case study · 2024
CrowdStrike Falcon update outage, July 2024
A faulty security content update crashed about 8.5 million Windows devices. No attacker involved.
Outage · change management
What happened
- On 19 July 2024, CrowdStrike delivered a Rapid Response Content update (Channel File 291) for its Falcon sensor on Windows. Affected machines crashed and could not restart normally.
- Microsoft estimated that about 8.5 million Windows devices were affected, less than one percent of all Windows machines, but they included systems at airlines, banks, hospitals and broadcasters.
- Many systems had to be fixed by hand, one machine at a time, which made recovery slow. macOS and Linux hosts were not affected.
- CrowdStrike published a root cause analysis on 6 August 2024. It was not a cyber attack.
Root cause
A new sensor template type defined 21 input fields, but the sensor code supplied only 20. Earlier content used a wildcard for the 21st field, so the mismatch was never triggered. The 19 July update was the first to use a non-wildcard value for that field, causing an out-of-bounds memory read and a system crash. The content validator checked against the 21-field definition, so it did not catch the problem.
Control failures
- Testing did not include a case for a non-wildcard value in the 21st field.
- The content update went to all affected hosts at once rather than in staged, monitored rings.
- Customers could not control when this type of content update was applied.
- Many organisations had no fast, tested way to recover large numbers of endpoints, especially those with disk encryption keys to look up.
Lessons for GRC
- Security tools are change too: include vendor updates to privileged, kernel-level software in change and third-party risk reviews.
- Ask critical suppliers how they stage releases, test and roll back, and what control customers have over update timing.
- Business impact analysis should cover “our security tool takes us down”, not only attacks.
- Keep recovery runbooks and encryption recovery keys accessible when the main systems are down.