CrowdStrike: 97% of Windows sensors back online after outage

This story was updated July 26, 2024:

CrowdStrike CEO George Kurtz said 97% of Windows sensors are back online after a highly publicized global IT outage.

The CEO announced the update via a LinkedIn blog post Thursday evening. The post was used to share the latest on CrowdStrike's recovery efforts following the release of a defective channel file update that triggered blue screens of death and reboot loops in millions of Windows devices.

The outage, which occurred in the early hours of July 19th, caused significant disruptions to businesses and critical infrastructure services around the world such as airlines and hospitals. Microsoft said on Monday that under 1% -- approximately 8.5 million -- Windows devices were affected, but disruptions continued into this week.

CrowdStrike presented its initial findings regarding the cause of the outage to its remediation and guidance hub on Wednesday. The security vendor found that the issue resulted from a bug in the Falcon platform's content validator, which missed an errant configuration update for its Windows sensors last Friday. Moreover, because the update was classified as Rapid Response Content rather than Sensor Content, it did not undergo the same types of pre-release testing it would if it was classified as the latter.

Addressing the impact to affected devices proved challenging. Recovering each device requires manual restarts in safe mode to remove the defective file. To assist with the recovery, Microsoft and CrowdStrike released workarounds and recovery tools for customers.

Kurtz said in the LinkedIn blog post on Thursday that "over 97% of Windows sensors are back online as of July 25." He thanked customers, partners and employees for their "tireless efforts" to achieve this progress but acknowledged that work was not done until CrowdStrike had achieved complete recovery.

"To our customers still affected, please know we will not rest until we achieve full recovery. At CrowdStrike, our mission is to earn your trust by safeguarding your operations," Kurtz said. "I am deeply sorry for the disruption this outage has caused and personally apologize to everyone impacted. While I can't promise perfection, I can promise a response that is focused, effective, and with a sense of urgency."

Although CrowdStrike has provided significant detail quickly as it responds to the outage, questions remain regarding the nature of the content validation bug as well as whether it has been remediated. TechTarget Editorial asked for additional information, though a CrowdStrike spokesperson said in an email on Thursday night that Kurtz' statement was the latest word from the vendor at this time.

On Friday afternoon, a CrowdStrike spokesperson told TechTarget Editorial that the logic error that played a role in the faulty update was fixed and is currently being tested for full implementation into production early next week.

"The erroneous channel file template was removed from Channel File 291 at 2024-07-19 0527 UTC," the spokesperson said. "The logic error in the content validation system has been fixed and is currently being tested. The implementation is in our back-end systems, not customer-facing, and we will be releasing this to production early next week."

According to Kurtz, recovery efforts were enhanced "thanks to the development of automatic recovery techniques and by mobilizing all our resources to support our customers."

"CrowdStrike is committed to building on our mission to stop breaches, with a renewed focus on customer controls and resilience," he wrote. "Customer obsession has always been our guiding principle, and this experience has only strengthened our resolve."

Alexander Culafi is a senior information security news writer and podcast host for TechTarget Editorial.