CrowdStrike details errors that led to mass IT outage

Remediations and improvements

CrowdStrike was heavily criticized over the last several weeks by critics who said the cybersecurity vendor's testing processes and safeguards for content updates were woefully inadequate. CrowdStrike said in its preliminary report that content updates like Channel File 291 are what the vendor calls Rapid Response Content, which previously were not subjected to the same internal tests and reviews as software updates.

In its root cause analysis, CrowdStrike outlined several mitigations and changes to improve the process. The company said it updated the Falcon platform to give customers greater control over how Rapid Response Content is deployed. "Customers can choose where and when Rapid Response Content updates are deployed," the report read. "We are continuing to enhance this capability to provide more granular control over Rapid Response Content deployments together with content update details via release notes, to which customers can subscribe."

Content and template type updates will also undergo more testing procedures, including fuzz testing, CrowdStrike said. Additionally, the company will use staged deployment for such updates going forward.

"Staged deployment mitigates impact if a new Template Instance causes failures such as system crashes, false-positive detection volume spikes or performance issues," the report read. "New Template Instances that have passed canary testing are to be successively promoted to wider deployment rings or rolled back if problems are detected. Each ring is designed to identify and mitigate potential issues before wider deployment."

Lastly, CrowdStrike said the content validator, which allowed the defective channel file to pass through to the content interpreter, has been modified to only allow wildcard matching criteria in the 21st field, which mitigates issues in sensor updates that only provide 20 inputs. The vendor said it will release an additional fix for the content validator later this month.

"The Content Validator is being modified to add new checks to ensure that content in Template Instances does not include matching criteria that match over more fields than are being provided as input to the Content Interpreter," CrowdStrike said. "This fix will be released to production by August 19, 2024."

Along with the root cause analysis, CrowdStrike said it is conducting independent reviews of internal processes. "CrowdStrike has engaged two independent third-party software security vendors to conduct further review of the Falcon sensor code for both security and quality assurance," the company said. "Additionally, we are conducting an independent review of the end-to-end quality process from development through deployment."

Dustin Childs, head of threat awareness at Trend Micro's Zero Day Initiative, told TechTarget Editorial that CrowdStrike's overall response has been good, with mostly consistent messaging and a rapidly mobilized response team to assist customers with remediation.

"In a lot of ways, for crisis response, they did relatively well," he said. "I hate to say that for a competitor, but I'll give props where it's due."

However, Childs had concerns about the root cause analysis report.

"It's really hard for me to make an accurate judgment, because I don't think they are being 100% transparent," he said. "I think they're withholding information because that's sometimes what CrowdStrike does. They've always played things very close to the vest."

As an example, Childs said CrowdStrike has typically required security researchers to sign NDAs regarding vulnerabilities found in the company's products. "They're not very transparent when it comes to their security."

Senior security news writer Alex Culafi contributed to this article.

Rob Wright is a longtime reporter and senior news director for TechTarget Editorial's security team. He drives breaking infosec news and trends coverage. Have a tip? Email him.