santiago silver - Fotolia
Mitre enters product testing with Mitre ATT&CK framework
The first round of evaluations using the Mitre ATT&CK framework has gone public, putting on display how different endpoint products detect advanced threat activities.
Mitre has entered the security product testing and evaluation fray, and the organization is using its Mitre ATT&CK framework to judge vendors.
Seven vendors of endpoint detection and response (EDR) products submitted their endpoint security products to Mitre for evaluation testing. The objective of the evaluation was to demonstrate how the endpoint detection and response products responded to advanced threat actor activity via the Mitre ATT&CK, or Adversarial Tactics, Techniques and Common Knowledge, framework. In this case, the evaluations simulated attack techniques that have been used by the APT3/Gothic Panda threat actor, which has been active over the past few years using spear phishing and other tactics and techniques to gain access to victim organizations.
The Mitre Corp., a not-for-profit research and development organization headquartered in Maclean, Va., runs the Common Vulnerabilities and Exposures (CVE) system. The organization developed Mitre ATT&CK as a knowledge base of adversary tactics and techniques that can be used to flag attacks in progress -- especially when the attackers dispense with malware and opt to actively hack systems using a mix of exploits and techniques against vulnerable systems.
In its announcement, Mitre said the product evaluation program is intended "to help its government sponsors and industry make more informed decisions to combat security threats and advance industry threat detection capabilities." However, the organization also emphasized that the purpose of the testing was not to find a "winner." On its results page, Mitre noted that the "evaluations are not a competitive analysis. There are no scores, rankings, or ratings. Instead, we show how each vendor approaches threat detection in the context of the ATT&CK matrix."
Vendors laud Mitre ATT&CK evaluation process
Participating in the first phase of testing were EDR vendors Carbon Black Inc., CounterTack Inc., CrowdStrike, Endgame, Microsoft, RSA and SentinelOne; Cybereason and FireEye have already signed up for the next phase of testing. One interesting twist on the testing process was that Mitre required all vendors' products to be configured to issue alerts only, because the testing measured how well the products would be able to detect active attacks. Enabling "response" functions would shut down active attacks in process, Mitre said.
Mark Dufresne, vice president of research and development at Endgame, told SearchSecurity "we think we did great," and while noting his team didn't agree with a couple of the results, overall the testing process and assessment were very fair, in part because the evaluation was not focused on rankings or scores. Instead, the results were treated as additional data to be incorporated into the Mitre ATT&CK knowledge base.
"The testing was well organized, the data captured thorough, and the finalization of results fair and collaborative," Dufresne wrote in a blog post. "That last point is especially noteworthy given the huge amount of nuance and inherent lack of any one universal 'right way' to address much of ATT&CK."
"The testing approach of leveraging a complete nation-state campaign was very useful," said Shlomi Salem, research group manager at SentinelOne, who praised the thoroughness of the Mitre ATT&CK evaluations. "In total, 20 different stages were used across two different attack types. Having the ability to detect, automatically correlate and respond to all stages of the attack in the context of a single threat story was critical for defenders."
Mike Davis, CTO at CounterTack, approved of the process. "The Mitre ATT&CK framework is the best we have seen in that it is truly representative of real-world attack patterns," he said. "Just understanding the various ATT&CK details is a huge positive step, arming cybersecurity professionals with much needed knowledge about advanced, long-running attacks."
Amy Blackshaw, director of product marketing at RSA, was also enthusiastic about Mitre's foray in security product testing. "The Mitre ATT&CK evaluation was one of the best product evaluations that we've participated in. The objective nature of the evaluation mapped real life attacks to the product's capabilities, rather than just testing if the product found executing malware," she said. "By mapping to capabilities and techniques used in sophisticated, real-life campaigns, Mitre helped all security practitioners to gain an understanding of the strengths of all tools tested."
"This sort of open, real-world testing is very important in our industry," said Scott Taschler, director of product marketing at CrowdStrike. "Mitre's testing drives home that adversaries are adept at using native tools and hiding their attacks to blend in with normal day-to-day activities. Mitre's adversary emulation test is the first successful attempt to show how today's security technologies behave when subject to a full-spectrum, real-world simulated attack."
Microsoft and Carbon Black declined to comment on participating in the evaluations.
Mitre's noncompetitive ATT&CK evaluation process contrasts with the recent rancor over cybersecurity software testing run by NSS Labs Inc. In September, NSS Labs filed an antitrust lawsuit against CrowdStrike, Symantec and ESET, as well as the Anti-Malware Testing Standards Organization. The lawsuit accuses those organizations of working together to impede product testing by NSS Labs. Crowdstrike's response was dismissive, calling the lawsuit "baseless."
