Machine learning in cybersecurity: How to evaluate offerings

Evaluate outcomes, not ingredients

Sam Curry is the CISO of Cybereason, a threat detection and defense company with technology roots in the Israel Defense Forces. Curry said security managers should be grilling vendors not in terms of machine learning tools, but the outcomes.

"I'm much more interested in knowing what kind of data you collect, where you put it and how you store it," he said. "How do you slice it? How do you render it? How does someone interact with it? How do you use it?"

The answers to these questions should be clearly expressed in the language of an enterprise's particular IT and application landscape.

Equally important, the vendor should be able to specify how its offering exceeds or meets the pace of your adversary's innovation.

"Since bad actors are so focused on the problem, it becomes a content race," Curry said. "If they claim to use Bayesian analysis, for example, will they switch to another tool if that doesn't work over time?"

Bayesian analysis is one of about a dozen machine learning methods typically used; other methods are logistical regression, simple linear regression, K-means clustering, decision trees and random forests. You can make a study of them -- look up Data Science on Coursera -- but you won't learn enough in time to prevent the next malware attack.

On a continuum between deep understanding of machine learning models and viewing it as a black box, Larry Lunetta, vice president of security solutions marketing for Hewlett Packard Enterprise's Aruba Networks, finds a middle ground. He says that security analysts should understand how machine learning in cybersecurity works in general, and specifically that its results are not binary.

Alerts to threats are expressed as probabilities, which must be put in context with additional evidence. He adds that Aruba's IntroSpect product, the result of its February 2017 acquisition of Niara, presents this underlying evidence along with attack probability scores.

"What kind of network traffic did we see? What kind of log information was relevant? What was the history of the user or device in question? These are all types of evidence that the security analyst would already be well-versed in," Lunetta explained. "We look upon machine learning as a force multiplier."

Simpler tools may suffice

Dennis Chow is the CISO at SCIS Security, a Houston-based reseller that was formed a year and a half ago by longtime IT security veterans. He cautions against concentrating on the latest buzzwords for machine learning in cybersecurity when simpler analytical tools and SIEM systems, if properly set up, will prevent most attacks.

Chow also agrees that enterprises don't need a doctorate in data science to evaluate products.

"Security managers can simply ask the questions that address the highest risks to their organization," based on careful risk assessments, Chow said.

"If we do a professional services deployment for a customer, we go over their top use cases," he said. "Say you're a healthcare entity; you might be more worried about HIPAA-based data exfiltration, or about biomedical devices being hacked through MS08-067 -- one of many old vulnerabilities common to the industry. We'll go over the top three, four, five, and create modeling based on that."

One thing Chow does believe managers should know is the difference between trained, or supervised, and untrained model sets. In a supervised model set -- all he has deployed thus far -- "we give the system training data and tell it how it should be classified or identified."

An unsupervised or untrained model isn't trained for anything, in theory. The algorithms figure out their own baselines out of raw data, then determine clusters of outlying data points and assign these clusters labels. The solution vendor then maps those labels to things that have meaning for security; for example, a brute force attack.

"The issue is the unknown," says Chow. "If the model comes back and says something the user doesn't recognize, he'll have to sit there and dig through it just like any other security analyst without the tool."

While Chow can roughly explain the two machine learning algorithms they use most often in their work -- K-means clustering and simple linear regression -- he recommends leaving the front-end research to a value-added reseller (VAR).

"Any good account manager is going to have presales engineers to help you choose." And again, frame questions in terms of outcomes: "Can this technology tell me when there's anomalous user activity at an odd hour, or it's coming from a country we don't do business with?"

In addition to leaning on a VAR, buyers often get recommendations from peers in similar companies, as well as input from industry analysts.

The proof is in the POC pudding

In the end, an in-house security engineer or a trusted third party should validate security vendor claims in a proof of concept (POC) trial. Chow recommends testing for advanced persistent threats at all points along the kill chain, from initial reconnaissance to uncovering vulnerabilities to payload delivery, all the way to data exfiltration.

Testing should also establish a baseline of tool effectiveness in detecting known signatures -- something that doesn't require machine learning.

"Then switch it up with a different exploit kit, a different piece of malware, a different infection point -- say a USB flash drive instead of an infected URL," Chow said.

After exercising all the rule-based detection, Chow suggests testing the machine learning aspects with variations on expected bad behavior. See if the system still sees enough similarity to catch it without additional programming.

A thorough POC trial can help enterprises determine the technology that best suits their needs. Machine learning tools may offer superior security for enterprises, but they must be properly researched, vetted and tested before being deployed.