Information Security
lolloj - Fotolia
Threat hunting techniques move beyond the SOC
Tired of waiting for signs of an attack, companies are increasingly adding threat hunting capabilities to their playbooks to find likely ways their systems could be infiltrated.
When James Mercer decided to hunt for potential adversaries targeting his company's data centers, the CIO of Flexi-Van Leasing Inc. looked at ways that attackers infiltrated other commercial and industrial equipment companies and searched for signs of similar attacks.
"We found a malbehaving SQL server environment by looking through the various possible threats and approaches," Mercer said. "First, there was a panic -- Oh my god, we were seeing login information -- but then we discovered it had been misconfigured by someone many years ago."
Although the hunt did not reveal an actual attack, the process convinced Mercer that using threat hunting techniques is a valuable exercise. "An ounce of prevention -- that's the great value of threat hunting," he said. "It is not about whether you find something or not, but whether you find a pathway that could have been exploited."
For that reason, the 250-person firm, headquartered in Kenilworth, N.J., has become more invested in threat hunting techniques. It's an advanced capability mature security operations centers (SOCs) employ that many security professionals assume is too complex for companies without teams of analysts to adopt. In Flexi-Van's case, monitoring static perimeters and system alerts to detect anomalies and vulnerabilities wasn't enough. The company -- which manages 125,000 mobile cargo containers, or chassis, in more than 300 locations across the United States -- needed to know that it could trust the more than 1.3 billion bills of lading and events produced annually that allow it to track its rail, ship and trucking business.
The emerging practice of threat hunting can include a range of security and monitoring operations. Tier-3 security analysts think about how adversaries could harm information and other corporate assets, and they hunt for traces of past and present attacks using malware analysis, data analytics, pattern recognition and forensics.
Threat hunting, generally, encompasses two parallel lines of thinking, however. First, threat hunting is not an automated process based on rules and algorithms; it is driven by hypotheses of what advanced threats toward specific assets might look like during a bounded operation or timeframe. The corollary is that threat hunting is generally not triggered by compromise; the ultimate goal is to improve the performance of automated systems so that they can block previously undetected attacks.
"It comes down to, 'I have a hypothesis on an attack technique, and I want to test the hypothesis out in my environment to see if I'm vulnerable,'" said Matt Downing, senior principal threat intelligence researcher at Alert Logic Inc., a managed security services provider that helps firms, like Flexi-Van, with threat hunting. "The whole point is not being completely reactive."
James Mercer
Other companies agree, which is why threat hunting -- whether ad hoc or based on formal procedures -- has taken off as an increasingly preferred way in which organizations large and small can employ a proactive approach to security. In the "2018 Threat Hunting Report" based on a Crowd Research Partners survey of 461 professionals in the Information Security Community on LinkedIn, more than half of the respondents (56%) indicated that reducing exposure to external threats was the primary goal of their organization's threat hunting program. For roughly half (52%), improving speed and accuracy of threat response ranked highest, while lowering the number of breaches and infections was also favored by close to half (49%). The remaining goals of threat hunting programs were: decreasing the time to containment (45%), reducing the attack surface (42%), minimizing the exposure to internal threats (41%), reducing dwell time from infection to detection (39%), and optimizing resources spent on threat response (34%), while a small percentage chose other (8%).
Don't start until you are ready
Threat hunting allows companies to be proactive and look for potential threats, rather than wait for attackers to trigger alarms after they have already infiltrated an organization's systems. Yet not every company is ready to employ threat hunting techniques.
"The nutshell is that almost every entity out there could benefit from some threat hunting," said Rob Lee, curriculum lead for digital forensics and incident response at the SANS Institute. "But companies need the baseline security information capability -- you need to know your network and what you are defending."
To do that, companies need at least an entry-level SOC capability, he said.
Threat hunting typically evolves from the incident response function of SOCs at bigger companies.
"You might have tier-3 analysts looking at the big picture and bringing it up as a business risk," said Gene Stevens, co-founder and CTO of ProtectWise Inc., a SaaS provider that offers cloud-based network detection and response. "But once you know who the actors are -- and their methods -- you can go out and look for other things they are known for and see if those appear in your environment."
Because threat hunting often involves sifting through massive amounts of data, companies shouldn't underestimate the value of data analytics and visualization.
"You can't hunt without an insane amount of data, but you need to gain good visualization to have an opportunity to see how things relate to each other," Stevens said. "With the correct visualization, we can look at why that is different today. And that is a form of hunting, and people can do that for themselves."
Can small firms hunt?
Organizations with a few thousand systems may not have the depth of skills -- nor a deep pool of anomalous events -- to make threat hunting techniques worth it. Simple tools and relatively shallow analysis, however, can still turn up interesting findings.
DevOps and security consultant Apollo Clark, who is based in the greater Boston area, woke up one morning, searched a list of common attack vectors and found an anomaly: One admin for a major cloud provider had logged in from Eastern Europe and dumped a couple hundred SQL instances from a database cluster.
It took an hour to find out that what could have been an attack in progress was actually a staffing oversight. The company had hired a new database administrator who had been tasked with backups and had not registered his account with security.
"There is always going to be that human element that needs to be checked," Clark said.
CrowdStrike Inc. -- known for its threat hunting work on the Sony Picture attacks and the email hacks of the Democratic National Committee -- has found that just network telemetry, a well-defined list of assets and basic threat intelligence can uncover some pernicious attacks. The company, which offers endpoint protection, antivirus and threat intelligence through its cloud-based Falcon platform, stresses that threat hunting techniques should match the situation and the sophistication of the team.
Jennifer Ayers
Everyone has to start somewhere, noted Jennifer Ayers, senior director of CrowdStrike's Falcon OverWatch threat hunting and security response group, but organizations need to scale their threat hunting capabilities accordingly.
"There is no upper end for when you should be doing threat hunting on your own, but there is a lower end," she said. "If you are looking at the much, much smaller companies -- those with a much smaller footprint, say, 100 systems -- you might not get the value-add from something like threat hunting, because not a lot is going on."
The other issue is whether smaller organizations can use the results of threat hunting to improve their defenses. To make use of the information gleaned from threat hunting, companies need a place to process the information and incorporate it into their current defenses, according to the team leader of threat hunting and incident response for a larger retailer, who spoke on condition of anonymity.
"You need to have a full-time incident detection and response capability, even if it is not a 24/7 SOC operation," the director of threat hunting stressed. "You are going to find issues and incidents with threat hunting, and you need processes in place to use that information in your defenses."
For that reason, most threat hunting groups need to be doing it full time and as part of a larger security team.
Cloud remains a blind spot
The biggest challenge for many companies, however, is hunting for threats in the cloud. Most organizations do not have access to log information from their public cloud providers, which inhibits visibility into cloud infrastructure.
"You can't go in and check out what is happening," said Flexi-Van's Mercer. "So, there is a huge, intrinsic blind spot where you have to trust the cloud provider is blocking the threat better than you could."
Yet, done right, different levels of hunting can benefit from the same scalability and continuous deployment that helps applications deployed in the cloud.
At the SANS Threat Hunting and Incident Response Summit last year, Netflix, a media services provider based in Los Gatos, Calif., showed how it used Amazon Web Services and a variety of cloud technologies to instantly deploy necessary resources to aid in hunting.
The ability to change things around on the fly and have infrastructure as code can make hunting much more scalable and efficient, noted Alex Maestretti, the engineering manager of Netflix's security intelligence and response team.
"Rather than have to build a large deployment of endpoint monitoring or logging pipeline, you can use all the things that AWS has given you for free and go ahead and push out a live response script," he said, adding, "You take the skills that you could use to investigate one or two boxes using SSH and, all of a sudden, you are doing it at cloud scale."
Cloud-based hunting may be the most complex endeavor, however. While the infrastructure benefits companies' hunting teams, it also gives attackers access to native cloud technologies that can be used as innovative new vectors to infiltrate the infrastructure of the business.
Attackers are now much less likely to drop special malware and rootkits and, instead, use stolen administrator credentials and native tools that will not raise flags.
"The really high-end threats prefer credentials over dropping tools," Maestretti said. "If they can get [credentials] and act as a legitimate user, that is a much quieter way to move around your environment."
In the end, hunters will have to adapt to attacks that are not malicious on their face, but follow a pattern that matches a probable attack technique.
"When you consider hunting, there are two general categories," Maestretti said. "Either you have an indicator of compromise you want to find or you are just looking for weird."
