During a decade in the US Air Force, Chris Gerritz focused on responding to incidents and finding attackers and their attempts to compromise the network. As part of that mission, the USAF often had ad hoc hunt teams investigating incidents and attempting to find any attacks. These days, most large companies are in a similar place: They have the ability to investigate incidents and to hunt down threats, but there is no overarching process for conducting the investigations, said Gerritz, founder of the threat hunting platform Infocyte.
"They have too much data and too many alerts, and they are missing things, and so they have to hunt," Gerritz said. "But they are usually hunting on some kind of periodic basis that is usually determined by need and the amount of resources they have."
As organizations' security programs mature, an increasing number are not content to merely defend their networks by erecting digital walls and preventing malware attacks and compromises. Instead, their security teams are investigating incidents on their networks and looking for signs of attackers before a compromise turns into a full-on breach.
The majority of companies, however, continue to launch investigations only when necessary, rather than having a process for continuously evaluating whether threats may have infiltrated their network and systems. About 84% of companies perform some sort of threat hunting, but the majority of those—54%—only do so as needed, according to a recent SANS Institute report.
Companies that do regular threat hunting, however, have seen significant benefits. About 60% have measurably improved their security based on information discovered during their hunts, according to SANS. In mature security organizations, threat hunting uncovers approximately 40% of security incidents, said Gerritz.
Here are four ways to start threat hunting the right way.
1. Just get started
Threat hunting is a developing discipline, and while there are some experts, it's easy to feel overwhelmed. And it continues to be an expensive proposition for companies. "You need very skilled people to do threat hunting, or you outsource threat hunting, or you have to have extremely sophisticated tools to do threat hunting—sometimes you do all those things," said Rishi Bhargava, co-founder of Demisto, a maker of incident-response tools. "So I think what I'm seeing is that it is a function of how mature a company is in their security thinking and security strategies."
Regardless, security teams need to just jump in and start to overcome the two largest hurdles: skills and data, said Mary Karnes Writz, director of product management at HPE Software's ArcSight. Prior to her current role, Writz conducted threat hunting both for HPE and on behalf of its clients.
“You don't need to be a data scientist or a mathematician. It is hard to find the skill sets, and even if you find the skill sets, you might feel a little bit insecure about your ability.”—Mary Karnes Writz, HPE
Carving out the time to hunt down threats is also critical. Most threat hunters have other duties they regularly perform and only hunt when warranted by an incident. Companies that have security operations centers—with a group of skilled analysts—may be able to use that model, but smaller companies that have a small team, which is often the IT team as well, will have more difficulty.
“If you are the lone security guy in the company and you are handling security as part of your duties, you are not going to have time to hunt.”—Ely Kahn, co-founder and VP at Sqrrl
2. Go beyond reacting to incidents
Most companies start threat hunting because of an incident. Alerts generated from a variety of security tools—such as security information and event management (SIEM) systems or firewall logs—often are the impetus to launch an investigation. More than 87% of companies would initiate a threat hunt because of an alert, according to the SANS Institute.
Yet an incident investigation is a very narrow case of threat hunting—and some security experts don't consider it threat hunting at all. Threat hunting often starts with a hypothesis, where a hunter creates a scenario based on threat intelligence, data analytics, or an anomaly, said Sqrrl's Kahn. "Hunting really begins with a hypothesis," he said. "It can be inspired by alerts or advanced analytics, but the start is really a what-if question."
A company could initiate a threat hunt if intelligence indicated that its industry had been targeted by a specific attack or attacker, said Kahn, who formerly worked as the director of cyber security as part of the National Security staff in the White House. In addition, viewed through the lens of the Cyber Kill Chain or Attack Chain—two ways of analyzing attackers' actions—an initial alert could lead to hunters investigating whether the attack led to further compromise, say through lateral movement.
"Once they get inside the network, they need to get access to central data and passwords in the network—they hop from machine to machine," Kahn said. "Searching for signs of that movement is a classic way to hunt."
Other signs of compromise could be anomalies in network or application performance—traditionally the purview of the information-technology group, said Eric Ogren, a senior analyst with the 451 Group:
“Part of the security team will always man the ramparts and prevent the attacks. But having mobile forces that can chase down threats as necessary is extremely important.”
3. Use the right tools
Hunting for threats also requires the right tools. About 90% of companies use existing tools to help hunt for threats, with another 61% using customizable tools, such as scripts, according to the SANS Institute report.
Yet trying to modify current tools to aid in hunting can slow down the process, said Gerritz. "The No.1 thing that people do wrong is underestimating how much effort and the level of skills that is required to hunt with their current toolset," he said. "If you are just aggregating data and expecting your hunter to go through that bucket and find threats, you are wasting a lot of time."
On one hand, security teams want a hunt solution that is not noisy and does not have any false positives. Yet the tools should allow them to easily access the raw data, if they need to. Abstracting the information too much can help hide attackers. It is more important to not miss anything when expanding your search, he said. "You want the system to be a force multiplier to help the analyst, not replace the analyst."
Threat hunting tools remain a nascent market. Only about a quarter of survey respondents used third-party tools from a threat-hunting vendor, according to the SANS Institute.
4. Operationalize what you learn
To move from ad hoc threat hunting to continuous threat hunting, companies need to incorporate the lessons learned while threat hunting into their process and tools. If a threat hunting team has to continue to repeat the same steps every time it hunts, it is a waste of time, said HPE's Writz. "Once you have figured out a good question to ask, one that is producing fruitful results, you don't want to be asking that all the time. By the third time you ask it, it should be operationalized."
Initially, the learned process can be added into a playbook. Such formalized processes are key for companies to incorporate lessons into their incident-response and hunting processes. Security teams need to ensure that playbooks are about process and not about tools, said Demisto's Bhargava.
"Playbooks should not be specific to tools—they need to be general in nature," he said. "If you start to tie your playbooks to a tool, then you have to change your playbooks when you change your tools."
The next step is to automate any best practices. The ability to automate can help speed up threat hunting and make it less likely that telltale evidence of a compromise is missed. "You need the right tools in place, and you need the process," said Writz. "And by automating, you can make sure certain parts of the process are optimized."
Finally, companies should operationalize threat hunting lessons by using incidents as training material. "A lot of it is really about having a program to transfer knowledge from your more experienced hunters to your less experienced hunters," said Sqrrl's Kahn. "That is certainly what we have seen in the top-performing organizations."
Getting real counts on getting started
Threat hunting is a security process that is undergoing significant change. While many companies' management teams claim that they are doing threat hunting, security teams are often less certain about whether their activities constitute formalized threat hunting.
"When I talk to CISOs, they all say that they are doing threat hunting, but everyone behind the executive feels like they are not very far along," Writz said. "But don't worry about not being far along. Everyone is in the same boat."
Keep learning
Learn from your SecOps peers with TechBeacon's State of SecOps 2021 Guide. Plus: Download the CyberRes 2021 State of Security Operations.
Get a handle on SecOps tooling with TechBeacon's Guide, which includes the GigaOm Radar for SIEM.
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed on cyber resilience with TechBeacon's Guide. Plus: Take the Cyber Resilience Assessment.
Put it all into action with TechBeacon's Guide to a Modern Security Operations Center.
