You are here

You are here

4 lessons from SOC metrics: What your SecOps team needs to know

Rob Lemos Writer and analyst

When Dan Basile became executive director of the security operations center (SOC) of the Texas A&M University System, he wanted to make sure the organization tracked as many metrics as possible.

By creating measures around every aspect of the SOC's operation, Basile aimed to find the most telling ways to determine whether the security team was doing its job of protecting all of the information and hardware at 11 universities serving the more than 150,000 students that make up the university system.

Problems quickly popped up. Grading analysts on how quickly they closed issues could result in employees focused on only the easiest issues. Ranking them on how often an issue turned out to be a significant incident could punish investigation.

The first lesson for Basile was that metrics can't become the entire focus.

"Metrics are hard—finding metrics that are meaningful is even harder, but you have to start somewhere. There is a difficult balance between those metrics that make it look like you are really busy and those that show, hey, we are doing good work here."
Dan Basile

Finding metrics to measure the effectiveness of security-operations teams is key to improving businesses' ability to improve their security. In Micro Focus' 2019 State of Security Operations, researchers found that SOCs tend to lack clarity in their goals and have problems documenting and executing repeatable processes.

Both of these issues can be highlighted and made more actionable through metrics, said Preston Wheiler, product marketing manager for Micro Focus ArcSight.

"A lot of the same problems are persisting year after year, and SOCs have not really tamed those challenges."
Preston Wheiler

Here are four lessons gleaned from SOC metrics that SecOps teams should heed.

1. Metrics have a downside

First, a warning. While the adage, "You can't manage what you don't measure," continues to be true, the downside for companies using metrics is that workers will change their behavior to maximize their scores on performance benchmarks.

While metrics can give management a way to see what is happening in their SOCs, they should avoid the "call-center problem," Wheiler said. When call centers measure how quickly employees complete each call, they are not measuring how well workers complete their tasks, but how quickly. The result can be performance that skews work toward worse performance on the actual company goals—employees try to get off the phone quickly, rather than solve a customers' problems.

The same considerations go for analysts in the security operations center, said Forrester analyst Joseph Blankenship.

"Some of those things can be counterproductive. If I'm trying to maximize the number of incidents I'm handling, perhaps I'm doing less due diligence for each one."
Joseph Blankenship

2. Obvious metrics can still be valuable

Despite the potential for problems, companies should still track the most obvious measures of their security operations, while being aware of their shortcomings. The number of incidents triaged by an analyst, the mean time to close each issue, and the number of alerts that are found to be false positives are all valid metrics, as long as companies do not put too much weight into their significance, Blankenship said.

"I don't think that makes the metrics less valuable or the veracity of the security they are measuring less valuable. Just because you are going to get cavities doesn't mean you should stop brushing your teeth. Just because you have a risk does not mean you should not try to measure it."
—Joseph Blankenship

Metrics that focus on the entire security process can be a strong measure of how secure a SOC is making the company. Attacker dwell time, for example, can only be determined after a full investigation, but it can give management a clearer picture of the overall effectiveness of the security team.

3. Gamifying leads to better statistics

Texas A&M University System's Basile has created statistics on analysts, sort of "baseball cards" that give a picture of their most significant contributions to the security of the university system. The analyst cards can be thrown up on the SOC monitors and give the analysts data on what the others are doing, but also spur them to compete on the metrics that mean the most to them.

One analyst might focus on closing issues quickly, while another focuses on deeper investigations. By including a variety of metrics, SOC team leaders can entice more competition, Basile said

"Create metrics around everything. One analyst might be closing a ton of incidents—maybe that is their thing. Another analyst might spend more time on investigations, but he or she is taking the more difficult investigations. Thirty minutes might not be enough time to do a good investigation."
—Dan Basile

4. Coverage measures progress

Many security teams are transforming operations—for example, moving to automated testing, adopting comprehensive data analysis, or implementing metrics. An important metric is to measure progress of such initiatives, especially the degree to which a security effort has covered processes, servers or software.

While other metrics can help analysts focus on specific aspects of security operations, making project plans and then measuring progress toward those plans can help motivate workers toward transformation, Forrester's Blankenship said.

"If you don't have a plan, everything is ad hoc."
—Joseph Blankenship

Think bigger picture for your SecOps

Metrics are critical to creating a mature SOC, but they should not be the focus of any transformation, because their is no single metric—or handful of measurements—that give a complete picture of your security operations.

Keep learning

Read more articles about: SecurityInformation Security