6 DevSecOps best practices: Automate early and often

Organizations that want to integrate security into their DevOps pipelines should adopt tools and practices that unite application development, IT operations, and security teams under a common DevSecOps rubric.

The goal is to make security part of the software development workflow, rather than bolting it on later in the cycle, as has been the case with waterfall development models.

This shift is upending traditional notions of how, when, and where security controls should be integrated into software, and is challenging once-siloed groups to find ways to work together to deliver rapid, but secure code releases.

"DevSecOps has seen a steady rise in popularity," said Ryan O'Leary, chief security research officer at WhiteHat Security. Companies that have combined development and operations teams under a DevOps model in recent years have been generally successful in releasing code at a much faster rate.

But the trend has heightened the need for security to be integrated into the process because the faster you release code, the faster vulnerabilities can be released as well, he said.

Here are key best practices for organizations seeking to implement DevSecOps.

Application Security Research Update: The State of App Sec in 2018

1. Make automation your friend

Speed is one of the main tenets of DevOps. In a continuous integration and continuous deployment (CI/CD) environment, how quickly you can get code out the door and into production trumps almost everything else. For security to be part of this workflow, it needs to be automated, said Chris Romeo, CEO, principal consultant, and co-founder of Security Journey.

Security controls and tests need to be embedded early and everywhere in the development lifecycle, and they need to happen in an automated fashion because organizations are pushing new versions of code into production 50 times per day for a single app, he said.

"With DevOps, you have to move super fast. There can be no 'manual' in that process. If you don't have automation, you'll never be successful."
Chris Romeo

Automation has become a key DevSecOps characteristic in organizations with highly mature DevOps practices. About 40% of respondents in a Sonatype survey of nearly 2,300 IT professionals earlier this year ran automated security tests throughout the entire development lifecycle. This is instead of what happens with waterfall development models, when automated security tests run just before production.

A growing number of tools with a range of capabilities has become available for doing security analysis and testing throughout the software development lifecycle, from source-code analysis through integration and post-deployment monitoring. These include Checkmarx, Splunk, Contrast Security, Sonatype, Tanium, InSpec, FireEye, and Metasploit.

Automate thoughtfully

When automating security testing, tread carefully, Romeo said. If you do static application security testing (SAST) on nightly builds, for instance, make sure you scan only for things of interest in the changes to your code that have been committed for that day.

Trying to run automated scans on your entire application source code each day can consume a lot of time and break your ability to keep up with daily changes.

Also consider embedding automated dynamic application security testing (DAST) into your software development lifecycle. Unlike static analysis, which focuses on finding potential security issues in the code itself, DAST looks for vulnerabilities in real time, while the application runs.

Automate DAST scans and run those against recent or new code changes to catch security vulnerabilities listed on the Open Web Application Security Project's (OWASP) list of the most common flaws, such as a SQL injection errors, that you might have missed during a static analysis of your application code, Romeo said. You need both in an automated fashion because you never know what you might miss with just one set of tests.

Adding automated security analysis within CI platforms can limit the introduction of vulnerable code earlier in the software development lifecycle, said Manish Gupta, co-founder and CEO of ShiftLeft.

The automation can be coupled with lightweight embedded agents to provide correlated runtime analysis about issues detected from your automated security analysis. Taken in tandem, these approaches make it easier for developers to prioritize what code problems they need to fix, Gupta said.

2. Check your code dependencies

Despite growing concerns about the risks of using third-party software components, enterprises are using more open-source software in applications, not less, according to a survey Black Duck Software conducted early in 2017.

A separate audit that the company performed on over 1,000 commercial applications showed that 96% of them included open-source components. More than 6 in 10 of the applications contained known security vulnerabilities in those components, and some had been there for as long as four years. Despite that, just 27% of respondents said they had processes for automatic identification and remediation tracking for known flaws in open-source software.

Understanding open-source use is key to wider adoption of DevSecOps practices, Gupta said. "The cloud has fueled innovation in unprecedented ways, enabling companies to meet customer requirements faster. This pace of innovation has also brought about the increased use of open-source software to assemble applications, rather than developing them from scratch."

Developers often don't have the time to review code in their open-source libraries or read the documentation, Gupta said, so automated processes for managing open-source and third-party components are a fundamental requirement for DevSecOps. You need to know if your open-source usage is causing contextual and other vulnerabilities in your code, and what impact those vulnerabilities might have on dependent code.

Code dependency checks are fundamental to DevSecOps, and utilities such as the OWASP Dependency-Check can help ensure that you do not use code with known vulnerabilities in your software, Security Journey's Romeo said.

The OWASP utility works by scanning your code and dependent open-source component libraries to see if they contain any key OWASP flaws. It works against a constantly updated database of all known vulnerabilities in open-source software. 

3. Don't bite off more than others can chew

SAST tools allow developers to scan code as they write it so they can receive instant feedback on issues that might cause security problems. The tools help developers identify and remediate potential security vulnerabilities as part of the normal workflow, and are therefore should be an essential component of your DevSecOps practices.

The key when introducing such tools, though, is to think small. Often, when a security team implements a static testing tool in the CI/CD chain, the team tends to turn on checks for a whole slew of security issues and ends up only creating problems for developers, Romeo said.

Instead, it is much better to turn on one or two security checks at a time and get your developers used to the idea of having security rules incorporated into their workflow.

For example, when introducing a SAST tool in development, you could begin by turning on only the rules for catching SQL injection errors. Once your developers see how the tool helps them catch errors during coding, they are much more likely to want to work with it. "You need to build trust in the tool before you turn on more and more rules."

When building out a DevSecOps capability, break things down into manageable chunks. "The best way to approach this is to look at the holistic set of all the different activities that go into a DevSecOps build," Romeo said. Choose one to start with and prove it works before moving to the next thing.

Security professionals who go in and disrupt things will just slow things down and experience conflicts with developers, he said. Think small in the beginning, and put some success behind you before taking the next step.

"What you are dealing with here is a different culture."
—Romeo

4. Some tools are more useful than others

Many of the tools required to insert security into agile DevOps are still emerging, so keep a few key considerations in mind when shopping for them.

Integration, for instance, is critical, said O'Leary of WhiteHat Security. "Security products need to be able to integrate into the development pipeline and enable both the development and security team to work together instead of just throwing things over the fence to each other," he said.

If developers must go out of their way to initiate scans, there's a good chance they will abandon the scanning tool. "A security product needs to make it easy for developers to quickly initiate scans and get results without having to leave their existing toolset," O'Leary said.

The other key requirements are speed and accuracy. Security tools should work fast. But false positives can be an absolute killer in a DevOps environment, O'Leary cautioned.

A tool that requires a developer or a security engineer to take time out to verify the findings of a scan is of little help. The results that your tools generate need to be fast, accurate, and immediately actionable.

"False positives are the death of rapid development."
Ryan O'Leary

Security tools designed for waterfall development models are of little use in a DevSecOps world. You need tools that can help your developers identify and prioritize vulnerabilities as they are writing software, ShiftLeft's Gupta said. The identification of code vulnerabilities needs to be based on an understanding of the software itself, as opposed to matching it against signatures.

Gupta had a few suggestions for what organizations need to think about when shopping for such tools. Any tools you choose should protect you against not just known vulnerabilities but unknown threats and key OWASP Top 10 risks.

The tools should also help you identify and address risks in the open-source software components you use, and identify issues in a manner that helps you reduce the mean time to resolution.

"The challenge of widespread DevSecOps adoption is to come up with tools and processes that do not create a drag on release and deployment cycles."
Manish Gupta

5. Threat modeling is hard, but do it anyway

The SANS Institute recommends doing threat modeling and risk assessments before you shift to DevSecOps. A threat modeling exercise can help your security organization get a better idea of threats to your assets, the types and sensitivities of your assets, the existing controls for protecting those assets, and any gaps in your controls that need to be addressed.

Such assessments can help identify flaws in the architecture and design of your applications that other security approaches might have missed.

Conducting threat modeling in a DevOps environment can be challenging because of the notion that it can slow down the velocity of a CI/CD environment, Romeo said.

"DevOps is more about designing on the fly. In fact, there are some DevOps folks who don't do any design at all and say the code is their design. That is dangerous."
—Romeo

You can't automate the threat-modeling processes in the same way you can for almost every other facet of DevOps. But threat modeling is still crucial for the overall success of your DevOps efforts because it gets your developers to think of their software from the perspective of an attacker, Romeo said.

6. Train your developers on secure coding

You'll face multiple challenges when adopting DevSecOps. One of the biggest is simply getting buy-in from your stakeholders. Development, security, and operations teams often operate in their own silos, and have their own agendas and tasks, O'Leary said. "Making a case to the company that these organizations can be combined is sometimes a bit of a struggle."

Getting the investment and time needed to train the development team on secure coding is another big challenge. "Developers often don’t know they’re coding in an insecure way. It still is not taught very often, and is not a priority for the development team," O'Leary said. Investments have to be made in training developers on security, he said.

"The best way to prevent a vulnerability is to never code it in the first place."
—O'Leary

Many DevSecOps practices and tools are still emerging, and there is still little consensus on the definition of DevSecOps today. But it's evident that in a world of continuous integration and rapid release cycles, you can't ignore application security any longer. Security is other common theme, said Gupta.

"It is increasingly becoming a shared responsibility, built upon the idea that everyone in the software development lifecycle is responsible for it."

Topics: Security