You are here

You are here

Why DevOps is essential for security at scale

Shawn Masters Vice President of Solutions Engineering, Novetta

DevOps is all about going faster. This is obviously important for speeding up release cycles. But DevOps is also crucial to achieving security at scale.

Back when enterprises only needed a few on-premises servers, a small team could maintain security with ad hoc fixes.

Today, enterprises rely on hundreds or even thousands of applications, services, and security tools scattered across data centers around the world. The scale of these systems makes ad hoc methods cumbersome and ineffectual.

DevOps offers a better alternative. Here are a few tips on using DevOps to defend the enterprise.


Tools are a good place to start. Ten years ago a good systems admin could make large sets of patches using parallel SSH or some simple scripts. Now, the same admin needs to manage and deploy configurations, updates, and changes using more advanced toolsets like Puppet, Chef, or SaltStack.

Even the configurations and tools to manage them are advanced enough to benefit from source control and automation of their own. This is where the more modern formality of DevOps enters not just for running operations, but also for providing secure infrastructure and services.


Collaboration has become necessary as our systems have grown in complexity. To fully understand, manipulate, and operate our systems at their fullest potential, we need the wisdom of the crowd. Rarely does any one individual understand all aspects of any single system, let alone multiple systems and how they interact.

If you are ever in doubt about the growing level of system complexity, search the Internet for JVM tuning options or look at the 14,384 words in the Linux mount command's man page. Both are fundamental technologies that blend into the ubiquitous background, but have incredible complexity in their settings.

In the area of security, complexity is exponentially more challenging, because security problems are not part of the standard concept of operation. No one configures a system to be insecure. Instead, security gaps arise from misconfigurations or poor implementations.

The good news is that collaboration both within an enterprise and with the larger global community can address these problems. Information sharing between dev, ops, test, and security can shorten diagnostic time, reduce work duplication, expedite implementations, and mitigate unintended countermeasures.

Collaboration beyond the corporate walls means that groups running similar systems can pool their knowledge to better understand attacks and determine solutions. When collaboration reaches a global scale, problems and solutions are often found and documented elsewhere before your company encounters them.


Enterprises need to respond to threats as they occur and solve weak security positions across systems and networks. This can be a major challenge for multistep attacks that use a foothold from an earlier compromise, or a weak link in a service that is being exploited by an external attacker.

The keys to adapting to such attacks are community collaboration and DevOps empowerment. By working with a larger community, you can quickly identify the weaknesses in your system. And by giving your DevOps team authority to make changes, you can quickly fix the problems.

A classic example comes from late 1988. A first of its kind, the Morris worm started to deny service to the early-stage Internet. The worm infected Unix systems running Sendmail and RSH, using up network resources and rendering many systems unresponsive.

Once it was clear there was a worm spreading, operators started to pass analysis and insight about how it operated via Bitnet, UUCP, and phone calls. Noticing that the worm grabbed and compiled a small C payload to be portable across systems, many DevOps teams of the day renamed their C compiler until they could configure RSH and patch Sendmail.

Other patches and fixes started to appear soon after that, helping solidify defenses and clean up the infected systems. If these systems had to wait for vendor patches to be tested and delivered, or for formal review processes, it could have been days or even weeks before systems returned to normal, and the overall economic damage would have been much greater.

In the reaction to the Morris worm we see how even relatively primitive collaborative communities disseminated diagnostics and mitigation strategies faster than the worm itself. The most adaptable operational organizations that could make quick changes fared the best and had the least interruptions. Where adaption required larger consensus or higher approval, the reaction was often too late to matter.

In today's environment, with IT systems being critical to our physical and financial world, the impact of something like the Morris worm is exponentially greater. Disruption for power generation, distribution, water supplies, or communications could be a component of a terrorist attack. Denial of services could be used by criminal elements to negatively impact a company's stock value.

Open source

Open source plays even more of a role in the security portion of DevOps than it does elsewhere. Using open source and frameworks, the DevOps community can quickly make adjustments, patches, and enhancements to critical infrastructure and address problems at the root of the cause.

By working within the community one can use best practices, patches, and concepts from others, to make adjustments to enterprise security with the highest probability of success. The continual integration of software and configuration changes provides a mechanism where testing happens quickly and offers alternative implementations in minutes, ensuring a successful outcome.

Collaborations in DevOps communities will inform practitioners when those changes are not viable and what alternate paths may be available. Having the source to modify and open standards to implement against, a good operator can adapt in an almost unlimited manner to new and emerging threats.

By using and extending open source software for automation, problem-solving and collaboration, the DevOps community can react and handle situations as they emerge, often before problems occur within their enterprises. While this pace and responsiveness is great for productivity, it is absolutely essential in the growing arms race for securing services.

DevOps makes security better

The DevOps community can adapt and implement new protections, policies, and practices as required and apply patches and modifications in a rolling implementation as the information becomes available to them. By sharing details, code changes, and security information, solutions can move through DevOps communities as fast as the use of exploits, which is key for defending systems.

Keep learning

Read more articles about: DevOpsSecure DevOps