Table of Contents
Overview: Why app sec is in the hot seat
The software landscape is changing rapidly as agile development methodologies and DevOps become more widespread, a greater share of application code is now derived from open-source projects and components, and more applications are developed and deployed in the cloud.
It should come as no surprise, then, that application security has become more complex. While DevOps and other agile methodologies break software development into more manageable pieces, the process requires that developers take on a greater role in securing their code. To improve application security organizations must ensure that open-source components are tracked and managed and that they protect cloud-native applications both during the development process and in deployed software.
Architectural decisions, such as whether or not to use microservices, can also wreak app security chaos. Dan Cornell, CTO and co-founder of the software security consultancy Denim Group, said there is a joke about microservices.
"If you couldn't make one giant, monolithic ball of code run right, how is breaking it into 30 smaller pieces going to make your problems any easier to solve?"
—Dan Cornell
Because application architectures are becoming more complex and you have more pieces that are interacting in different ways, security must take that into account.
As application infrastructure moves away from monolithic servers and software to microservices and containers in the cloud—and development is broken up into smaller, more agile teams—the discipline of application security has become more complex. While each development task has become simpler, the sheer number of components in a modern application means that catching vulnerabilities in software is both more difficult and more important.
Key trends affecting app sec teams
For many years, security revolved around two main security silos: static analysis security tools (SAST) and dynamic analysis security tools (DAST). In the past decade, however, runtime application self-protection (RASP) and interactive application security testing (IAST) have become more common.
Now WAFs, container security testing, configuration management, and improved security pattern recognition in development environments are all part of potential tool sets for application security groups. "You have more pieces that are interacting in different ways, and you have different kinds of pieces," said the Denim Group's Cornell.
The focus in application security has evolved from merely finding vulnerabilities in code before they are exploited to improving code quality, testing configurations before deployment, and creating manageable development lifecycles that lower risks.
Application security starts with managing the software supply chain and the software bill of materials, integrating security tools into development to catch insecure coding practices and inform developers, and reaching out to deployment to check configurations and gather information about attacks.
Laurence Bierner, director of application security for GitLab, a development tools and services firm, said that application security has adopted the broader stance of application reliability and quality. If you have what is truly considered to be quality code, then it is secure, well written, easy to maintain, manageable, flexible, and extensible, he said.
"You can't have quality code that is not secure; it all goes hand in hand. All these really great software engineering concepts. You've thought those out; you've designed the code around them."
—Laurence Bierner
Your security mileage should vary
Security has become more difficult for companies that try to apply the same security precepts from large development projects to today's more agile development lifecycles.
Companies should start any evaluation of their application security process and capabilities by assessing their people—especially developers. Creating a culture of quality software development and teaching the proper way to secure code is key to a robust application security process.
Part of this is to shift many security checks into the development process and to do so as early in the dev cycle as possible—shifting left, as it's known—putting the "sec" in DevSecOps, said Kevin Heckel, cyber application offering leader for the consultancy Deloitte.
"The further we can move security left, the better. Not as a punishment, but as a cultural norm."
—Kevin Heckel
The goal is to find bugs when they are easy to fix, because finding bugs earlier pays off: Studies have estimated that bugs caught during design or coding cost anywhere from one-third to one-hundredth what it costs to fix a bug found in production. And that is not even considering the cost of security vulnerabilities that could lead to a breach.
Securing open-source and legacy apps
The software supply chain has also become a growing part of application security. Because so much of development is now based on open-source frameworks and components, code quality starts with vetting the open-source projects that your company uses.
In its 2020 Open Source Security and Risk Analysis report (PDF), software component analysis (SCA) firm Synopsys found that almost half of codebases—of the more than 1,200 applications the company audited in 2019—contained high-risk vulnerabilities. Some 82% of the codebases included components that were more than four years out of date.
Legacy code is one part of the problem. Deloitte, for example, has found that most of its clients do not have a robust application security process and have significant legacy code that, its clients believe, cannot be touched without running the risk of hobbling the business.
In addition, many companies have internal or back-office applications to which they apply a lower standard of security and quality checks.
"We see a lot more clients not touching the ERP clients or the legacy side of the house," Deloitte's Heckel said. "But it needs to be subject to DevSecOps and your overall environment as well."
No longer 'the department of no'
Overall, application security is evolving from a mindset of "Don't do this" to "Find a way to do this securely," said GitLab's Bierner.
"We should be enablers, and not the security teams that say no; we have to move away from that," he said. "A lot of the security mindset tends to go in that direction when they are building out the program, and that is exactly the wrong way to go."
App sec tooling in 2021: Best practices
As the definition of application security has broadened to include the entire software supply chain, all the way to deployment and operations, the classes of application security tools have broadened as well. A decade ago, application security professionals may have run a static analysis of their source code to find software defects and vulnerabilities, engaged a penetration tester to use dynamic analysis tools, and protected the application with a simple firewall.
Today, these tests are on-demand and often run on a daily or weekly cadence, are tailored for the specific environment, and will often block code check-in unless the code passes. The classes of tools include:
- Software composition analysis (SCA) tools, which help manage open-source components and container images
- More in-depth static analysis security testing (SAST) tools, increasingly augmented by fast security linters that give developers advice as they code
- Dynamic analysis security testing (DAST) and interactive analysis security testing (IAST) tools, which allow application security professionals to analyze running applications for flaws
- Orchestration and automated deployment pipeline tools that conduct tests and configuration checks to ensure that applications are securely deployed to the cloud
- WAFs, RASP, and API security tools that organizations increasingly deploy to ensure that applications do not expose business logic flaws
- Penetration testing and software architecture review tools used for exploratory analysis that can uncover software security problems that other tools may not find
Over time, most of the tools have migrated to the cloud for efficiency and for quick iterations in capabilities. Unless there is a compelling reason to not use cloud-based tools or services, most organizations can gain flexibility and resilience by focusing on cloud-native development and adopting tools that support that infrastructure.
Companies that cannot move some applications to the cloud due to compliance requirements or that are wary of exposing their source code to cloud-based tests should focus on a hybrid approach.
Adopt a software-security maturity model
Do not start by focusing on tools. Instead, application security professionals and their development counterparts should think about what their process will be to create secure software.
Measuring progress in application security can be difficult, but it is necessary as a tool to analyze an organization's development capabilities. For the past decade, software maturity models have become the best way to find the weaknesses in a company's security software development lifecycle. Maturity models typically evaluate a company's progress in key areas and let you compare your organization against others in your industry.
"No company is out there doing everything right. Based on their development process and exposure, they should evaluate where they need to drive their development and security processes."
—Kevin Heckel
Maturity models typically focus on specific areas, such as education, threat modeling, architecture design, secure software building and deployment capabilities, vulnerability and defect management, and incident response capabilities.
The two major maturity models are the Building Security in Maturity Model (BSIMM) and the OWASP Software Assurance Maturity Model (OSAMM), but other frameworks exist. These include the Application Security Maturity (ASM) model and the Capability Maturity Model Integration (CMMI) Cybermaturity Platform.
Maturity models can also help companies define new capabilities that they need to develop. In its last iteration, for example, the BSIMM group added software-defined lifecycle governance, the monitoring of software-defined asset creation, and automated verification of software-defined infrastructure to their model. These additions were added because some companies wanted to speed up security maturity to catch up with how quickly the development organization was delivering new functionality.
The Enterprise Strategy Group, which has its own Cyber Security Maturity Model, found that the majority of organizations leading in cybersecurity look at the security team as an enabler of business, rather than as a roadblock.
Those leading companies were also much more likely to exceed revenue goals by a significant amount (7% or more), according to the ESG report "The Relationship Between Security Maturity and Business Enablement." The more mature security teams had better success blocking data loss from both insider and external threats and preventing denial-of-service attacks, the report found.
DevOps is not just a buzzword
As companies start to consider tools, they need to fit them within their chosen agile development methodology.
Most companies have either adopted or are planning to adopt DevOps or similar programming practices, according to Google's annual DevOps Research & Assessment (DORA) report. Only 12% are currently low performers in four key software metrics, an improvement from 15% the prior year.
The metrics do not include security, but are closely related to an application security program: deployment frequency, lead time for changes, time-to-restore service, and change failure rate.
Low-performing organizations deployed software once a month or less frequently, required at least a month to process change requests, took at least a week to restore service, and had a change failure rate of 46% to 60%.
Elite performers could deploy on demand, required less than a day to consider changes, needed less than an hour to restore service, and encountered change failures less than 15% of the time. In 2019, elite performers made up 20% of organizations, up from only 7% in 2018.
Companies need to figure out their process and refine the development cycle before considering tools, said GitLab's Bierner.
"One of the big mistakes that I see application security programs make in the past, and honestly that I have made, is looking first at the technology before the process. That usually ends up being a costly mistake and then you are stuck trying to make technology work with a process that it was not designed to work with."
—Laurence Bierner
Decide whether integration or accuracy is more important
The first choice for many companies is whether to focus on finding the tool with the best accuracy or ensuring the tools will work well together. DevSecOps, which focuses on adding security to the software development lifecycle, typically focuses on integration. If the tool can be integrated into the development pipeline in a non-intrusive way, then it makes sense to adopt it.
Best-in-class tools that add depth need to be considered for applications that require more analysis or as a more detailed check on the application at a less frequent cadence than in-editor linters and other tools. In addition, if they are moved out of the daily DevOps cycle and applied at certain checkpoints, such as code check-in, they can continue to be part of an agile development cycle.
"Getting the best quality analysis is very important," said Matt Stanchek, Fortify-on-Demand architect at Micro Focus. Even if the tool is easy to integrate, if you are getting poor results, then "dealing with verification is going to burn a lot of time," he said.
Companies should take a holistic approach to application risks and not stop at a single scan of the code. Continuous testing and monitoring are important, and finding tools that can support that strategy is critical.
The only constant is change
A variety of software development technologies could change how companies create code. Automated systems using machine learning to find secure coding patterns could result in machine-generated code.
The Massachusetts Institute of Technology, Intel, and the Georgia Institute of Technology have teamed up to create a process called "machine inferred code similarity" (MISIM). The technology attempts to understand what code does and find the best way to code a certain task.
OpenAI's GPT-3 algorithm, better known for creating human-like writings given an initial sentence or two, can create code from an English description of a graphical element. And DeepCode.ai learns about good and bad software from pull requests and commits to recommend secure coding solutions.
Developers and application security professionals should evaluate current tools with an eye toward the future. While automation and pattern recognition currently uncover defects in code and applications created by humans, the industry will likely move to a point where a great deal of code is generated by systems.
How to build a modern app sec toolbox
There is no perfect set of tools for application security. Companies have different requirements depending on their infrastructure, development processes, degree of cloud usage, deployment targets, and regulatory requirements, to name just a few factors.
When considering security tools, there are five major considerations. The tool should provide:
- Suitability, by including all necessary features and applying to the scope of the work, whether on premises or in the cloud
- Automation, by delivering machine-driven processes using software-defined rules that are understood by the developers and the application security team
- Integration, by working with other tools and technology incorporated into the development pipeline and delivering results that match the expected input of the next tool in the chain
- Accuracy, by generating results that have a low number of false positives and false negatives
- Visibility, by producing accurate alerts and leveraging the ability to quickly analyze results
Focus on security tools only after the application security team discusses the process with developers; the idea is to first determine the most appropriate software development pipeline that incorporates security as much as possible.
Micro Focus' Stanchek said you must understand what applications you have and how they are deployed and what languages they are written in.
"Know what you have in your environment. Without that, you cannot select the right tools, the right testing processes, or the right folks to hire for pen-testing."
—Matt Stanchek
Security tools for developers
Incorporating security into the software development lifecycle should focus on finding points at which developers can easily fix errors and, at the same time, have security lessons reinforced. Security tools targeting developers should minimize disruption and focus on education.
"Rather than send all the developers to training—because many of them have no context for training—bring the training to them," said the Denim Group's Cornell.
In the past, many programs would run a big analysis—either static code analysis or dynamic testing—to collect a large number of results of varying quality, and then drop a massive report on the developer's desk.
"That does not help anyone, and it starts a bad dynamic in the relationship between developers and the security team," said GitLab's Bierner. Getting developers in the product organization within your company to accept accountability for designing and developing secure software "requires working closely with each other" and focusing on developers' needs, he said.
Software composition analysis (SCA)
Agile programming requires the use of open-source components to maintain the speed of development, standardization, and functionality required in modern software development, many experts believe.
SCA can help with this by delivering two major benefits: Development teams can quickly ascertain the level of security scrutiny an open-source component receives from its project managers, and, when a vulnerability is discovered, the tools can hunt down every affected component in the software supply chain.
There may be no discovered vulnerabilities in an open-source library, but if one is discovered after deployment "and you have embedded it in 50 different things, you need to be able to retroactively find all those components," said Rik Turner, principal analysis for cybersecurity at Omdia, a technology consultancy.
Security linters
While linters have their origins as command-line tools used in old development environments, the modern linter is a runtime service that can check code for correctness, policy violations, and security.
Because they flag software defects and vulnerabilities while the developer is coding, linters are a great way to shift left some of the responsibilities for security to developers.
However, companies need to be aware of their shortcomings: Linters typically enforce secure patterns of developments and look for common software flaws, such as cross-site scripting and perhaps SQL injection, but will not catch more complex vulnerabilities.
"I think the linter now is that next step in lightweight, fast security evaluation of software. We still need the heavyweight analysis of static scanners, but the quick turnaround, fast analysis of using a linter has its place right now in the CI/CD pipeline for sure."
—Laurence Bierner
Unit tests
Test-driven development focuses on requiring that code and configuration files pass important sanity checks as part of the development and deployment pipeline. Unit tests are the simplest checks. They require little effort on the part of the developer to understand and hold developers responsible for making sure their code meets a minimum level of functionality, quality, and security.
Unit tests can implement fairly complex security checks—even incorporating some static code analysis, for example—as long as the results are fully explained to the developer and corrective actions are recommended.
Tools for the application security team
Another set of tools requires management by application security teams, so that results can be tailored to the company's development process to minimize disruption to the software development lifecycle.
Static analysis security testing (SAST)
To work with a DevOps software development lifecycle, in-depth analysis tools such as SAST need to be integrated in the proper place. Extending coverage to a new codebase should be done in focused steps to minimize the initial shock of returning a large number of results.
Application security engineers should work side by side with a development team member to tackle results in pair programming exercises, an effort that will help both developers and security professionals understand the other's role and result in cross-training.
Finally, focusing on the most accurate results can help minimize developer frustration with the process, said Micro Focus' Stanchek. There are many opportunities to deploy those tools, and "teams need to understand the different places in the process—check-in, build, and deployment—where they can best fit."
Dynamic analysis security testing (DAST)
Scanning application and APIs for vulnerabilities, also known as "black-box testing," can catch a variety of issues, such as cross-site scripting and SQL injection. DAST can be used after a codebase is feature-complete—at the testing and staging step—to detect exploitable misconfigurations and vulnerabilities.
While SAST will often find patterns that suggest an exploitable flaw, dynamic testing leads to the discovery of vulnerabilities that are actually exploitable.
Interactive analysis security testing (IAST)
Instrumenting applications through code libraries, or with an agent running on the server or as part of a container platform, can give application security teams more insight into the state of an application while it is being attacked.
Known as IAST, or "glass-box" testing, the technology can catch vulnerabilities that SAST and DAST might otherwise miss.
Because IAST can help security teams discover issues during the testing and staging steps of the software development lifecycle, the technology can reduce the cost of vulnerabilities that would otherwise not be found until an application is deployed. With automated tests and staging, IAST can find potential issues immediately after code is checked in and recompiled.
Security tools for operations
When security tools are deployed in operations, they not only protect against in-the-wild attacks, but can also collect data on the threats to and the performance of the application in the real world. Such data can help a company create threat models based on actual attacks that can help red teams tailor future tests.
While analyzing actual attacks can be time-consuming, a lightweight approach can still provide significant benefits, said GitLab's Bierner. "It is really part of a mature security program as a whole. Threat modeling—and the data gathered at runtime—can be used across your entire application security program."
WAFs: Not just blocking
A well-managed web application firewall (WAF) can help provide an extra layer of defense against attacks. In addition, from an operational perspective, WAFs give businesses the ability to apply "virtual patches"—rules that block attacks for a known vulnerability as a way to give developers time to create a fix.
"You can mitigate a vulnerability with a single push of a rule, giving your development and product teams the ability to plan a fix for that issue," while you as a security organization "have guaranteed that you have mitigated the risk," Bierner said. "It does not get any better than that, because now the security organization becomes the friend of the product and development teams."
Runtime application self-protection (RASP)
When applications are instrumented during deployment and in production, operations teams gain insight into the real-world attacks that affect the application. By enforcing policies on instrumented application, businesses are granted extra protected against anomalies that may presage an attack.
Finally, unlike web application firewalls, RASP products—similarly to dynamic analysis—are typically low-noise, because they issue alerts only for activity that is anomalous, signifying either an attack or a runtime issue. Gaining such visibility, especially with cloud-deployed infrastructure, is important, because many security teams—81% in one study by AlgoSec—complain of a lack of visibility into the security of cloud-deployed applications.
Prioritizing tools and automation
Companies should integrate only one new tool at a time, unless they have the resources to assign a team to each project. If starting from scratch, companies should focus on adding security to the software development lifecycle. At the basic level, tools that help developers produce more secure code should be a priority, as well as unit tests that check developers' work when they commit code.
Most SAST, DAST, IAST, and RASP tools require dedicated security resources to present actionable results to the developer, said GitLab's Bierner. WAFs can then be added to secure production applications and gain visibility into real-world attacks. "Most critical, choose a tool set that works well together," he said.
Automation is key, and companies such as GitLab are ubiquitous in their automation. The company attempts to "repave" much of its infrastructure every 12 hours, reinstalling clean copies of software and rebuilding their systems.
While a cash-strapped company can initially forgo automation, once a tool or two is added to the software development process, automation is essential. Otherwise, development teams will run the risk of losing the gains from any adoption of DevOps methods.
Finally, companies should also take advantage of another facet of cloud-based tools: the ability to freely try the software before committing to the security tool. There are "a lot of opportunities to deploy those tools," said Micro Focus' Stanchek.
How to create an RFP for security tools
This section aims to help application security teams develop a request for proposal (RFP) for security tools. Because every situation is different, these questions are examples that you should pick and choose based on your own situation.
Before you start
Once the application security team and the development group have worked together to create a plan for improving the security of the company's applications and software pipeline, a little bit of research can help scope the market.
- Using search engines and reports from market analysts, develop a list of products that may suit your needs.
- Keep track of the search terms used to generate the list and any additional criteria that searches turn up.
- Find initial information about pricing for each product to form a baseline estimate of cost.
- Note whether the vendor offers a free evaluation process.
1. Suitability
Not all products will fit the needs of your business. The application security and development teams should develop questions that ensure the product fits their needs. These may include:
- Does the tool support the programming languages and frameworks necessary for our development process?
- Does the tool support both cloud and on-premises applications? Is the security tool itself a cloud service or an on-premises application?
- What types of vulnerabilities and software defects does the product find?
2. Automation and integration
The speed of modern development and the interleaving of so many layers of security and code checks require that tools support an automated pipeline and integrate with other CI/CD tools.
- For which CI/CD pipelines can the product be automated? How easy is the integration?
- How often does the company need to deploy/test software? Has the company adopted, or will adopt, DevOps or some other agile programming methodology?
- Do your developers use a large number of open-source frameworks? If so, which ones does the security tool need to be aware of?
- Where is your company in terms of its application security maturity? What level do you hope to achieve in the near term?
3. Accuracy
The ability of a product to accurately find vulnerabilities and pinpoint the correct portion of code in which the defect occurs is critical.
- Has the vendor benchmarked the product against competitors for false positives and false negatives? Has it produced a confusion matrix?
- What technologies does the product use to produce more accurate results? Does the tool integrate with other vendors' products to further improve the results?
- How does deployment of the product help or hinder developer productivity? What actionable steps does the product communicate to developers?
4. Coverage and visibility
Security tools should be able to cover a significant number of vulnerability types to be useful and should present them to developers and application security managers in a way that can communicate improvements in security.
- What types of vulnerabilities or attacks is the product best suited to finding? What types of issues are not visible using the tool?
- How does the product report its findings? Can the tool integrate with collaboration environments, or does it produce a dashboard?
5. Pricing
Finally, an estimate of pricing and whether the cost is per user, per seat, per application, or per scan is key to estimating the affordability of the tool. Provide the vendor with your likely use case, so it can also estimate your likely costs.
Key takeaways
- Adopt a software-security maturity framework to give your company a compass for its application-security efforts. The Building Security in Maturity Model (BSIMM) or the Open Software Assurance Maturity Model (OpenSAMM) are good places to start.
- Determine your starting point, not just in terms of maturity, but also figure out your immediate application security priorities and where your company is falling short in terms of protecting critical assets.
- Educate developers and train application security professionals in the necessary topics to reach the next step in your prioritized security list.
- Identify and create a strategy for legacy code and components, and adopt tools that help you maintain the security of those components.
- Integrate and automate security testing into the development process as much as possible, and in a way that avoids slowing down developers.
- Add security layers to deployed code not only to protect the applications, but also to give application security professionals and developers visibility into how the code is being attacked.
- Find the right tools for your software development and deployment process. Do not buy tools and then fit your process to the tools.
