Traditionally, companies wanting to use penetration testing to uncover hidden flaws and vulnerabilities would engage with a pen-testing consultancy. Their consultants would perform the test within a specific time window (e.g., two weeks) and produce a final report that describes the findings: a list of vulnerabilities, their impact, and perhaps recommendations for remediation.
Companies that went through this process experienced varying degrees of success. Many found the post-test validation process challenging. The tests a consultancy performs represents the state of applications at that point in time. As a result, organizations sometimes shelve the pen-testing reports,and simply leave flaws or vulnerabilities unchecked or unfixed.
To tackle this problem, new ways of delivering pen-testing services are emerging. The most interesting is pen testing as a service (PTaaS), delivering pen testing in a modern SaaS model—continuous, interactive, and transparent. The hope is that this new model will reduce costs and ultimately lead to better management of vulnerabilities across the board.
I recently investigated PTaaS in a study, through the lens of return on investment. The study, “Return on Investment of Pen Testing as a Service,” was the result of work with many application security professionals who manage pen-testing and remediation projects. Through in-depth interviews and other interactions, I found that PTaaS platforms can have a substantially higher ROI, particularly when you factor in the hidden costs of pen-testing services.
Here are the four hidden cost metrics I investigated in the study.
1. DevOps agility
Many organizations that practice DevOps find it difficult to accommodate the “slice-in-time” nature of traditional pen testing. One vice president of engineering of a SaaS provider told me that it was doing 5,000 builds a week. With traditional pen testing, you must wait a few weeks for the pen-testing tests and the final report to be completed. And as soon as the report comes out, the findings are obsolete.
“This professional services model only gives you a 'slide-in-time' view, and simply wasn’t moving fast enough for us.” With PTaaS, the organization can view in real time which tests are being performed, receive findings as soon as a test is completed, and engage with the testers in real time should there be a change in the application. In other words, the organizations using PTaaS were able to incorporate the test process as part of their DevOps workflows.
2. Time to result
With traditional pen testing, triage and remediation tasks can't start until the entire test report is available. In contrast, PTaaS platforms produce findings as soon as each test concludes. That means your organization can start triage and remediation tasks right away, even while the rest of the tests are still in process.
The organizations I interviewed called this out as a significant benefit. With PTaaS, the time-to-first-result shortens from two weeks to a day, or sometimes just a few hours, reducing the window of exposure for vulnerabilities.
3. Triage efficiency
With each finding, your team needs to validate the result, assign priorities, and determine if remediation tasks are within scope. Often, this involves going back to the testers to seek clarification. Engaging with testers after the fact in the traditional consultancy model can be challenging. But with PTaaS, organizations reported a more expedited, seamless triage process.
The director of security at one enterprise software company said his team used to spend hours on the phone or with email just to ask questions and resolve the “lost in the translation” difficulties between the developers and testers. “We were like the mediator between the testers and the dev teams. It’s difficult to communicate this way—everyone is frustrated, especially us because we didn’t want to spend our time this way.” The PTaaS platform lets dev teams and testers engage directly, with the relevant test information right in front of them. This facilitates faster, more transparent communication and problem resolution.
Overall, the investigation found that the triage time per vulnerability with traditional pen testing is about 89 minutes per vulnerability. With PTaaS, this number is dropped to 20 minutes.
4. Managing results
Traditional pen-testing consultancies produce a testing report—often a PDF document that can be more than 50 pages long. Not only does the organization face the arduous task of wading through the lengthy document, but it's difficult to manage the report artifact across multiple tests over a long period of time.
Imagine an organization that wants a more programmatic approach to pen testing. For example, you might want to perform an overall trending or statistical analysis with specific types of vulnerabilities. To do so, you'd need to expend considerable effort to process the PDF and extract the data. With PTaaS platforms, the data is readily available. That's why managing results is much simpler with PTaaS than with traditional pen testing.
Get more for your pen-testing money
When you take into account hidden-cost metrics together with some of the more straightforward metrics, such as pricing or management overhead, PTaaS services yield a compelling ROI over traditional pen testing—an increase in ROI of almost 96%.
Keep learning
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed fast on the state of app sec testing with TechBeacon's Guide. Plus: Get Gartner's 2021 Magic Quadrant for AST.
Get a handle on the app sec tools landscape with TechBeacon's Guide to Application Security Tools 2021.
Download the free The Forrester Wave for Static Application Security Testing. Plus: Learn how a SAST-DAST combo can boost your security in this Webinar.
Understand the five reasons why API security needs access management.
Learn how to build an app sec strategy for the next decade, and spend a day in the life of an application security developer.
Build a modern app sec foundation with TechBeacon's Guide.
