DevOps is commonly introduced into organizations for some well-understood, and usually highly justified, reasons. DevOps speeds the overall pace of the software development and release cycle, improves the quality of deliverables, and helps to resolve problems more quickly. But all of these intangibles should, and do, flow down to the bottom line. Which means that any well-designed program should include a healthy perspective on ROI.
So why isn't DevOps ROI routinely measured?
Measuring the unmeasurable?
One school of thought holds that DevOps ROI isn't measured because it really can't be.
Michael Cote is the poster child for this idea at the moment. He recently wrote that the question of "what is the ROI for DevOps?" was "simultaneously absurd and important." In a recent blog post for FierceDevOps, he says, quite accurately, that DevOps is not so much an investment as it is an operational process, and that the number of variables in an organization is too great to make this calculation possible. "DevOps is an unmeasurable process with respect to ROI," he writes. "It has value, to be sure, but is nearly impossible to measure independently and precisely."
Cote goes on to offer suggestions for CIOs who are tasked with making a DevOps ROI calculation. Typically, these involve fudging numbers so executives will be comfortable with the adoption of the strategy.
DevOps ROI can't be measured, but the ROI of the projects you initiate and the products you develop and release after DevOps is instituted can be measured, Cote argues.
Is Cote right? Is DevOps too intangible to be measured with the degree of accuracy that leads to a proper ROI calculation? The other side of the argument is, of course, that it can and should be measured, and that there are a variety of mathematical constructs that can lead you to a valid ROI.
Consider a few of these approaches.
Time is money
One of the biggest benefits, if not the biggest benefit, of DevOps is the promise of speed. DevOps enables more and faster code deployments, which means a decreased time to market and more opportunities to capture revenue from customers.
The relatively simple math works like this.
Presume you experience the bulk of your sales when a product is first released, and these sales gradually decline until your next product (or update of the same product) is released. Over time, your revenue develops a familiar pattern, first spiking, then declining, then spiking again, creating a sawtooth graph.
What happens if you speed up the pace of development and accelerate your release schedule by a factor of two? You get the same sawtooth graph, but with twice as many spikes. In this example, your overall revenue is doubled, likely making your ROI from DevOps quite high even after accounting for any costs incurred in the course of implementing a DevOps framework. DevOps.com explains this (including an easy-to-understand graphical explanation).
This is an oversimplified example, and it raises more than a few questions. Not every product release is equally profitable. In fact, not every product release is profitable at all. DevOps releases typically function as more incremental, smaller updates, and many may not be focused on generating revenue at all but rather responding to customer feedback, fixing bugs, or doing other maintenance-related updates. It would be foolish to make an assumption that every software release has a significant financial impact.
At the same time, these are all numbers that can indeed be quantified. Overall revenue is a fundamental business metric. Sales can be broken down for each product and on a daily (or even hourly) basis. With some careful but relatively simple analysis, you can generate a model that describes this pattern. If you've implemented a DevOps program or even piloted one, you can use those numbers to generate a contrasting picture of revenue plotted against the original data.
Financially measuring quality improvements
Another way to measure DevOps ROI is a corollary to the velocity-of-revenue calculation outlined in the previous section. This approach considers the increased quality of work performed under a DevOps framework and calculates ROI from a cost-avoidance standpoint, rather than a revenue one.
Toon Verbeek, developer evangelist for Wercker, used Keen IO analytics to graph the pass/fail rate of Wercker's development team. Through a system like this, Verbeek is able to see explicitly whether a DevOps framework improves build quality as promised. He has found that it has, with actual metrics to prove it.
He believes his findings can be converted into a monetary value suitable for an ROI calculation. "You can break it down with the hourly rate of development time," says Verbeek. "Assuming an average salary of $120,000, developer labor roughly correlates to $60 per hour. So let's say each failed build takes an average of three hours to fix, and you've reduced build fails by a factor of two, week-over-week. You're looking at a cost 'savings' of $360 week-over-week."
This is a limited example, but it does showcase a legitimate, real-world ROI calculation that can be extrapolated on a broader scale and tracked over time.
Enhanced productivity as an ROI driver
A third way to look at DevOps ROI considers the team's overall productivity, specifically a calculation of idle time and how it is avoided.
"IT leaders need to connect the strategic objectives of the business to the work their teams do in order to create a DevOps transformation," says Pete Cheslock, senior director of operations and support at Threat Stack. "If a goal-oriented culture is being achieved, it will lead to more efficient collaboration. The easiest way to measure this is by looking at the number of deployments per day per developer.
Additionally, IT leaders who are able to manage more infrastructure with the same number of team members can point to that improvement as a measurement of success."
Of course, productivity is more difficult to quantify from a financial standpoint than some of the metrics we've discussed above. Says Cheslock, "Attempting to put a monetary value on deployments per day does not give an organization an accurate gauge of success. It's not the value of deployments per day; it's the value of efficiency in your deployment process, which could potentially result in more deployments per day and a reduction in the cost of deployment."
Efficiency and frequency
That doesn't mean that financial metrics surrounding IT productivity are impossible to generate. You just have to go about creating them in a slightly different way.
Nicole Forsgren, director of organizational performance and analytics at Chef, has done extensive research into DevOps deployments and has found that high-performing DevOps organizations are twice as likely as other companies to exceed profitability, market share, and other financial goals.
One of the key measurements she has identified involves metrics that quantify what she calls overall IT performance, which is in turn composed of three key factors: deployment frequency, lead time to deploy, and mean time to restore. These are all designed to show throughput and stability in the development process. "This is an important measurement," says Forsgren. "My research has shown that throughput and stability move in tandem."
Forsgren's research classifies organizations into three groups based on these IT performance metrics. Specifically:
- High IT performers. Have high deployment frequency, fast lead times, and fast mean time to restore. These companies are the best at significant statistical levels on all of these fronts.
- Medium IT performers. Have medium deployment frequency, medium lead times, and medium time to restore.
- Low IT performers. These perform the worst, statistically, on all measures.
Now, when you look at the financial performance of these three groups, Forsgren says that you find that the high IT performers group is strongly aligned with high overall organizational performance, which means higher overall profitability, greater market share, and overall better financial results. Similarly, the low IT performers group does comparatively poorly on the same financial measures. "This shows the metrics are tightly related and suggests that IT performance is predictive of organizational performance," she concludes.
What this fundamentally means is that some of the softer IT-focused metrics that measure productivity may not be able to be explicitly measured in financial terms, but they still have financial relevance that can be extrapolated through comparisons to other organizations that have embraced DevOps. In other words, managers should feel confident that a successful DevOps strategy that succeeds on all of the above IT metrics will lead to improved business performance because of the strong correlation between the two.
Looking at long-term ROI
The ROI of anything requires a long-term analysis, and it involves looking at more than one factor at a time. This kind of multivariable analysis means analyzing the financial impact of DevOps across a spectrum of accounting line items.
CloudMunch, for example, offers a detailed sample financial analysis that breaks down cost savings across a multitude of categories. (See page 7 in the linked slide deck.) By measuring the amount of time saved over a year in categories like change deployment, automation of repetitive tasks, and support, and then multiplying those numbers by the average cost per hour of developers' time, hard ROI in annual dollars saved can be calculated. Again, some of these figures will be estimates by necessity, but the more finely grained you are able to break things down, the closer you are likely to get to the true financial data.
As another example, Manoj Chaudhary, CTO and VP of engineering at Loggly, uses five categories for the calculation of his company's ROI from DevOps. These include:
- Release frequency. How fast code releases to production.
- Infrastructure recovery. How fast production recovers from "fires."
- Infrastructure resiliency. Reduction in downtime/problems with infrastructure.
- Infrastructure efficiency. How well resources are shielded from production deployment problems.
- Automation. Reduction in human intervention to fix problems.
Again, each of these categories can be compared through "before" and "after" data points, and time saved can be assigned a dollar value based on the cost of labor.
Is DevOps ROI a valid question?
There's no doubt that measuring DevOps ROI is difficult. Measuring the value of a change in process is always going to be more difficult than measuring the ROI of, say, buying a piece of equipment or launching a new product. Nearly every proponent of DevOps ROI makes allowances for this, and most of these ROI models include a hefty allowance for "intangibles."
Is Cote right that even if you do calculate an ROI that it is not the ROI of DevOps itself but rather the ROI of the results that DevOps has made possible? DevOps guru and author Patrick Debois recently stated in an interview that "we should be thinking about [ROI] in terms of ... accelerated benefits realization, and shortening that [cycle]. It's really not the ROI of DevOps, it's really more that the ROI of your original project can be realized sooner."
Remember that the entire DevOps concept is still young, and while it has shown results time and time again, financial bosses still have to be won over. But over time, ROI calculations for DevOps will no longer be required any more than, say, ROI calculations are requested before the installation of Internet services or Wi-Fi.
By fully integrating development and ops into the business, DevOps is transforming IT from a cost center to a business enabler. And that's part of any recipe for financial success.
Image credit: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
