As we quickly approach the 2015 DevOps Enterprise Summit (DOES), I've been taking a closer look at many of the conversations around agile, DevOps, and continuous delivery to keep a pulse on the who, the what, the when, the how, and most importantly, the why. It's important to understand what's required to ultimately transform an organization, whether it's the new technologies and processes they adopt, the culture they instill, the leadership they evoke—or a combination thereof. But, as DOES15 speaker Dr. Steven Spear so eloquently puts it, "Transformation is a process—all solutions are temporary."
What Dr. Spear said really hit home and inspired me to go back and look through several of the past DOES14 presentations and recent DOES15 video chats that discussed enterprise transformations. I also remembered a recent Continuous Discussions (#c9d9) episode on the top KPIs of DevOps with Gene Kim from IT Revolution and Patkós Csaba from Syneto.
Based on these industry discussions, it's obvious that in today's business world, paying close attention to the key metrics is more important than ever so that your organization can become what it wants to be in the marketplace—the leader. To achieve this, organizations are starting to put more of an emphasis on the "doing" as opposed to the "thinking." The age-old saying from Aristotle, "For the things we have to learn before we can do them, we learn by doing them," holds a lot of truth in this context. Dr. Spear also notes that the creation of a sustainable competitive advantage is done through constant and continuous learning or "dynamic discovery" to transform the way work is done and to measurably improve the outcomes.
Having the right goals, asking the right questions, and learning by doing is paramount to achieving success in any number of endeavors. For those of us working to streamline software delivery innovation through DevOps, having specific milestones and KPIs can play an instrumental role in guiding the adoption and progress of today's more ambitious and advanced IT processes within a business.
Below are four important KPIs to consider when gauging the success of DevOps and continuous delivery, thanks to research by Gene Kim, Jez Humble, Nicole Forsgren and Puppet Labs, and some additional considerations from the community.
1. Frequency of deployments
According to a recent report from IDC, the average number of deployments per month is set to grow significantly and perhaps even double in just a few years. Why? Businesses need to deploy changes more frequently to respond to customer and market demands. More frequent changes typically results in smaller change sets. And delivering smaller change sets more rapidly also means quicker feedback loops to developers. By providing development teams with as close to real-time feedback as possible, organizations enable quicker learning and speed the overall development cycle. As cycle time goes down, every other measure goes up, according to Kim. With increased deployment frequency, success rates go up as well.
There are of course many factors that play into deployment frequency. When you hear of companies like Amazon deploying updates every 11.6 seconds, you know they deployed a lot less frequently to begin with, but were able to remove the bottlenecks and manual processes through automation to increase their time to market significantly. Find ways to measure deployment frequency to improve this critical process—concepts and tools such as application release automation (ARA) can help improve the process of packaging and deploying applications (or updates to applications) in an automated, repeatable, and consistent way.
2. Speed of deployments
As you can imagine, frequency of deployments goes hand-in-hand with the speed at which you deliver them. Software delivery at scale requires discipline and automation. According to Csaba, the most important metric any developer can concern themselves with is speed. In the development process, for example, the speed at which tests and the continuous integration system reply after a commit or a push to the repository is essential. "When you are trying to run 6,000 to 7,000 tests in less than two minutes, you need gobs and gobs of speed."
When it comes to optimizing your delivery process for speed, it's essential to take a look at your processes and "cut out the fat." To me, this equates to an organization's ability to make as many repeatable processes as possible, while also eliminating as many manual tasks as possible in order to accelerate the process from code check-in to production. However, Kim points out another important but often overlooked metric here: lead time.
We tend to focus quite a bit on how long it takes from commit to production, but lead time is part of the equation as well; design time, the creative aspects, and then release time. If you're doing continuous delivery (CD), in theory your lead time is zero. But if you're not doing CD, lead time is a very important metric to keep tabs on. In fact, it's a commonly held belief that lead time is the most effective predictor of internal quality, customer satisfaction, and even employee happiness.
3. Time it takes to fix failed releases
When things go down, or major bugs cause disruption, how fast can your teams respond and make sure applications are performing properly? If it's not already, this should be a key initiative within your dev and ops teams. Knight Capital Group learned this lesson the hard way, when the company deployed new trading software that hadn't been properly tested. The software led the firm's computers to rapidly buy and sell millions of shares of more than a hundred different stocks. This went on for almost 45 minutes before the problems were even detected. According to Knight Capital Group, this software glitch cost the company $440 million.
In order to get quality software into the hands of customers, you have to make sure it works properly across a variety of platforms and at scale. More frequent releases of smaller batches of software can help alleviate these pressures as well.
4. Cultural implications and satisfaction
The time, effort, and risks associated with restructuring an entire organization can induce fear and paralyze leadership in the enterprise. Those at the top must view DevOps as a natural, undeniable evolution necessary in today's survival-of-the-quickest business ecosystem. Ask yourself what sort of cultural measures would be valuable to track. You can only have great lead times, for example, if you have a great culture where your employees feel comfortable and where they can fix problems as they see them. Culture and "tone on the top" are incredibly important—productive employees need to be able to say, "I can't work on a new feature until I make this one work," and feel comfortable doing so.
DevOps is also a bottom-up and horizontal practice, not just a top-down approach. A good approach I've seen is from Jon Whitney, a Principal Software Engineer at AppDynamics, where they started onboarding the dev lead with an ops ambassador. Whitney says the ops ambassador attends weekly planning meetings and daily stand-ups, which inspires more holistic thinking. "Dev is more likely to consider the difficulties of identifying bugs in production when the people affected are in the room, and ops better understands why certain compromises were made. Inviting ops to be a trusted partner in development elevates un-sexy items such as feature flags and logging from afterthoughts to integral parts of the design."
To cultivate your sustainable competitive advantage in today's fast-paced world, Dr. Spear says, "You must first create the internal capacity to improve and innovate—fast and without letup." With the above metrics and considerations in mind, you can help sort through all the numbers that are out there and figure out the best measurable goals for your organization. When your attention focuses on the metrics that really matter, you can improve your software development process by doing things differently to empower and connect the software development and IT operations teams, and see real and powerful impacts to your bottom line.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
