As IT organizations seek to bring developers and operations staff into harmony through a DevOps approach to collaboration, automated technology and process improvements play a crucial role in continuous, iterative development and deployment. But above all else, a sound DevOps culture fundamentally solves one of IT's biggest people problems—bridging the divide between dev and ops teams in order to get them to stop working at cross-purposes with one another.
In fact, Gartner estimates that when infrastructure and operations teams try to drive a DevOps initiative without nurturing a cultural shift as well, they'll fail 90 percent of the time. In order to truly get DevOps to work, both sides of the house must be willing to cooperate in the behavioral modifications necessary to effect cultural change.
"With respect to culture, DevOps seeks to change the dynamics in which operations and development teams interact," explains Laurie Wurster, research director at Gartner, in a piece by Consultancy UK. "Key to this change are the issues of trust, honesty, and responsibility. In essence, the goal is to enable each organization to see the perspective of the other and to modify behavior accordingly, while motivating autonomy."
Of course, all this is easier said than done. Dev and ops represent the biggest mismatch in perspectives and temperament since Oscar and Felix, the original Odd Couple. Developers are innovators, paid to be agents of change. Operations staff pray at the altar of stability, tasked with maintaining uptime and continuity.
Up until now, it might seem that IT leadership has set up this schism by design. At most organizations today, software engineers and operations staff are pressured by a different set of objectives handed down by executives. They rarely work together enough to understand the value each group brings to the ultimate objective of improving customer interactions. Both sides operate in fear of missing their objectives, and too frequently they engage in finger-pointing when experiments go south. If organizations want to tap into the efficiency and speed of innovation afforded by DevOps, IT leadership must find a way to bridge the gap between these two technology tribes to get them to remember they're playing for the same team and encourage them to find new ways to achieve mutual objectives.
Gain developer buy-in
The first step in bridge-building is getting developers to buy into the DevOps ethos, which is sometimes a tall order, considering that a lot of the initial impetus behind DevOps can be attributed to the pain felt by operations staff when developers foisted unstable code on their fine-tuned environments. This is a pain that grew in regularity as dev embraced agile philosophies.
"One of the issues of agile is it was all about speed and we lost the reliability and quality because we took all of these shortcuts," says Mike Kavis, vice president and principal architect for Cloud Technology Partners, an IT consultancy. "And I really believe the DevOps movement came from those system administrators tired of getting paged at four in the morning."
In a collaborative culture, developers both empathize with their operations brethren and also understand why ops problems are development problems, too. But how do you get there? DevOps success stories show that there are some key ingredients to lasting cultural change.
Incentive shifts
In most organizations, the reason that dev and ops work on such divergent planes is that they're paid to. "[IT organizations have] created a situation in which dev and ops are incented to do different things, and this is at the heart of where the conflict between dev and ops arises," explains Kurt Bittner, analyst for Forrester Research.
When developers are only measured by the speed at which they roll out new features, and operators are only judged by uptime, getting the two groups to see eye-to-eye is next to impossible.
"One of the challenges that I see is that people say 'Hey, we are going to move to DevOps,' but they don't change people's incentives to behave differently," explains Kavis. "I come from the app side and I remember at this one company, we had our 'top 10' list and the infrastructure had their top 10 list, and they didn't match. So if I needed a server and it wasn't on their top 10 list, then I was SOL."
Reduce feature guilt
As both Kavis and Bittner explain, the more leadership drives both groups against a common set of incentives, the easier it will be to start shifting culture. "Essentially what we need to do is measure everybody the same way," Bittner says. "And part of that means measuring devs on stability, and part of that means measuring ops on innovation. But the best thing is to reward everybody on improving the customer experience."
This is exactly how teams at Orbitz started to transform their culture in order to drive a DevOps practice that ultimately helped the travel reservation company greatly improve the customer experience.
Ori Rawlings, senior software engineer for Orbitz Worldwide, spoke at Velocity last year: "One of the things we did early on was established a shared goal across dev and ops teams: to double the hotel search capacity across our whole site." Setting up that shared goal helped drive developers to work incrementally on obscure bottlenecks that had been plaguing the site for a long time and ultimately achieve big wins in performance and capacity.
Changing that incentive structure also jump-started a shift away from a developer mentality that can really kill DevOps collaboration—it's called feature guilt. According to Greg Burton, software engineer for Orbitz Worldwide, one of the reasons it can be tough to get developers to help operations solve long-standing performance problems is that they consider it "unproductive" time. "We've got to view operations-driven development as productive work. That is a cultural thing that has to change," he says. "The belief goes all the way from management, down through your peers, and all the way to the business that features work is the productive work and operations work is the thing you have to do real quick on the side get it done—out of the way—and get back to the feature work. But that will kill you."
As Orbitz went on its quest to improve the customer experience, one of the things it worked on was getting developers to the point where they understood that tending to the health of a web app was just as important as developing new features for that app.
Making failure less expensive
One of the major drivers of the DevOps continuous delivery practice is the experimentation that is possible once an organization is running on all cylinders. And innovation thrives on experimentation. Unfortunately, one of the big cultural hang-ups that impedes DevOps progress is an environment that heavily penalizes failure.
"The cost of failure has to be as small as possible. Right now in the enterprise the fear of failure is endemic," explains Randy Bias, vice president of technology for EMC and director at the OpenStack Foundation. "It's practically a culture of fear. It'll take a long time to change that culture of fear but one of the primary ways to do it is to start driving down the cost of failure."
This can be a bit of a catch-22 for organizations with monolithic application infrastructures, but the more organizations can make it possible to move toward bite-sized development chunks, the less expensive these iterations become, says Kavis. "Get to small deliverables," he recommends. He also believes that blameless post-mortems will not only help give developers confidence that they won't be raked over the coals every time they try and fail, but it will also present a huge learning opportunity.
"Even if your experiment failed you probably learned something from it," he says. "A lot of technology we have today is from failed experimentation."
In fact, one of the great DevOps success stories credits blameless post-mortems as playing a huge part in instilling a collaborative culture. E-commerce giant Etsy was able to grow rapidly based on this experimentation-friendly practice.
"One of the things I allowed people to do is make mistakes more freely," Chad Dickerson, CEO of Etsy told Business Insider. "The best way to learn to ride a bike is to ride the bike and fall down. We have a ground rule that the purpose of a post-mortem is to find out what happened and how to make it better, not to find a person to blame."
Developers on call? Sure
There's no better method of instilling empathy than having a person understand a colleague's role and responsibilities. And for engineers, that means putting them right in the thick of the defining duty of the operations role: working on call. It's the old "walk a mile in another person's shoes" concept.
Having developers work on call—as the developers at CenturyLink do after three weeks on the job—helps instill an awareness of the consequences of unstable releases.
"Developers are starting to feel responsible for their code's behavior. That's directly due to developers being on call," says Nick Goodman, director of platform engineering for Bunchball. His company's move to put developers on call regularly wasn't pleasant at first, but this has gone a long way toward breeding an attitude of respect for stability among engineers.
On the other hand, simply pushing developers into the on-call deep end may not be fair. Paul Beltrani, webops at MC10 Inc. and former techops at for Onshape, a CAD SaaS provider, says his firm has modified what on-call duty looks like. They try to bring developers into the fold without too much of a shock or disruption. For example, Onshape has a partner-based, on-call setup that puts one dev person and one ops person on a single team to help establish a mutual mentoring dynamic. "As developers are getting more of a feel for how the infrastructure operates, the people who focus more on operations are getting a much better handle on how the code operates and how the various services talk to each other," he says.
Don't be impatient
Whatever methods organizations use to help foster a culture of collaboration between software engineers and operations personnel, the biggest thing to understand is that every approach will yield slow results. Cultural change takes time and tends to stem from grassroots efforts. One of the best ways to amplify these efforts is to do a good job of finding champions among engineers and helping them evangelize among their peers.
"You are asking teams to do things they don't want to do for various reasons. They don't have experience, they have objections, and so on. It's really hard to break through," says Chris Nowak, senior vice president and head of DevOps services for Bank of America. "What we would do is find some good champions that would work with us, app dev teams that were good and vocal, and we got them on our side and eventually could help other teams let go and embrace a new way."
Ultimately there's no single, easy way for effecting the kind of organizational change that results in DevOps success. The reason cultural shifts are so difficult to achieve is that every business has unique market requirements, industry considerations, resource constraints, and appetites for change. The comments from practitioners here offer some good suggestions on how to get started bridging the gap between dev and ops, but each organization will have to figure out their own cultural matrix to make the sort of meaningful changes that DevOps success requires.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
