You are here

You are here

Lessons learned from 6 high-profile DevOps journeys

Christopher Null Freelance writer

More companies are turning to DevOps to develop their applications, but not everyone is seeing increased productivity and profitability. So what are the ingredients of truly successful DevOps transformations?

Here's a look at six high-profile initiatives that succeeded at Target, New Relic, Microsoft’s Visual Studio, Walmart, Netflix, Walt Disney, and Capital One—and what they all have in common.

They started from the grass roots

All of these companies started from the ground up, rather than from the top down. Jose Fernandez, former lead software engineer at New Relic, noted that the genesis of the company’s transformation was the establishment, by a single software engineer, of a site reliability engineering (SRE) team in 2012. Some of the inaugural SREs became DevOps champions, and the approach blossomed from there.

Similarly, Target's DevOps makeover grew out of internal evangelizing until, said technical architect Dan Cundiff.

"[It] started out in small corners of development and infrastructure teams … caught on like wildfire."
Dan Cundiff

That's not surprising. Software engineers in the trenches know the most about what can drive improvements in how they work, and they're motivated to try new methods. But at some point, said Carmen DeArdo, DevOps transformation leader in the financial services industry, management needs to lend support so these movements don't languish as local optimizations.

"Management doesn't need to impose transformations. It needs to set direction and focus on optimizing the whole, while removing the silos and organizational structures that impede thinking horizontally across the delivery value chain."
Carmen DeArdo

They made it easy to collaborate across teams

People who launched successful DevOps transformations share a willingness to rethink traditional roles and break down barriers between teams and departments. They don't shy away from a trial-and-error approach to rethinking organizational structure to facilitate collaboration.

This was key to Microsoft’s Cloud + Enterprise engineering group's DevOps journey, said Lori Lamkin, director of program management for Visual Studio Online. The group combined traditional developer and tester roles into a single software engineering discipline. It then gave those folks accountability for the entire software process, from developing a product to fixing problems.

"Customers expect more and more frequent delivery, and we just can't have any traditional roles getting in the way."
Lori Lamkin

Walt Disney faced also had to overcome structural barriers and red tape. It had 200,000 employees across four divisions: Studios, Consumer Products and Interactive, Parks and Support, and Media Networks, each with its own autonomous CTO at the top and several tech management layers beneath.

Jason Cox, director of systems engineering, and his team were charged with company-wide change that broke down silos and emphasized mutual objectives. 

"[We had to get teams to] talk to each other, both culturally and technically."
Jason Cox

If you look across a company's delivery value stream, it requires work across planning, architecture, design, development, testing, and infrastructure support to shorten lead time and increase deployment frequency without compromising the change success rate, DeArdo said.

"We know that smaller batch sizes are key to delivering more quickly," DeArdo said, and that requires collaboration with the product owner. Monolithic architectures need to be broken down into APIs and microservices. You also need development and design practices such as test-driven development, continuous integration and deployment, and dark launching. Infrastructure needs to be self-service. And other traditional silos, such as performance and security, need to be integrated earlier as part of development, he said.

They emphasize automation

Automation is a ubiquitous element of all the DevOps transformations. Building it into the development lifecycle was critical to ensuring consistency and a measure of predictability for each software release. It also physically and mentally liberated developers from mundane, time-consuming tasks, allowing them to focus on processes that require more creative solutions. Ultimately, it yielded a host of benefits, from higher productivity to faster time to market.

"Overcoming fear of deployment and improving deployment frequency requires more automation earlier in the process."

This includes both automated testing and automation to remove the wait states associated with getting infrastructure for test environments. Another area for automation, DeArdo said, is alerting to provide feedback when a committed change has a negative consequence.

They use their own tools when necessary

The tools an organization uses to support its DevOps transformation are as unique as its business needs. Most use a mix of open-source and commercial options in their toolkit, but these companies also built custom tools as needed.

Capital One created its own DevOps dashboard in Hygieia, then open-sourced it to the community, where it has been a tremendous success.

Netflix famously created its Simian Army, a suite of automated tools to stress-test the business' infrastructure so it could get ahead of potential problems before they affected customers. And Walmart built Hapi, a Node.js framework for creating applications and services. It lets developers focus on writing reusable application logic, instead of spending time building infrastructure.

But a custom-built tool no more facilitates a magical DevOps transformation than will any readily available tool. While you should develop your own tools when it makes sense, DeArdo says blending best-of-breed tooling with an integration technology is ultimately the way to go.

They embraced cultural change

The biggest constant across all the transformations was the ability of these businesses to roll with the often uncomfortable changes that a DevOps initiative demands. This sometimes meant re-imagining or eliminating long-standing roles and processes to empower personnel, increase efficiencies, and give participants a bigger ownership stake in the final product.

Needless to say, changes of this magnitude require considerable flexibility on the part of the organization.

"DevOps is about empowering teams and removing blockers."

Command and control structures often found in ITIL (Information Technology Infrastructure Library) and PMBOK (Project Management Body of Knowledge) "need to be transformed," DeArdo said. Traditional project management, where people are allocated to projects, must be transformed into a product management model where work is allocated to teams.

A cultural shift starts with "servant leadership"—team leaders working in the trenches—approaching management, he said. From there, management must nurture it to ensure that it flourishes company-wide.

Move forward with a DevOps mindset

In the end, mindset is the most important element of culture change, DeArdo said.

"Your company has to develop and support a learning/teaching and continuous improvement mindset. People need to be encouraged to experiment and fail, in order to quickly learn."

He said he talks a lot about the need for a DevOps mindset, which he describes as always seeking to learn, to improve, to experiment, to transform, and to serve in order to meet the goals of the company, "which should be to become more responsive to its customers."


Keep learning

Read more articles about: App Dev & TestingDevOps