Back to waterfall: When agile and DevOps don't work well
DevOps has revolutionized the way we approach development, but that doesn't mean it's right for every situation or project. Those who leverage agile and DevOps are often surprised to find this out. Some of them are now pushing back, opting for traditional waterfall approaches. And in many cases, they're right. There are some scenarios where DevOps will slow you down and waterfall is the better option. I'll show you the trick to understand where and when DevOps is an appropriate fit, and when it is not.
DevOps and agile are not silver bullets
When we have nice, shiny tools, we have a tendency to push them upon everything and anything, hoping for the same positive results from one project to the next. But no matter what tools or method you’re looking to use, you’ll find that it’s never a good practice to assume that the same ones will always work. The core mission of IT is to select the right approach and toolset and apply them to the correct problem domain. There are no universal solutions or silver bullets, and agile/DevOps is another instance of that adage’s truth.
When is DevOps the right approach? I’ve found that it’s not an easy question to answer. Most system shops are likely to use DevOps no matter what. That blanket approach can end in disaster for some projects, while others will be perfectly fine.
So what are we learning from the patterns of use that lead to failure and those that lead to success? It’s a complex question that requires us to review some recent project outcomes and how the use of DevOps either helped or hurt the project.
These are unpopular views, by the way. We’re still in the honeymoon phase with DevOps, and any pushback seems to be met with passive-aggressive anger. However, I’m not saying that DevOps won’t work, just that it won’t work all of the time, pretty much like everything else we’ve done with IT over the years. There are exceptions to every solution, and the exceptions must be understood.
DevOps is also in a serious disruption phase. Traditionalists cling to the waterfall methodology, which has long been favored by enterprise environments for its rigorous documentation and governance. However, waterfall has taken a serious black eye from agile/DevOps zealots inside startups and enterprises, who consider it too slow and clunky for their needs. But what’s the truth?
It’s about speed and agility, right?
I've walked into startups that were seeking my guidance and asked to see the current technology offering, the technology roadmap, or a plan for the implementation of features and functions. The response, in many instances, was, “We don’t have one.”
It’s usually explained to me that, as an agile shop, they work in 30-day sprints. DevOps is there to support their agile processes, which further compresses the length of sprints, allowing them to accomplish more work in less time.
All of which is great. DevOps is a game-changer within many organizations. It takes a long time and a lot of training to implement, so I’m always happy to see it working. However, if its use is circumventing planning, then it may have a diminishing value.
"We’re still in the honeymoon phase with DevOps, and any pushback seems to be met with passive-aggressive anger."
In the case of some startups, there was never a plan to have a roadmap. And they are proud of that fact. However, they will discover that their customers have other ideas. Customers like to see long-term plans for feature/function implementation, and if the DevOps team can only show a series of iterations labeled, “We’ll do our best,” the customers will likely move on to another product that can show a long-term roadmap. The idea of tossing any sort of planning out the window is not the purpose of DevOps, nor is it a functional business model.
The issue with DevOps is that we seem to be either all in or all out. DevOps becomes the magic pixie dust that, when tossed on any development efforts, makes problems go away, including backlogs, quality issues, security, and integration with other productions and platforms. But DevOps is really just a set of tools that we leverage for automation—nothing magical happens. It’s a concept we’re still learning about that brings about varying degrees of success.
The reality is that, in many cases, DevOps should be the underlying way that we build and deploy software. However, we can’t extend it to every process that we have in the IT organization. By doing so, we’re actually putting the organization at risk.
When DevOps works
With DevOps, we work to eliminate any latency that may exist with the development and operations teams, which means that we automate pretty much everything. DevOps is all about continuous development, continuous integration, continuous testing, and continuous deployment. The idea is that a developer can write the code, define the data, press a button, and the programming goes through a series of tools that places it on the final production system, ready to run.
The growth of cloud computing drives much of the interest in this arena, considering that cloud uses centralized deployment of applications; developers can update the code pretty much anytime they need to. Gone are the days of 1.0, 1.1, 2.0, etc., and that's a positive step.
In Figure 1, we see the use of agile, driven by DevOps, as significantly reducing the “failed” or “challenged” projects. Compare that to waterfall, which typically means all traditional development methodologies rolled into one.
Figure 1: DevOps brings with it a tremendous value for most organizations, if leveraged effectively.
Any software development project that can gain speed and agility from DevOps should probably use that method. But I believe it's a bad idea to apply DevOps and agile to all software development projects or, worse, to other projects that are not directly software-related, such as the long-term roadmap example that I talked about earlier.
When DevOps does not work
DevOps seems to lay an egg when it comes to some software development projects and most planning and system design projects. Software development projects that are nearly always a misfit for agile are older systems, such as those based on COBOL and ISAM. Those systems just don’t work and play well with modern tooling. Moreover, there is typically very little automated effort available to roll out changes to those systems on a daily or weekly basis. When applying DevOps, most of those types of software systems quickly fail.
"DevOps becomes the magic pixie dust that, when tossed on any development efforts, makes problems go away, including backlogs, quality issues, security, and integration with other productions and platforms. But DevOps is really just a set of tools that we leverage for automation—nothing magical happens."
Planning and design are the worst fits for agile and DevOps. Take system design, for example. Most people think that iteration, also known as continuous improvement, is the most effective method for work. However, working through the design of a software system systematically without boxed iterations allows architects to create a much more successful design and the design to be completed in a compressed amount of time.
Why is this? Well, the design of an application is not the building of the application; it’s the process of thinking about how the application will be effectively implemented. Moreover, the design of the database is the process of thinking about its relationship to the application and requirements. When you apply DevOps principles to design, then you’re doing less thinking and more automation of the design. The result is that you’ll get a design that’s easy to change but filled with flaws.
Planning, as we noted above, is often circumvented by DevOps. The battle cry for developers these days is that the use of agile means that planning is obsolete. Tell that to C-levels who need to tell investors when products will drop and to customers who need to understand when their issues will be fixed.
The reality is that, while DevOps is powerful and has a great deal of value, we can’t leave the planning to a bunch of developers in a virtual drum circle where whatever pops out of the far end of their DevOps processes is going to be a surprise. That’s no way to run a business, and it is totally unacceptable to me.
Understand where DevOps and agile help, and where they hamper
The fact of the matter is that we have all of these new technology trends now: Big data, cloud computing, machine learning, and DevOps. However, none of these technologies should be used in all cases.
Those who extend the use of DevOps to pretty much everything will not see overall value, since DevOps will be diluted by the amount of failures driven by its overuse. However, those who understand DevOps' limits will understand that, when used in the right ways, it is very effective. Take a good look at what processes in your software lifecycle might be hampered, rather than helped, by DevOps and agile.
Have you had any experiences where agile or DevOps caused problems and weren't the right fit for a certain piece of the SDLC? Maybe you disagree with the author?