You are here

How I led 6 R&D groups through an agile transition

public://pictures/image2015-3-31 9-57-45.png
Ruly Weisbach, RND Director, HP

Over the past 10 years, I've led six enterprise R&D groups through a transition to agile. Most of those teams believed that having two-week sprints, weekly scrum meetings, and a backlog full of user stories made them agile—they were wrong. Development started on too many features at once, so the work in progress (WIP) was always overloaded, making it impossible for the product manager to get feedback, react to it, and change priorities in the backlog. These teams still had eight-month to year-long release cycles, with development for most features completed toward the end of each cycle. They then went through a two-month QA cycle before shipping the product.

That's not agile.

An agile transition isn't about daily scrums, two-week sprints, or sticky notes on a board. It's about getting the team to focus on delivering the most important features first and assuming that the rest will change due to customer feedback or market demands. To successfully transition to agile, you must prioritize your feature backlog and deliver the most important features within a release cycle of three months or less.

This may sound simple, but for an enterprise R&D organization that had been using waterfall for the last decade, it was a huge change. Here are the principles I followed to make it happen.

[ Get up to speed on quality-driven development with TechBeacon's new guide. Plus: Download the World Quality Report 2019-20 for lessons from leading organizations. ]

Be productive, not efficient

My typical starting point was a long, prioritized list of features and a large R&D organization that needed to implement them. Managers have a tendency to give each developer one feature to work on. They justify this by claiming it's more efficient than having several developers working on a handful of features. The result:

  • A huge backlog in which all features are constantly in progress

  • A product manager who has no flexibility to change priorities, even if more important requests come in

  • Features that take a long time to develop and are only ready (assuming they pass QA) toward the end of the release cycle

  • Resources such as QA, technical writers, dev, and ops, who are idle most of the time but become bottlenecks toward the end, once all features are developed

These managers didn't understand that being productive is more important than being efficient.

Consider the alternative: cherry-pick the most important features and stream them so the entire lifecycle workflow can consume them. This may seem inefficient, but it means that developers are available to help with QA, work on automation testing, do research, or even take courses to expand their horizons, rather than dispersing to other features that may create noise for those working on the important ones. Once those few important features have been thoroughly tested and fully documented, and dev and ops are ready for the upgrade, then they're truly "done." That's what being productive with agile is all about.

During a recent project, I realized early on that it would take two years to get through the group's backlog. I set about paring it down to something more realistic. As the development manager, I decided to make the effort to understand each feature in the backlog. I challenged the team to justify each feature and explain why it was in the backlog and more or less important than other features.

Rather than take this up individually with each team leader, I called a meeting where everyone went through the backlog together. We ruthlessly removed items that didn't meet our criteria for a minimum viable product (MVP). Our goal was to define the top five themes and no more than 50 features. We agreed that the meeting would continue until we reached that goal. The result was a manageable and achievable backlog and a viable product.

Yes, cherry-picking is an effort, but it's worthwhile.

[ Get up to speed with TechBeacon's Guide to Software Test Automation. Plus: Get the Buyer’s Guide for Selecting Software Test Automation Tools ]

Structure your R&D organization vertically, not horizontally

In an organization governed by Conway's Law, in which the architecture of its products mirrors the structure of its R&D organization, there are two factors that prevent the transition to agile. First, there's the practical factor. Features usually affect all architectural layers, which means that different groups, teams, and managers need to work on them at the same time. Each has its own plan, perspective, and dependencies. This pushes the organization to the classic waterfall model, where infrastructure must be complete before the business logic group can proceed. Only then can user interface (UI) development finally begin. And conducting a design or planning meeting requires maneuvering eight managers into a room at the same time.

Second, there's the human factor. When each group strives to excel in its own domain, you end up with a generic platform, only 20 percent of which is ever used. You also end up with a UI infrastructure that can handle a vast array of use cases, but again, only 20 percent of those ever make it to production.

When you structure your R&D organization vertically as a group of feature teams, with each team responsible for a complete functional domain of the application, you can write or change code in all layers. You may have architects and other experts to govern critical areas and participate in design and code reviews, but the ultimate end-to-end responsibility is within the group. Expect some teams to resist this type of reorganization by claiming that it's more efficient to have the corresponding experts work on the back end, business logic, and UI. But as I've already pointed out in the first guideline, it's more important to be productive and allow a team to complete one feature from beginning to end.

Track features as well as user stories

User stories are critical to agile. Equally important is how to get them truly "done." This works at the team level, but as a manager overseeing many teams with hundreds of engineers, you can't track progress at the user story level. You should be concerned about features, the high-level, deliverable functionality that, once done and added to your application, customers can use. Your primary concern as a development manager is to deliver new functionality that provides value to your customers at least once every three months.

To achieve that, you need to:

  • Manage and track a feature's WIP to ensure that teams aren't working on too many features at once.

  • Keep features focused and limited in scope. Don't let features get any bigger or more complicated than needed.

  • Track the quality of features.

  • Make commitments to your product managers regarding the resolution of features. They should be able to set priorities for features and track progress.

Plan short term, not long term

Waterfall R&D organizations that plan for the long term tend to design and implement too much functionality and spend too much time on low-priority features.

This leads to:

  • Complicated and generic infrastructure that supports any potential feature that might come along. But since requirements constantly change, you end up with complicated software that needs to be rewritten.

  • A huge and unmanageable backlog that includes use cases that may arise in the future, along with people arguing about functionality that may never be implemented.

It's much more productive to implement only what you need for the next delivery and refactor your code later if necessary. Even if a feature seems extremely important, it's less important than the features before it in the backlog. If it can't be completed within the next two releases, there's no point talking about it until you break it down into smaller parts. Train your teams to think MVP, prioritize the backlog items from '1' to 'n,' and focus on what needs to be delivered in the next three months. This gives them a better understanding of the project's scope, and they'll be more likely to stay focused on the functionality that is currently most important.

As for the cherry-picking exercise I mentioned earlier, that's not enough. Once you have the cherries in the basket, you need to keep other low-hanging fruit away. Product owners tend to think their feature is the most important, and each will try to cram as much functionality as possible into that feature. To avoid a gradual and steady bloating of your backlog, follow up every few weeks with your teams to make sure that "featuritis" isn't creeping into the backlog. As the group's manager, it's your job to maintain a holistic view of the product and be ruthless about new features that any product manager attempts to introduce. I find that I must be vigilant to keep the backlog in a manageable state and true to the MVP.

The results

Until eight months ago, the R&D organization I managed worked in waterfall mode and was used to releasing software in 12-month cycles. The organization's agile transition wasn't easy and required a change in both mindset and culture. These guidelines were instrumental in driving the transition. With the help of a continuous integration (CI) system and test automation, we now deliver real value to our customers every three months.

[ Learn how to apply DevOps principles to succeed with your SAP modernization in TechBeacon's new guide. Plus: Get the SAP HANA migration white paper. ]