The road to a true DevOps and an agile environment requires more than simply adopting new tools and processes. It consists of implementing the most important building blocks of agile development, building a stable continuous integration (CI) infrastructure, and building an automated pipeline that moves deliverables from development to production. The entire build process should be transparent, and it should enable and support development and operations.
This is all part of an overall transformation that involves significant changes in culture; roles and responsibilities; team structure; tools and processes—with Program Management Officers (also called PMOs) being the primary change agents and drivers of this transformation.
What today’s PMO aims to achieve
In my current role as a PMO, I have found that the move to agile and DevOps requires new strategies and practices, and a new way of thinking. When moving to DevOps, many large enterprise R&D teams get stuck at the "Scrumfall." The first building block for moving to continuous delivery (CD) is for recently waterfall-based development teams to adopt agile practices and experience some success.
For example, the team should first adapt to better agile practices within their teams and get a broader perspective of the lifecycle. It’s important for PMO leaders to emphasize that the development lifecycle doesn’t end when releasing the product, but rather getting customer feedback and determining how products are being used in the field continues the lifecycle through to the next release and beyond.
Additionally, operations teams are often used to working in their own silos, with no link back to development or testing—only to production. DevOps gives the operations team a new opportunity to better understand the product and bring valuable feedback to developers. That feedback should ultimately work both ways, and if the dev and ops teams work together on shared goals, they are more likely to achieve them.
Part of a PMO’s responsibility today is to be the "change agent" for encouraging both teams to adopt this type of broader view of products, and for thinking in a new way—from management to the dev and ops teams themselves. The goal is effective program management that pushes for greater agility, having all parties collaborate with each other, and creating increased customer satisfaction.
Three key challenges to driving the transformation
As waterfall-based approaches evolve to new methodologies based on agile and DevOps, there are three main challenges that I see large enterprise teams facing when taking on agile-based projects.
Providing increased value with the release backlog
Product owners (POs) tend to describe big concepts as features in the product backlog, due to long releases that take months to develop. PMOs need to work with POs to change the way features are defined. When moving to CD, features in the backlog should be "minimal marketable features" (MMF), defined primarily by the value they bring to the customer. This is one of the main challenges to overcome when moving to CD; in the process of breaking big concepts into small features, you should be careful not to build small features that provide no value to the customer.
Creating better release quality
As opposed to waterfall/Scrumfall testing, agile regression testing should be part of sprint development; we cannot delay testing of new content, or fixing of new defects or regressions in later stages. When implementing this type of methodology, we often hear developers say that they don’t have enough time to develop content within a sprint. As PMOs, our role is to better educate developers (who like to move fast to develop new content) and product managers (who like to see new content).
Specifically, we want developers to reduce the velocity of new content for the sake of higher quality, and get them involved in fixing defects and working on maintenance as well—not only on developing new content. As a PMO, I won’t push features to production unless they’re developed with good quality (as defined below in the "definition of done").
Helping build vertical teams
In an agile world, teams should be able to take a feature and develop it from start to end, with minimal or no dependencies on other teams. While that's not always possible due to organizational structures, building a vertical team—where all development, testing, and operations capabilities are rolled up into one—is critical to CD success.
New PMO practices for creating change
I want to share some practices for PMOs to get started with in supporting the R&D transformation to DevOps and agile.
Defining new roles and responsibilities
Roles and responsibilities must change and be redefined when moving to a CD methodology. There are different accountabilities and authorities for new roles, with each person having a different set of responsibilities. Redefining these new responsibilities comes with a need for creating a new roles-and-responsibilities matrix, as well.
| Accountability | Responsibility | Authority | |
|---|---|---|---|
| Dev Manager | Execution | Estimation | Release scoping |
| Feature Lead | End-to-end feature | Feature planning | |
| Functional Architect/Product Owner | Prioritized, manageable backlog | Backlog management | Priorities |
| System Architect | System architecture | Design review | Design approval |
| QA Manager | Quality status | Quality coverage | Final quality say |
| Product Manager | Roadmap & Vision | Release goals | Backlog decisions |
| PMO | Release planning | Release tracking | Push to production |
Creating better backlog management
We practice Scaled Agile Framework (SAFe), which uses the concept of a program increment (PI), a set of two-week iterations spanning 8-12 weeks. Our PI lengths are 12 weeks long, and we start planning our next PI 6 weeks in advance. This does not de-focus the whole team from the current release, as only the dev and QA managers, product owners, architects, and feature leads are part of this process. At the end of the PI planning process, we sign off on the release content and make sure that tasks such as release backlog priorities, feature effort estimations, and expected capacity are ready.
Improving release management
While many startups push to production almost daily, as a large enterprise we typically deal with monthly releases or two-week cadences, based on our features’ status. Only features that achieve the definition of done (DoD) can be pushed to production.
We monitor our progress in delivering the signed-off content at the PI level—which in SAFe is called the Program Backlog. For example, the below illustration shows our PI forecast.
The first thing I do here is to make sure that the trend of the ‘done’ items meets the PI timeline. The upper (pink) line is the scope I need to achieve, and the green line is my velocity. The line in the middle shows my planning—which in this case was more optimistic than the actual velocity.
This is a classic case of over planning. The planning was wrong because we calculated higher velocity than actual. Going forward we'll look back on our real velocity and extract the same velocity for our next release.
Sprint lifecycle
To ensure we don’t have any technical debt during our new content development, we have to make sure that our sprint ends with the proper regression testing, security and performance testing, etc. Two days before the sprint ends, we stop developing content to allow for exploratory manual testing, regression fixes and completion of missing automated tests.
This is a great position to be in. The team has matured from committing to too many story points in a sprint, to one that proactively reduces the number of stories that it develops in order to achieve the highest possible quality for the stories that it does develop. Developers often complain that they don't have enough time; but in our case developers have reached the point where they do have enough time, because they have more realistic goals and a well-defined definition of done.
Quality tracking
We track new content defects, regressions, and escaped defects ("after the fact" defects that were found after the user story is finished and tested). New content defects are fixed as part of the DoD of a user story; escaped defects are analyzed and fixed and tests for the fix are added to the test flows of the product; and tests for regression defect fixes are added to the automation suites. We investigate why defects weren’t caught, and apply the results of the investigation to ensure that similar defects will be caught in the future.
Three test automation layers
We have three layers of automation, and each one gives us a layer of confidence for obtaining a stable and reliable automation infrastructure.
- The first layer gives a go-ahead for the development team to continue working
- The second layer gives a go-ahead for QA consumption
- The third layer gives us a release candidate build.
On the road to DevOps we invest a lot in automation, but we need to have processes in place to ensure efficient use of the automation framework and automation test results. The PMO should define these processes to ensure stable CI that gives us quick and reliable feedback.
Working with R&D is only part of the change that a PMO has to implement. PMOs need to ultimately help change R&D practices, invest in automation, and create operations-related improvements. By implementing these and other best practices, PMOs today can help pave the way for large enterprises to achieve true DevOps and agile success.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
