Imagine that you have been tapped to be the testing tsar for a multi-million dollar business-critical IT project requiring thousands of hours of work. Owning a super-sized testing project can turn out to be either a career-defining opportunity or a professional nightmare. I’ve experienced this scenario several times in my tenure as a quality assurance and testing leader for several top companies, including Equifax, FICO, T-Mobile USA, Healthcare IT Leaders and Run Consultants.
I recall getting one such call from the CIO of T-Mobile, then coming home and telling my wife that my family won’t see much of me for the next three months. And even though I had to work many nights and weekends, it proved to be an invaluable experience, and I’d like to share some of the lessons I learned during this and other projects to hopefully make the path smoother for those of you who are facing software testing projects of exceptional scope, scale and size.
Whether you work for a startup or a large corporation, you have likely encountered high-visibility, high-risk, high-profile projects that put your test organization, preparation and execution skills to the test. Although most of the same guiding principles and practices apply for projects of all sizes, these mega-monster projects require a fair share of special considerations, careful planning, and organizational and cultural adjustments.
Just as a chef requires special preparations to cater a large party, the head of testing must make arrangements to face the unique challenges and requirements of a large-scale testing project and lay the groundwork not only within the software testing team, but across the entire organization to ensure that the testing phase goes smoothly and doesn’t cause project delays—or business failures.
To be successful in a large-scale testing project, you first must have an agreement within your organization that delivering high quality is not just about finding bugs. It’s about delivering an outstanding customer experience, exhibiting operational excellence, and in some cases, making work easier for front-line employees.
Why do mega-testing projects happen?
Mega-testing projects may be required due to business, regulatory or technology factors. Here are just a few factors within each area that may push you into a mega-project.
On the business side, the most common factors are mergers and consolidations, new product introductions and creation of inter-company relationships that alter essential business transactions. When I led Software QA at Fair Isaac (FICO), a mega testing project was born when the executive team decided to develop an Enterprise Decision Suite, integrating several product lines that came together through an aggressive acquisition strategy. We had a multinational testing team spread across three continents. Our team used a risk-based testing approach to determine the priority of the test cases to run. Due to the enormous scale of the credit data involved, we had to focus on test automation and performance testing. The testing team used a combination of open source and commercial test automation tools. One of the most memorable automation tests ran on multiple platforms and multiple languages. To keep everyone aligned, we kept all of the test results and defects in a central repository, which allowed us to provide standard metrics in a real-time dashboard.
Then there’s the regulatory side: changes in compliance requirements, new regulations and governance practices that cause your company to revise its practices and processes. I was recently involved in a mega testing project at a large hospital network, which was required by the government to upgrade all of its systems to handle a new set of disease codes (ICD-10). The hospital system had only a limited number of trained testers and didn’t even have suitable workspace to accommodate the 80-plus extra testers required to carry out the work, so
Healthcare IT Leaders provided workspace and housing for additional testing professionals with specialized skills. We decided to use rural office locations in North Carolina and Georgia to keep costs low. The entire cross-functional team agreed upfront on success criteria of the testing effort. The hospital TCoE leader provided clear roles and responsibilities, which was critical to keep focus for each team member.
As we move into the digital age, one of the most common drivers for creating extra-large testing projects is technology advancements: large-scale platform upgrades, system integrations, introduction of new solutions to support new product functionality and data center migration/consolidation initiatives. All of these – or most likely a combination of these factors – can significantly impact the way you structure and run your testing projects.
One of my most memorable mega testing projects was $1 billion-plus IT modernization initiative at T-Mobile USA, which included a data center migration and web customer care and retail application overhaul. With the need to run 250,000 tests to validate 100-plus applications and weekly software releases, we didn’t have enough full-time testers and needed to quadruple the size of the team. We chose a specialized telecom testing services provider, Amdocs, for their expertise in our industry, knowledge of business processes and library of tools and automation solutions to not only add bodies to the testing team, but optimize and accelerate the entire testing project. Together, we built a team structure based on the follow-the-sun model, utilizing our internal resources and the Amdocs onsite and offshore testers, completing the project ahead of time with excellent results.
A small team of test analysts, test environment engineers and test architects designed behavioral models and data flow diagrams for each of the critical business processes that would need to be tested in a release. Together with representatives from the business team, we developed a test strategy for functional and non-functional testing, reusing many test scenarios from a library of recent telecom projects. For planning and scheduling, we created a day-by-day calendar of testing across several environments that included manual and automated testing in both agile and waterfall projects. This was important because we needed to test several complex business scenarios, including multiple billing cycles and time-based discounts, and we built a process to shift logical systems dates in the billing platform. The team executed end-to-end tests to verify complex business processes, and operational goals were met.
The stakes on the project were so high, and the entire cross-functional team, including executives, reported to the command center for the night of deployment testing, working side by side with the development and operations teams. We had a minute-by-minute plan for testing, deployment and troubleshooting. Not only was the project an overwhelming success, the successful partnership helped create a sustainable culture of transparency, accountability, improved quality controls and reduced business risk. We planned for the command center to be open for four weeks to quickly remediate issues, but the war rooms were closed after a few days because only a few post-production issues came in.
Keys to success: collaboration, communication and continuous improvement
Collaboration, communication, transparency: Without these three ingredients, it may be impossible to achieve good business outcomes, especially in a large-scale project.
When you have a large team, leadership and accountability are extremely important. Agile teams must scale their approach to the enterprise level with daily team calls, the Scrum of Scrums and steering committee meetings led by the product owners. It is helpful to have a “lessons learned” meeting after a mega testing project to review what worked and what can be done better; however continuous improvement goes beyond release-to-release process enhancements. Innovative leaders are always looking to learn from others in your space. If you are working with a specialized testing service provider, a large-scale testing project may be a great opportunity to review industry best practices, look at trends and prepare for the future.
Every testing project is unique and requires special considerations and planning, but it’s the mega projects, the monsters of the testing world, that can help your team grow professionally. I hope that reading about my experiences and observations has helped bring to light some distinctive properties of large-scale testing and will help you effectively plan for the challenging but extraordinarily rewarding experience of mega testing.
Image: timlewisnm/Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
