The premise behind continuous testing (CT) is simple enough: Test in lockstep with development so you don’t get blindsided by surprise bugs late in the game. Makes sense. So how do you do it?
Dave Allen, director of DevOps engineering at Sungard Availability Services, starts his answer with an anecdote. “Years ago when I was working for a hardware vendor, I had one of our testers open a machine, wedge a screwdriver onto the backplane, and try to boot. When it didn’t work, he filed a defect."
“Back then everyone was upset about the whole thing, but today, it’s expected to think in terms of ‘What is going to happen when something completely unexpected goes wrong? Will the system endure the calamity or not?’" Allen said.
"We now think about rolling failures, component failures, human failures, and more to find a way to make the system continue servicing the end users seamlessly.”
—Dave Allen
TechBeacon asked experts to weigh in. Here is a sobering look at the state of continuous testing.
Automation is everywhere
Testing has become a key part of the development process, and the front lines of testing, for several years, have focused on imbuing the testing process with automation.
Automated testing has become essential for most businesses. For example, Google says it has 4.2 million individual tests running continuously (both before and after code submission), with a total of 150 million test executions each day. Virtually all of that is automated.
Boris Chen, co-founder and vice president of engineering at tCell, says that automation doesn’t merely save one person’s time; it makes the entire team function faster.
"If you had infinite resources and could hire as many people as necessary to run automated tests by hand, you would still fail to achieve the benefits provided by automation. That is because automation isn’t just about speed. It’s about consistency and letting machines do what machines do best.”
—Boris Chen
Sungard’s Allen explains his own automation-heavy continuos testing methodology in more detail. “CT implies a robust automation capability and focus that is table stakes, followed by the ability to test new functionality during a sprint via automation. One needs a solid tool, connected to the build environment via something like Jenkins."
"There must be a set of sanity and regression tests that are triggered off code check-ins and builds. The organization must have a branching strategy, and notably the master, hot fix, release, and development branches must have build triggers that also result in testing triggers," he notes.
"The goal is to automatically deploy and test the deliverable to ensure that all components and functions are working in a deployed environment without any human intervention."
—Dave Allen
That’s not to say that automated testing is easy, particularly in a legacy environment, says Subu Baskaran, senior product manager for Sencha. He says that despite the general availability of frameworks for testing early in the cycle, software development teams that are still maintaining legacy applications find it hard to go back and write unit or end-to-end tests.
"The millions of lines of code make it extremely difficult for teams to think about unit testing, as that will severely hamper new feature development. Also, legacy applications have inherent complexities that make end-to-end testing very slow, vague, and brittle. Hence, teams that maintain legacy applications resort to manual testing.”
—Subu Baskaran
What’s next: Beyond automation
Is automation the only solution to enable continuous testing? Is manual testing really so bad? “The simplistic view of continuous testing is that this is a repeatable testing process that is done the same way every time,” says Matt Heusser, managing director of Excelon Development, a CT consultant, and a regular contributor to TechBeacon.
But the industry is slowly evolving that mindset as it comes to the realization that this simplistic view isn’t always effective at finding bugs, says Heusser. The biggest problem with continuous testing, he says, is that as a project becomes more complex, so does the testing process associated with it. A test run may take only a few minutes to complete when an application is fresh, but after several months of work and rework, an automated testing run can become so complicated that it takes a day or more to complete. “Eventually, devs just give up on it and stop testing,” he says.
“Today, the broader view is that we still need to test every feature, but we know we can’t automate everything, so we fold in different facets of testing that aren’t as easily automated.”
—Matt Heusser
Michael Pilaeten, learning and development manager at CTG, concurs and points to specific challenges around the automation of UI testing.
"Too many companies still focus on the ‘wrong’ side of automation—at the GUI level. While automating the process in the GUI itself can seem to be a point-and-click activity, GUI test automation is very expensive if it’s not preceded by the correct set of automated unit and API tests.”
—Michael Pilaeten
Isabel Vilacides, QA manager at CloudBees, notes that the industry is also moving towards distributed systems where your product is made up of different services, and where testing integrations is essential to success.
“End-to-end testing is expensive to run and maintain, and if you have several services integrating, it is challenging to keep tests stable. Monitoring then plays a key role and becomes the new testing, reducing the need for end-to-end testing even further and converting the testing pyramid into an hourglass.”
—Isabel Vilacides
Common problems with continuous testing
In practical terms, continuous testing can be fraught with problems.
Two scenarios are particularly common, says Heusser. In the first, a tester finds that the testing script is generating a mountain of bugs. After researching the issue, the tester finds that many of these are problems with the testing process, not legitimate errors in the code. The solution, Heusser says, is to edit and prune the testing script to focus on true errors.
At Google, so-called test flakiness, a problem in which a test reports both a pass and a fail result when testing the same code, is a big problem, accounting for 1.5% of all tests. “Some level of flakiness” is associated with almost 16% of tests, a huge proportion that indicates that testing protocols are often badly designed. Detecting problems with testing (and quarantining flaky tests) is seemingly as big a job as detecting problems with production code.
A second, and perhaps more noxious, problem, says Heusser, is when testing reveals no errors at all. In this case, testing needs to be beefed up since, as every developer knows all too well, bug-free code just doesn’t exist.
No ultimate fix here
Of course, the biggest problem of all is the assumption that testing is the panacea that will solve all of your coding issues.
“Most people, they think they need to do a lot of CT because the code is buggy. What they really need to do is figure out how to write code with fewer bugs. Testing is just a scale. If you’re fat and you step on it 10 times a day, it’s still going to say you’re overweight every time. It doesn’t fix the problem.”
—Matt Heusser
From a tactical perspective, Heusser says, those embarking on a continuous testing strategy should consider the principles published at context-driven-testing.com—principles that state (in part) that there is no universal best practice when it comes to testing but that best practices can only be determined via the context of an individual project.
It’s very hard to have sustained, long-term success in continuous testing, he says.
"It’s easy to do a cool demo, but keeping it up is difficult—which is true with most methodologies, like agile and Scrum. I tell people, ‘Come back and talk to me in three years after you get started.’”
—Matt Heusser
Stay calm and carry on
The road to continuous testing is exactly that: a journey, not a destination. One thing that becomes clear upon talking to people who have successfully implemented it is that there are no absolutes.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
