In 2016, software testers moved away from using broad suites for task automation in favor of best-of-breed tools seen as more capable of handling the demands of DevOps and agile development. That trend is likely to continue in the coming year.
A growing demand for testing automation tools
The increased interest in automated testing tools tracks with overall changes in software development. "Continuous integration, DevOps and agile have exploded in the last year and a half," says Rich Clyde, vice president of application lifecycle management engineering at revision control system maker Perforce. And that explosion is driving interest in test automation, because the more you use DevOps, the more you need test automation.
"DevOps doesn’t work without test automation," says Georg Thurner, product manager at test automation software vendor Tricentis. "In reality, it should be called DevTestOps. Continuous testing is required to ensure that you put a version into production without critical bugs, or at least with a very low chance of critical bugs."
"Continuous test automation is a must for companies, as classical manual testing can't cope with the demands of DevOps." —Georg Thurner
Test automation upstarts on the rise
Tricentis is just one in a group of up-and-coming test automation vendors that have emerged as an alternative to the large testing suites. Others include Ranorex, Worksoft and Testplant.
Large, broad suites of testing software have been losing favor among users. "People aren't looking for a single solution that's good enough," says Chris Kraus, director of product management at Worksoft, which makes an automated business process validation platform. "They're looking for a best-in-breed solution for what they need to do. People have multiple testing technologies, versus a single testing technology."
Open source tools are taking hold
In addition to the new commercial tool players, open source software is making its mark in the space, too, particularly in the areas of API and user interface (UI) testing. Such tools include SoapUI, for API testing; and Selenium and Appium, for UI testing.
Nikhil Kaul, manager at SmartBear Software, maker of open source API testing and software development tools, says the market share of open source tools, such as Selenium, will continue to grow. "Selenium is emerging as the W3 standard, and major vendors such as Mozilla and Chrome are putting their strength behind it," he says .
"Open source tools have given QA engineers great flexibility in creating automated test cases," Kaul says. He points to Selenium, which supports multiple browsers, operating systems, languages and testing frameworks, as a prime example. "QA engineers can thereby choose the bindings as per their needs."
"Similarly, API testing tools such as SoapUI offer freedom for functional API testing, without vendor lock-in." —Nikhil Kaul
But Thurner, advocating for commercial tools such as his company offers, has a word of caution for organizations inclined to use open source tools for testing. "The success in using open source to test software often depends on the community behind the open source project," he says. "If it is an active one, you will find solutions for almost any problem you have. If not, or if the community is shrinking, you might end up having to solve all the issues by yourself, without any help from the outside."
Mobile is changing the automated testing game
Interest in automated testing tools has also been spurred by an increased demand for mobile applications. Testing mobile apps requires an infrastructure that may need to support hundreds of browsers, and scores of operating systems, all which must be available as needed. And the infrastructure must be able to dynamically scale to meet the needs of an organization's development and Q&A teams. "Setting up an in-house test infrastructure that meets those needs can be a challenging, time taking, and costly process," Kaul says.
"Companies during the year tried to address the problem of testing many devices with the fewest amount of scripts, and without having to purchase all of the hardware, so a number of cloud-based solutions were launched," Perforce's Clyde says.
Performance testing continued to evolve during the past year. "The holiday season was littered with stories of performance issues," Clyde says.
"Companies really need to retool around stronger analytic platforms to aid in understanding where performance is affecting returns, and not just think of performance testing as a phase." —Rich Clyde
This, he says, is part of the trend toward the integration of performance testing with application performance management and real user monitoring.
Demonstrating the value of test automation tools
While testing automation continues to gain traction, getting full value from the technology can be challenging—especially in organizations that haven't fully deserted the waterfall approach to software development, or that view automation as a UI-only initiative. For those organizations, testing still means finding bugs, while in organizations using DevOps and agile it means driving quality and preventing bugs in the first place.
But discounting the value of test automation can be costly to an organization in the current software development environment. "Executives that don't pay attention to testing are opening their business up to losses," says Alex Chernyak, president and chief technology officer at Zaptest, maker of the eponymously named open source test automation tool. "It's a loss that's unrecoverable, because customers that go to a competitor will seldom come back to you."
"Competition is brutal. Clients and consumers are very unforgiving for poorly developed and produced software." —Alex Chernyak
Almost as bad as company brass who don't appreciate the importance of software test automation are the executives who misunderstand the benefits. But there is one exception: Without automation, mobile software can make testing costs spin out of control.
Before mobile, tests could comfortably be performed on a single platform or two. With mobile, tests may need to be performed on many combinations of devices, operating systems, builds, and carriers.
Android fragmentation: How to deal
Fragmentation in the Android market has created big problems for testers, because at any given time several versions of the operating system are running on substantial numbers of phones in the market. "The additional fragmentation resulting from screen size, device type and hardware specifications can easily make testing messier," Kaul explains.
"And with the Internet of Things (IoT), and gadgets such as Fitbit becoming more prominent, rampant device fragmentation means that testing mobile applications needs to be device-specific, often resulting in longer testing cycles than that of desktop or Web," Kaul adds.
For test automation tools, the IoT is one of the biggest challenges. "Every product is very unique, so you can standardize only to a degree," Clyde says. "The hardware is unique, the operating system is usually a variant of Linux, and the user experience is customized from device to device to device."
Skills can also be a challenge for companies moving from manual testing to automated testing. "Test automation requires technical skills—the required level depends which test automation is used," Thurner explains. "Not all manual testers are able to make the necessary transition, especially if the test automation tool demands that an experienced and skilled developer use it to be successful."
Why testing tools still can’t replace humans
While testing automation software can speed up the testing process, and reduce human error in that process, these tools can't totally eliminate "wetware" from the process. The reason? While automation is good at validating code, it’s less adept at what-if scenarios.
It's possible for a feature to pass an automated test, but fail to meet standards for usability and consistency. "A strong understanding of perspective and context, which is powered by the human mind in manual testing, is [essential] in many cases," Kaul says.
"Human perception and intuitiveness can be impossible to replicate using an automated script." —Nikhil Kaul
The human mind is needed to design tests in the first place. "Test automation will not replace the need for defining the right test cases," Thurner says. "If you test the wrong things, or if the test coverage is low, test automation will not help. So human experts are still very important in designing software tests."
But just as automated testing can benefit from human intervention, humans can benefit from automation, says Rick Riccetti, chief technology officer at Perforce.
"One of the beauties of test automation is that when you automate a lot of the easy stuff, you unlock your test engineers to do higher level functions and think up new ways to challenge an application." —Rick Riccetti
What's next for test automation tools
Testing automation acquisitions are expected to continue, and organizations that are getting to the DevOps table late are expected to hit a wall due to inadequate preparation.
Challenges posed by testing apps in a variety of environments, especially mobile and IoT apps, will further fuel the use of containers, which offer scalability and portability benefits. "From an agile testing perspective, Docker will continue to get a lot of attention in 2017, especially when it comes to solving testing challenges arising from different environment configurations in development, testing and production," Kaul predicts.
The year may also see more standards around the IoT. "I wouldn't be surprised to see international bodies putting up testing requirements around safety and security that will drive some testing automation in 2017 and beyond," says Clyde.
Image credit: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
