You are here

You are here

Test automation tools: Top trends and challenges for 2020

public://pictures/linda_rosencrance_photo-4.jpg
Linda Rosencrance Freelance writer/editor
 

Test automation tools have been steadily evolving—a trend that shows no sign of slowing down in the coming year. Several key advances to watch for over the next 12 months should make life easier for test automation engineers, consultants and tool vendors say, while others to watch out for are only likely to add confusion.

Expect to see more artificial intelligence (AI) capabilities, a potential ramp-up of robotic process automation (RPA) in testing, continued fragmentation in the test automation tools market, and the steady rise of secure DevOps, or DevSecOps.

Here's what your team needs to know about the state of test automation tools.

AI-based testing capabilities will go mainstream

"Although some want to ignore it and some embrace it, artificial intelligence for software testing is now a 'thing,'" said Jason Arbon, CEO at test.ai. "It's no longer a curiosity, and it can't be ignored.'

Just two years ago, only a few brave souls were applying AI to their testing problems, Arbon said. But by the end of 2019, multi-person test teams at large enterprises were turning to internally developed tools or applying vendor-based tools. “2019 was the year to educate yourself on the basics of AI,” he added.

“If you don’t at least have an intelligent-sounding opinion on AI for testing, you are falling behind as a leader and technologist.”
Jason Arbon

A few vendors will go “all-electric” and have truly AI-first solutions in 2020, Arbon predicted. Expect to see the rise of AI-first tools built by leading test teams for application-specific or internal testing challenges.

These will remain quiet, because they won’t need much marketing, or be viewed as secret, as they are competitive advantages, he said.

It seems as if everyone is talking about AI in some shape or form, said Renato Quedas, director of product management, functional testing portfolio, at Micro Focus. 2019 was the year people started talking about AI. But this year, at least on the commercial side, what has started to pop up more often is the rise of AI commentary, he said.

“Most of the time, the use of AI capabilities has centered around regression analysis, the physical analysis where you were looking at trends and using AI to help you better plan your test execution,” Quedas said. 

In 2020, people will start using those capabilities to solve hardcore problems in automation, such as increasing the resilience of the script.

“You’re going to see people trying to use AI to solve problems in syncing, tests, and application changes."
Renato Quedas

AI will also be used to help people automate more and to bring automation to less technical people. “For instance, you're going to see people talking about natural-language processing as a way for nontechnical users to write automation using plain English,” said Quedas.

The World Quality Report 2019–2020 survey from Capgemini/Sogeti found that organizations are already adopting AI-based tooling and processes to make testing smarter, more effective, and more efficient. But there were fewer AI initiatives in 2019 when compared to the responses in the 2018-2019 World Quality Report (WQR). That may be because many organizations realized at the time that new AI-based tool and features weren't as mature as they were led to believe. Overall, though, most respondents said they expect the adoption of AI to increase—and that trend will increase across all businesses, the report predicted.

Although AI projects may not be as prevalent as the 2018-2019 WQR suggested, there are still many opportunities to apply AI in testing. “[Artificial intelligence] can conduct real-time risk assessments; it can find the issues that need to be addressed; it can prioritize them; and it can optimize an approach to create testing that is both predictable and fit for purpose,” according to the 2019-2020 WQR.

To make the most of AI, however, teams need to incorporate newer skills and roles, such as AI-experienced quality assurance (QA) strategists, data scientists, and test experts, the report states.

Open-source tools will fall behind in the AI features race

“Open-source tools will continue to suffer from a lack of resources,” said test.ai's Arbon. “The folks at the core of these projects are genuinely just very talented and good humans, but so few dollars are actually invested and the open process is so slow."

"Open source is at a large disadvantage with the rapid evolution of AI-based approaches.”
—Jason Arbon

The new AI innovations are both very expensive to build and "monetizable," so there will be a significant disincentive to open source the new wave of AI-based test automation advances this year, he said.

RPA won't have quite as big an impact on test automation as it will in other areas

The one thing that stands out for Michael Fritzius, president of Arch DevOps LLC, has been the sudden emergence of RPA, a technology that automates repetitive, manual tasks across the business, from the IT help desk to HR, to increase productivity and cut expenses.

“It’s just going viral, and I haven’t quite figured out why, other than people are wanting to be relevant and it’s the new thing,” he said. “So everybody’s, ‘Ooh, I’ve got to learn this thing so that I can stay employable.’”

This year, he expects to see a fairly sharp “hockey stick ramp-up,” in the adoption of RPA-related technologies. “Right now, there are a few big players in RPA out there, but I think there are going to be a lot more of them in 2020, because they’re going to be getting on the bandwagon,” he said.

Arbon of test.ai, however, isn't quite sure RPA is going to live up to the hype when it comes to applying the technology to test automation. RPA startups have raised a lot of marketing money, and they are exploring the testing market, he said. RPA is good at automating apps that rarely change, often legacy systems, and doing the same sequence over and over. Great test automation, on the other hand, has to deal with constantly changing apps never seen before, and delivering coverage, not repetition.

"It will take a full year for the market to figure that out,” he said.

RPA tools have the potential to blend "record and playback" techniques with RPA's focus on data parameterization, and building custom integrations with standard test reporting and execution (CI/CD) systems, according to Arbon.

"For testers that have rarely changing UI/UX flows in their products—and testing is very data-driven—RPA may prove useful," he said. "But RPA seems destined to be another flash in the pan of hope for the bulk of testing problems, and will further fragment the testing tools environment." Which leads to the next prediction.

Fragmentation in the test automation tools market will create more user confusion

The adoption of commercial tools this year will be slower than many had hoped for, Arbon predicted. Testers will be faced with too many incompatible, or partial, solutions, and both testers and test managers will begin to look for one platform that can do it all. 

Today's software testing needs to cover a large mix of automated and manual testing across a wide array of platforms, programing languages, and software layers, but most of the new testing tools today only address a subset of testing problems and infrastructure, he said.

For example, some of the coolest new testing tools only support desktop web and don't have a story for mobile; some focus only on APIs and microservices; others only deal with visual testing with screenshots; and still others work only with JavaScript, Arbon said.

"These are all great tools, but they only make it more difficult to get a full picture of the quality of any product, and it forces testers to evaluate and leverage many different tools to do their jobs," he said.

Secure DevOps will continue its steady rise as a key focus area

The DevSecOps trend, which involves adding security elements earlier in the development process, will continue to grow this year, said Jay Lyman, principal analyst at 451 Research LLC. To be useful for developers, these security elements must be automated and integrated, he said.

“This has to do with a lot of the testing tools—things like vulnerability assessment, software composition analysis, dynamic application security testing, interactive application security testing, static application security testing—all of these different types of apps security testing. I think we’re going to see more of that.”
Jay Lyman

However, the integration of security into the software development pipeline, or integration into the developer spaces, hasn't taken off as experts had anticipated, probably due to limitations in the tools and the limitations within the development and testing cultures in many organizations, said Erdem Menges, former product marketing manager for Micro Focus's Fortify application security suite and now director of security product marketing at GitHub.

“From a tools capability and maybe from a technology standpoint, the tools and the technologies are still delivering some noise, which can be frustrating for developers and QA testers. The tools need to provide a better outcome. The tools themselves will definitely improve in 2020. They will have better and tighter integration capabilities, and this will open up a lot of new doors automation and integration."
Erdem Menges

The industry is moving toward a state where there will be fewer false positives and lower noise overall, said Menges. All the leading application security vendors are working to optimize rule sets, adapt machine learning algorithms to validate results, and prioritize scan results to serve this purpose.

From a cultural or social perspective, developers are still struggling to take ownership of security, even though it’s about securing the codes that they’re creating, said Menges. 

But since developers don’t have backgrounds in security, and they’re more geared toward meeting deadlines and delivering features, they still need some time to embrace security as one of their core responsibilities. This is why developers are a little hesitant to implement these tools or adopt these processes in their development environments.

“I think it is going to change and improve for the better, but it won’t be solved completely,” Menges said. One reason why it will improve is that there’s a shift in terms of investments and strategy in bigger organizations that are more mature in terms of application security and that are making application security as an available service to developers, he added.

This approach is having better results than just having security as a checkpoint or a gate before organizations release their applications, he said. It's also having better results than just asking developers to choose whatever security service they would like to use, and then just leaving the running or the operations to developers.

“Because that’s not what [developers] excel at," Menges said. "They generally excel at creating great software, and then they could use security as a service to improve their overall process.”

Expect the current tools to improve over the next 12 months, with better, tighter integration capabilities that will open up a lot of new doors for automation and integration.

“[By] embedding security checks in an essential part of the continuous integration and continuous delivery pipeline, we will be able to provide more results at earlier stages in development and we will be able to provide the results faster to developers so they can start working on fixing those issues."
—Erdem Menges

Test suites will combine functional and performance testing 

One trend that didn’t roll out as fast as it could have in 2019 but that's on track to accelerate this year is the evolution of test suites that let testers address test automation from different angles, said Diego Lo Giudice, vice president and principal analyst at Forrester Research.

Specifically, he said, these platforms will include the capability to pull together a functional test automation with a performance test. "There’s convergence between functional and performance testing to enable performance testing to be done earlier and sooner in the development cycle,” he said.

It’s not just a tool concept, but a suite concept that several tool vendors are ready and lined up to do because they’ve already acquired or built some of these different components, Lo Giudice said.

“In 2020 it won't necessarily be the case that these vendors' tools will be sold in that way, but it’s going in that direction, because our clients don’t want to buy a functional test automation tool from one company, a shift-left performance testing tool from another, and a test data management tool from another company."
Diego Lo Giudice

The big platforms vendors will take a big leap forward in testing

In 2020, you'll see platform teams—the Googles, Microsofts, Apples, Amazons—start defining, enabling and requiring basic testing of all the applications that run on their platforms, said test.ai's Arbon.

Their customers don't care much about operating system improvements, faster hardware, or even device design anymore. They have largely converged, and are at parity in the eyes of most users, he said. 

The true differentiation between operating systems and devices for consumers today is the quality of the user experience within their favorite apps. The platform teams now have dedicated teams to reach out to top apps to help them with crash reports or major issues showing up in the app store reviews, Arbon said.

"Anecdotally, I've heard the biggest reason people return their phones to the carriers and ask for new ones is that their favorite apps keep crashing—and there are tons of shipping containers full of them heading across the ocean," he said.

The platform teams also have the money, while most app teams don't have enough money or talent to test the applications themselves. Worse, much of that testing by individual app teams is basic, redundant, and wasteful. Platform teams are finally waking up to these issues, Arbon said.

In 2019, they all made progress in basic testing by scanning every app in their app stores for basic crash, security, privacy, and adware issues, he said. That trend will pick up speed this year.

“The platforms care more about the quality of apps than the individual app team developers around the world do. They have the money, they have the technology, they won’t leave app testing up to little, ad hoc, underfunded test teams spread around the world."
—Jason Arbon

Get ready for massive progress from the ultimate vendors in test automation—the platform companies, Arbon said.

Self-healing capabilities will continue a slow evolution

Although many test automation tools introduced some self-healing features in 2019, the technology is still in its infancy, said Joe Colantonio, founder, Test Guild LLC.

For example, Selenium IDE has automatic fallback locators. “When you record with Selenium IDE, not only does it record what it thinks is the main identifier for your element, it also automatically records all of the other ways you can use to identify that element,” he said.

ReTest released an open-source solution, recheck-web, and although it’s not “self-healing” it does make tests run more reliably, Colantonio said.

“Other companies that didn't get there in 2019, will continue to evolve their applications to eventually support self-healing. In 2020, expect more test tools to incorporate this feature as well as other AI/machine-learning functions, such as smart test execution."
Joe Colantonio

Look ahead before you leap

As you review your test automation toolbox and capabilities in the coming year, be mindful of these changes and challenges that lie ahead. Factor in the increased integration and use of AI capabilities, the potential ramp-up of RPA, the steady increase in secure DevOps, and other trends described above—and make sure the developers of the tools you plan to use are investing in these areas— before making any product purchase decisions. 

Keep learning