In the World Quality Report 2019-20, IT executives said that the primary objective of QA and testing in their organizations is to grow the business and improve business outcomes, beating out last year's top priority of end-user satisfaction. (Note: See the World Quality Report 2020-21 highlights for an update on these trends).
However, a lack of alignment between business goals and quality ambitions is impeding the rate of agile and DevOps adoption.
While automation rates are rising, many obstacles to achieving higher automation levels remain. Meanwhile, the move to security test automation has already borne results, reducing overall application security risk, respondents said.
Those are just a few of the takeaways from this year's report. Here are the key findings—and recommendations—from the full report (and view the World Quality Report 2020-21 here).
QA is helping business growth and improving outcomes
While end-user satisfaction now takes a back seat to contributing to business growth and outcomes as the key QA objective, getting there will be challenging. The share of budget allocated to QA and testing has continued to fall, from 35% in 2015 to 23% this year.
This might seem counterintuitive; after all, if you want to augment business growth, you need to increase investment, not reduce it. But, the report said, higher investments made in QA a few years back are now paying off in better business outcomes and increased growth.
Meanwhile, QA organizations have introduced new tools and techniques over the last few years that are more cost-effective.
Also, testing budgets are difficult to track in agile and DevOps organizations, since the entire dev and test team performs continuous testing. In other words, the true cost of QA is no longer visible in a separate budget line for the QA organization.
Automation rates are rising, providing value
As automation continues to grow, and organizations increase the amount of automation across their testing ecosystems, respondents said that they are getting increased value from automation, such as better control and transparency of test activities, reuse of test cases, and defect detection.
Figure: Challenges encountered in developing applications
Source: World Quality Report 2019-2020
There has been a slight reduction in benefits since last year, however, perhaps because some of the current crop of automation tools are behind the curve in terms of technological advances and business priorities.
But increasing automation is not without hurdles. The main complaint, reported by almost two-thirds of respondents, is that applications change too much with each release, and automation can't keep up. This could be an issue with each organization's level of DevOps maturity, with not having enough control of technology stacks and tools, and with the pace of application change.
AI-based testing, test data management are on the rise
To make testing smarter, more effective and more efficient, organizations are adopting artificial intelligence-based tooling and processes. Compared to last year’s World Quality Report (WQR), it appears that there are fewer AI initiatives this year.
But this may be due to the realization that AI-based tools aren't as mature as they were made out to be. Nonetheless, AI adoption is expected to increase across businesses as a whole.
Another recurring theme in the WQR is the difficulty of provisioning reliable, stable test environments, and the data needed to drive the tests. It seems that Test Environment Management (TEM) and Test Data Management (TDM) are sizable bottlenecks that have been inhibiting automation progress for years.
Removing those logjams will be a powerful game-changer for organizations that are trying to increase their automation and enjoy the benefits it delivers, the report said.
Security now a key objective for QA and testing
In this year’s WQR, security has risen in importance to become a separate theme in the report. Maturity in agile and DevOps processes, along with the introduction of the European Union's General Data Protection Regulation (GDPR) in May 2019, led to increased awareness of security testing as part of development, and to its status as a QA-related objective for organizations.
This includes increasing awareness of security across the team, introducing security testing earlier in the lifecycle, and increasing the security of the software itself.
Although security testing is more prevalent in 2019, survey respondents said their organizations still perform much of that testing manually. One challenge is that there are some steps in test processes that can't be automated, which impedes automation efforts.
Nevertheless, more than half (53%) of respondents said that automation has led to an increase in security, and a reduction in overall risk.
Security will always be chasing new risks and threats, and compliance requirements will continue to get stricter. In response, the report said, organizations must increase their levels of security automation to keep up.
Key recommendations from the WQR
The WQR doesn't just report on the state of quality in the industry; it also offers key recommendations to help organizations achieve higher quality. Among them are:
Build a smart and connected testing ecosystem with intelligent analytics
One challenge agile organizations face is the end-to-end testing of business processes. To accomplish this effectively, and to meet the goal of aligning business goals with quality, organizations should connect their testing ecosystems and use intelligent analytics to detect issues and adapt to changes.
Expand AI-related skillsets within the team
AI is not as mature as last year's WQR suggested. But there are many opportunities to apply AI in testing, such as for doing real-time risk assessments, locating and prioritizing issues, and optimizing testing.
To leverage AI effectively, teams will need to incorporate newer skills and roles, such as AI-experienced QA strategists, data scientists, and test experts.
Raise awareness and visibility of test environments
The biggest challenge in applying testing to agile environments is a lack of appropriate test environments and data. Unfortunately, nothing has changed since last year's WQR in this area.
To realize the benefits and the potential savings that change could bring, the report recommended raising awareness of configuration and integration to help determine virtualization strategies, and to work on issues with availability and incorrect configuration.
With software changes happening so frequently and systems sharing data, a center of excellence specifically for data can help organizations create and maintain real-time test data from production systems. This can also help provide it as required to the teams that need it.
Reimagine test automation as a platform
Test automation isn't at the level that it should be. Reasons vary from a delay in the intelligent analytics that were promised to the disparate nature of automation, with lots of tools that aren't living up to expectations.
To increase levels of test automation, consider it less as a capability and more as a platform, share tools and functions that work intelligently from end to end, and ensure it's all driven by business objectives.
Raise your game on security
QA teams should increase automation in security testing. In this way you can run more tests more quickly, which, the survey said, is likely to reduce risk. Factor in security and security testing starting in the earliest stages of development.
Get aligned with your business's goals
The biggest takeaway from the 2019-20 WQR is that a lack of alignment between business goals and quality ambitions is impeding the rate of agile and DevOps adoption. The needs of the business should drive development. Those needs require that teams connect their testing ecosystems to create a smart, business-driven test automation platform.
As testing and quality become increasingly embedded within everything your organization does, your teams must expand their capabilities and skillsets across automation, test data and test environment management, AI, and security.
