What makes a good QA tester? 4 KPIs essential to software testing

In QA organizations today, a tester must have technical know-how, good communication skills, and attention to detail. We know that a tester’s main responsibility is to test the software as it's developed to ensure that the product meets the quality standards expected.  Apart from that, it’s difficult to measure what exactly makes a good QA tester.

But the importance of QA in today’s R&D teams is growing, as the role has evolved into more than simply finding defects to protect the corporate imageAs a result, testers have to be more productive and more efficient than ever, and change their mindset to one that focuses on quality over quantity. It’s not just about finding bugs; it’s about continuing to measure and improve, and finding the right bugs in order to make the end-user experience better.

Here are some of the key performance indicators (KPIs) we at HPE use to measure our own testing efforts, what we’ve learned from each of them, and how our team improved its efficiency and productivity as a result.

Gartner Magic Quadrant for Software Test Automation

The metrics that only appear to matter

Agile teams test continuously. It’s the only way to ensure that the features implemented during a given iteration or sprint are getting done. Fast-paced agile teams often use rather obvious metrics as quality KPIs, but in reality, these KPIs don’t contribute as much as they would appear to at first glance.

  1. Total number of defects. As mentioned above, the quantity of defects is no longer as relevant as it once was. You may prefer that a single bug in your application critical path is found and remedied on the spot rather than just logging the other 40. A bug count gives you absolutely no indication of end-user satisfaction.

  2. Direct coverage. This is the automated and/or manual test coverage of a feature. In agile methods, however, this criterion is less significant than it was previously. What’s done is done; there is no possibility of missed coverage. If any feature is not covered by tests in a particular sprint, it will not be considered "done," and it will be moved to the next sprint until it’s completed. This ensures that all features and functions are continuously and fully tested.

  3. Unit testing coverage. Measuring unit testing coverage doesn’t necessarily contribute to the quality of the product. These days, there are plenty of technical ways to make sure the unit test coverage is complete. There will always be a trade-off between achieving 100% test coverage as opposed to actually testing the code that matters. While it is possible to test all of your code, the value of your tests will likely diminish as you approach this limit, given the tendency to write more meaningless tests for the sake of satisfying the coverage requirement. And even if you do have high unit test coverage, you’re still not guaranteed high quality in the final product. That's because the majority of defects are detected during integration testing and end-to-end testing, which is done after the unit tests have run.

The metrics that really matter: Quality beats quantity

There are certain KPIs that should be measured. Keep in mind that these are only a few of the testing-related KPIs we think are important; there are other performance indicators to consider as well.

  1. Percentage of high/critical and escaped defects. Any experienced tester can find 20 bugs without even knowing the application. The key is to make sure the tester is focused on what really matters to the customer. Escaped defects are those that are missed by the tester and found by the end user. Although they should not be a cause for panic if discovered by the customer, the team must investigate and learn from them to improve and prevent them from recurring. The percentage of high/critical defects also comes into play here. This KPI ensures that you’re focusing on the critical ones.

  2. Time to test. This is the time it takes a backlog item to get from "in-testing" to "done." In agile terms, this is how much time is needed to test a user story. We found this to be an important KPI for measuring the efficiency of the tester and determining how quickly they are completing tasks. This is another way to measure the tester’s effectiveness and efficiency, as well as the complexity of the feature.

  3. Defect resolution time. As the name implies, this KPI measures and qualifies the tester’s responsibility and ownership for their bugs. Our responsibility as a QA team looking for quality is to track all of the resolution time, and not just the time it takes to find the bug or verify the fix. We have to track and make sure the bug was fixed the way it was supposed to be fixed, and close out the issue in a reasonable time to satisfy users.

  4. Percentage of rejected defects. These are defects found in the product but not accepted by the developer as defects. A large number of rejected defects indicates whether the developer and tester are on the same page about the feature’s functionality and its purpose.

Efficiency and effectiveness

Aside from the KPIs mentioned above, software test efficiency and software test effectiveness are equally important ways of measuring the success of your testers.

  • Software test efficiency is the number of test cases executed divided by a unit of time (generally per hour). More simply put, software test efficiency is the organization’s internal gauge of how many resources were consumed and how many of those resources were utilized. Test efficiency tests the amount of code and testing resources required by a program to perform a particular function.

  • Software testing effectiveness can be measured if the goal and purpose of the testing effort is clearly defined. Above all, testing effectiveness should ensure the reliability of the software and that the user’s expectations are met.

A final note

The Systems Sciences Institute at IBM has reported that the cost to fix an error found after product release is four to five times higher than the cost of fixing one uncovered during design. This high price for fixing bugs can be better managed and prevented with the right KPI measurements and testing metrics.

Using the approaches I discussed in this article, you can monitor and analyze your agile testers’ results to generate better performance, improve their efficiency, and provide meaningful and measurable ROI on your testing efforts.

What quality KPIs do you use, and how have they helped you measure ROI in your QA efforts? Let us hear from you in the comments section below.

Gartner Magic Quadrant for Software Test Automation
Topics: Dev & Test