Micro Focus is now part of OpenText. Learn more >

You are here

You are here

How to slash the high cost of software defects

public://pictures/studiobooth_day2_session2_0208_-_techbeacon_0.jpg
Malcolm Isaacs Senior Researcher, Micro Focus
public://pictures/jamesborderick.png
Dr. James Borderick Master Strategist, Micro Focus
 

Software defects continue to make their way into production, despite efforts to detect and resolve them before they are released. Unfortunately, bugs in production software can have a negative impact on the business, regardless of severity.

Even low-impact defects can have a cumulative effect on your brand. But according to a new Harris Poll study of 200 developers and managers in large software suppliers, organizations that use agile and DevOps methodologies have reduced the number of production defects, and the defects that do get through are less serious.

Here are several key takeaways from the report, as well as practical tips to help you reduce both the number and impact of production defects

Agile/DevOps is better than waterfall

Agile software development grew out of frustration with software development methods that had a significant time lag between defining the software's objectives and delivering the final product.

Developers were looking for a way to build working software quickly, and get feedback from users that could help improve the product. This was win-win, because users saw value early on, and developers received faster feedback, which helped them refine their focus.

In agile and DevOps, testing happens earlier, so defects are identified and resolved sooner. While some bugs will continue to escape into production, there are fewer than in waterfall software projects. Furthermore, if you happen to discover a defect in production, you can fix it faster when it's an agile or DevOps project.

Of course, a more definitive study would be obtained by repeating the same project in both agile and waterfall, and comparing the results. But that doesn’t happen, certainly not in industry. Therefore, it's not really possible to state conclusively that there would be fewer production defects in a specific project if the project was done with agile as opposed to be waterfall.

However, it is possible to look at the impact that production defects can have, which is what the Harris poll did.

Three different levels of defects

The survey categorized defect impacts into three levels: low-impact, defects with little or no impact on users or revenue; medium-impact, which affect some users or activities; and high-impact, which have a widespread impact on users, affect key activities, or cause loss of revenue.

Respondents were asked to estimate the distribution of defects found in production across these three impact categories, in waterfall and agile/DevOps projects.


In waterfall, there was an approximately even distribution across the different categories. On average, 38% of defects had a low impact, 32% had a medium impact, and 30% had a high impact. Agile/DevOps projects have a similar proportion of medium-impact defects (33%), but there is a clear tendency toward more low-impact defects (40%), and just 27% of production defects have a high impact.

(Authors' note: While the sample size here is small, and the analysis is representative only of those who completed the survey, these results should still be of interest. We plan to follow up with a more comprehensive study in the future, with a much larger sample size.)

While agile and DevOps do not eliminate defects from appearing in production, any defects that do make it into production are more likely to have a low impact.

No defect is insignificant

Realistically speaking, there will always be defects that escape into production. But is it worth addressing low-impact defects, given that by definition, they have little or no impact on users or revenue? 

The survey presented several statements to respondents, and asked them to rate their level of agreement with each. Some 84% respondents agreed with the statement that "a single low-impact defect usually has a low effect on brand reputation, but many low-impact defects may have a significant effect." At the same time, 47% of respondents considered a defect worth fixing only if it caused an outage or data loss.

The lesson here is that an isolated low-impact defect might not be worth fixing. But if several defects are discovered in production, they may have a cumulative, detrimental effect on your brand's reputation even if they don't disrupt business continuity.

Security risks contribute most to the cost

People often talk about the cost of defects, but rarely do they quantify that cost. The survey listed factors that might contribute to that cost and asked the respondents to rate them. These included:

  • Security risks
  • Severity level
  • Loss of business
  • Damage to reputation
  • Resources required to fix

While each factor contributed in some way to the overall cost, 61% of respondents said that security risks contributed the most. 

The survey categorized defects into three areas: functional, performance, and security. Respondents were asked to rate the contribution of a few elements on the business impact, for each of the defect categories. These elements included:

  • Revenue
  • Brand reputation
  • Regulatory fines
  • Disruption to the team
  • PR and marketing costs

 

In waterfall, functional defects mainly affect revenue, while performance defects can result in regulatory fines. Security defects, however, affect both almost equally.

In agile/DevOps, security defects affect revenue and brand reputation more than anything, and these have more of an effect than do functional and performance defects.

The survey examined these three categories of defect in detail, as well as the impact of defects discovered at different stages of the lifecycle, for both waterfall and agile/DevOps scenarios. 

Go agile and shift left

There is a widely held belief that defects discovered in production are more costly to fix than if found earlier in the process. The data in this study supports this, and while agile/DevOps does not prevent defects from reaching production, those that do get through tend to be lower-impact defects than those produced when following a waterfall development methodology.

Also, it takes less time to fix defects in agile/DevOps than it does with waterfall.

Here are the key takeaways:

  • Fix known low-impact defects before release. Don't ignore them. They may have a cumulative effect on your brand's reputation.
  • Go agile/DevOps as soon as possible. You'll have fewer production defects, and the ones that do make it out there will have a lower impact. In addition, you will be able to fix them faster.
  • Shift testing left. The earlier you test, the sooner you find defects, and the quicker you can fix them.
  • Prioritize security testing. Security defects have the biggest effect and the biggest impact on revenue.

Follow these recommendations and you'll help your organization reduce the number of defects that escape into production, and mitigate the negative affect of defects on your brand. To learn more, download the study here (registration required). 

Read more articles about: App Dev & TestingAgile