Return on Investment (ROI)

Return on Investment (ROI)

It’s so easy to find case studies which show that finding and fixing bugs before going live is usually cheaper than doing it afterwards that I won’t even take your time by quoting any.  The question is, how can I show my organisation how much this is worth to them?  That means, “How do I get my own figures?”, because they won’t necessarily believe that another organisation’s figures will apply to them.

Well, you know much it cost to test the first release of that last product before it went live, right?  You know how many bugs you found whilst doing so, with a breakdown by severity, of course.  And you’ve got from the business some estimates of what ‘a bug like this’ costs in production.  To keep it simple, let’s start by saying that ‘a bug like this’ is judged by its Severity, and we know the approximate average cost of fixing live bugs for each Severity level – that’s probably too simplistic for most real-life situations, but it’s good enough to illustrate the principle and I’ll show later how it can be made more realistic.

The cost of the testing and fixing that found and removed those bugs (yes, don’t forget the cost of fixing, otherwise it’s not a true comparison) was €42,000.  So, the saving was 87,000 – 42,000 = €45,000.  ROI, therefore, = (45,000 / 42,000) * 100 = 107%.

Most managers will be happy enough with a business case that will show a profit after 2 or 3 years.  Studies have shown that most of the bugs that will ever be found in a single release of a product will have been found in the first six months after it goes live – so here we’re looking at something that will more than pay for itself in 6 months.  I haven’t seen an interest rate that good advertised by any bank recently .

For sequential lifecycle projects we can use the cost of all independent testing done from the system test level upwards.  For Agile projects we can do it for each release by including in the cost of testing and fixing for a release all test activities that could generate defect reports and all the work that is needed to fix the defects found; typically, this will cover whatever testing is needed after individual user stories have reached “Done” (most Agile teams don’t report defects against a feature until then).

The weakness in the method described above is that Severity was used to estimate each bug’s cost to fix in production and that was good enough to show how the calculation works but isn’t good enough for real life.  To get a sufficiently accurate understanding we need to count the actual cost of fixing various types of bug according to the number of and size of the artefacts that have to be corrected, retested and deployed to fix each type. Watch out for a subsequent article in which I’ll show how to estimate this better .

This article is a part of a series. You can check the next one right now.

Part 1                                                                                                                                                                                                                                     Part 3