 |
In Software Quality Assurance it is often hard to estimate the value that QA is bringing to the table. The impact of QA is so broad that it is really challenging to gather it all together and assign a dollar value to it. However, we can attempt to assign a value to QA provided that we will be able to enlist help from various parts of the organization.
How to calculate the price of a bug?
Let’s get to the basics.
How does QA help? It helps to prevent issues from escaping to the end product that is handed to the customers.
How do we measure issues? By bug tickets and specifically the ones that had been found in production and/or affected customer(s).
It is straightforward to find the list of issues like this in an issue-tracking system. Moreover, this kind of metric is widely known already as Defect Escape Rate.
That’s great, but do we assign a dollar value to it?
This is where we need to partner with people in Finance/Sales/etc. We can ask those people to provide us with data on how much business has been lost/not acquired within the last 12 months.
Once you have the data for the amount of business lost in the last 12 months (let’s call it “A”) and have the number of defects total which escaped to production in the last 12 months (let’s call it “N”), then you can calculate the cost of a bug escaped to production as:
Cost of a bug in production (C) = A/N = Amount of business lost divided by the number of issues escaped to production.
The original amount of money A which was lost due to bugs last 12 months we would call Dollar Escape Rate.
Next is analyzing why these issues escaped production. What was the main reason why they escaped? Was it because there was not enough bandwidth to do proper regression testing before each change was released?
If so then you can estimate how many tests are being performed on every change which gets to production. Let’s say, on average we can validate (let’s call it “M”) scenarios before releasing software to production.
Therefore, you can very roughly estimate that doubling the number of validated scenarios (M) might decrease the number of issues escaped to production (N) by 2.
Since we know the Cost of a bug in production (A/N), we can estimate that the amount of business lost will also be decreased by 2.
ROI Formula
This way we just calculated the ROI of doubling the number of tests which is A/2 or half of the amount of business lost to bugs in the last 12 months.
In general, the formula will look something like this:
ROI = A * M2 / (M1 + M2)
Where:
A = Amount of business lost in last 12 months due to issues escaped to production (the Dollar Escape Rate)
M2 = The number of additional new scenarios that will be validated with the new change
M1 = The number of scenarios validated currently
For example, if the amount of business lost last 12 months was $1,000,000 and we were validating 1,000 scenarios on every change on average, then adding additional 3,000 scenarios to validate will give us:
ROI = $1,000,000 * 3,000 / (1,000 + 3,000) = $750,000
Therefore any total budget below that amount would be a reasonable investment.
Conclusion
This ROI formula, in particular, helps to partially justify investment into test automation. Since the number of scenarios validated with manual testing is very low, automating tests to allow validating more scenarios on every change should increase the coverage dramatically, therefore delivering high ROI.
And, we believe testRigor is by far the fastest and easiest way to build end-to-end regression tests since your existing workforce (Manual QA, QA Automation Engineers, Product Managers, Business Analysts) will be able to do it about 15 times faster than traditional automation with Selenium.
Don’t believe us? See how it helped other companies here:
or talk to us here.
