Skip to main content

Quantifying Manual Test Technical Debt

Forced by circumstances (and an especially pragmatic client), I've recently been asking peers and "the blogosphere" the apparently naive question, "is it important to do automated testing and clear up what Mike Cohn calls 'manual test technical debt?'"

Theoretically, if you have no Big Upfront Design and you also have no automated test suite, you're pretty much just building a big unplanned mess and you'll never be able to change anything for fear of breaking something else.  I understand that, and so does my pragmatic client.  But can this issue be quantified?  Aside from feeling somewhat inadequate when reading agile theory, I mean.

The reason it's important to ask this question is that setting up and maintaining automated testing is expensive.  Unlike many agile practices, which can be set off with some sharpies and a clean wall, automated testing requires a large scale capital expenditure to bring one or more testing packages into an organization, and then some significant strategy development, staging, and training of resources to put those new packages into use and to keep them running.

You need some compelling financial case to show the likely return on an investment which may be half a million dollars for software alone in a medium sized corporation.  So how much is it worth to clean up your manual test technical debt?

There are some related questions out there with quantified answers.  Gartner and CAST
have recently published estimates of how much technical debt is out there globally and on a per-application basis ($500B and $1M, respectively). "Technical debt" is defined by these studies as "the cost of dealing with delayed and deferred maintenance of the application portfolio."  As most people who worry about technical debt note, this "cost to fix" is not the same as what it costs the business to have a software emergency of some kind, or to be slow in rolling out enhancements to the software.  And of course this "cost to fix" is not exactly the same as the cost accrued by an organization due to the lack of automated testing.  But it's a handy statistic to be able to throw around.

Within your own organization, you may want to look at the following, in terms of making your business case:
  • Regression Testing costs:  this may be your easiest way to quantify a return on investment.  What does it cost you to manually regression test your most expensive features?  How much would it cost you to set up automated testing on those features?  It is likely that you'll have some parts of your application where the one-time cost of setting up an automated regression test will pay for itself within the year, in terms of making your manual testers available to do other important testing of new features as they are written.
  • Software development costs:  if you're in a position to do so, quantify the number of points your project teams deliver per iteration, to get a "cost per point."  If your teams are routinely interrupted to fix production bugs and put in patches, you should be able to quantify the cost of these unplanned interruptions, in terms of undelivered feature points.  The costs will likely add up to tens of thousands of dollars quite easily.
  • Insurance analogycost of unexpected down time for your application.  Calculate the cost to your company for down time to your application.  Even if down time is not catastrophic, since manual work-arounds are available, you should be able to quantify the value of the time of staff diverted to doing manual work-arounds during software down time.  If automated tests give you better quality software and reduced down time, a large-scale investment in automated testing may be justified.
  • Business losses due to lack of speed in adding new features:  this may be harder to readily quantify, but as your software gets harder to change, you will lose the agility you intended to gain when your company first took on agile software development techniques.  You may want to put together one or two hypothetical or real cases of changes which were much more expensive to deliver due to the cost of manual regression testing than they would have been, had automated testing been in place.  Depending on what your software does, you should be able to assign a business value to software flexibility, in terms of business lost due to delayed time to market.
  • Total software replacement cost:  at some point your software will become partially or completely impossible to repair, due to the cost of regression testing the whole.  What is the cost of starting all over?  On the other hand, if your company was planning to totally replace the software during some short time horizon, then maybe your manual testing is fine.
It's all very well and good for us zealots to say things like "without automated testing, you Just Aren't Agile."  But if you're looking to get your CFO to release significant funds, you'll want a case relevant to the company bottom line, not just an alignment with some invisible Agile Correctness measure.

Popular posts from this blog

A Corporate Agile 10-point Checklist

I'm pretty sure my few remaining friends in the "small, collocated team agile" community are going to desert me after this, but I actually have a checklist of 10 things to think about if you're a product owner at a big company thinking of trying out some agile today.  Some of these might even apply to you if you're in a smaller place.  So at the risk of inciting an anti-checklist riot (I'm sorry, Pez!), I am putting this out there in case it is helpful to someone else.

Here's what you should think about:

1.Your staffing pattern.  A full agile project requires that you have the full team engaged for the whole duration of the project at the right ratios.  So as you provision the project, check to see whether you can arrange this staffing pattern.  If not, you will encounter risks because of missing people.  Concretely it means that:
a.You need your user experience people (if applicable) and your analysts at the beginning of the project, as always, b…

Requirements Traceability in Agile Software Development

One of the grim proving grounds for the would-be agile business analyst (henceforth "WBABA")  is the "traceability conversation."  Eventually, you will have to have one.  You may have seen one already.  If you haven't, you may want to half-avert your eyes as you read further.  It gets a little brutal.  But if you close them all the way, you can't read.
Dialogue:
WBABA:   ...so in summary, we complete analysis on each story card, and then we support the developers as they build it that same iteration!Corporate Standards Guy:  but how do you do traceability in agile?  You have to have traceability.  It's broadly recognized as an important factor in building rigorous software systems. These software systems permeate our society and we must entrust them with lives of everyday people on a daily basis. [The last two sentences are an actual quotation from the Center of Excellence for Software Traceability website!] WBABA: [cowed silence]Corporate Standards …

The Agile Business Case

Many agile teams have never seen a business case, ever, and they may even be proud of it.

Our mantra is that we deliver "business value," not just "software," quicker, better, and faster, but if so, we certainly don't spend a lot of time reporting on value delivery, and in fact we may be scornful about "analysis paralysis."  As software developers, we consider ourselves to be doing quite well if we can deliver the software every two weeks (or continuously).  And this is particularly if we've enabled this frequent high-quality delivery through automated testing and automated build-and-release techniques.  We've reduced business risk by making results visible more often, and allowing the business to change direction more frequently.  We assert that along the way of course we're also delivering value.  But how would we prove it?

I've recently posited that we shouldn't even think of doing agile projects without capturing and recording s…