Requirements Traceability in Agile Software Development

One of the grim proving grounds for the would-be agile business analyst (henceforth "WBABA")  is the "traceability conversation."  Eventually, you will have to have one.  You may have seen one already.  If you haven't, you may want to half-avert your eyes as you read further.  It gets a little brutal.  But if you close them all the way, you can't read.

From:  http://www.highestfive.com/mind/how-to-perform-a-successful-interrogation/

Dialogue:

WBABA:   ...so in summary, we complete analysis on each story card, and then we support the developers as they build it that same iteration!

Corporate Standards Guy:  but how do you do traceability in agile?  You have to have traceability.  It's broadly recognized as an important factor in building rigorous software systems. These software systems permeate our society and we must entrust them with lives of everyday people on a daily basis. [The last two sentences are an actual quotation from the Center of Excellence for Software Traceability website!]

WBABA: [cowed silence]

Corporate Standards Guy: right.  Well, go ahead and do your little story cards, but I expect that before you start papering the war room wall in Post-It note chartreuse, you will present me with a full Business Requirements Document and Functional Requirements Document, so you can trace everything through to the Systems Architectural Diagram and on into the test cases.  Show me the traceability matrix, and we're all good.

It appears as though someone has thrown a wrench into the speedy agile SDLC which will kill it altogether, or at least blow out its kneecap.  

But have no fear!  Unlike "pair programming," a sadly controversial practice in which the debate could go either way on any given day, "traceability" is ground on which you can beat Corporate Standards Guy down ANY day, hoisting him with his own petard!  How?  Why?  Because...agile is actually better at traceability than waterfall.  Embrace the dark side and let's take a look at that.

What is Traceability and Why Does it Matter?

All Center of Excellence hyperbole aside, "traceability" could be factually described as "the use of tracking and tracing systems and processes to match the incoming product requirements to outgoing product attributes."  Why would you want to do that?  Typically three positive reasons, and one negative one.

Positively:

• "Forward" traceability:  so you can ensure at any point that everything you require is in the plan somewhere, or, better yet, in the actual software.
• "Backward" traceability:  so you can ensure that everything in the software was developed for some identifiable reason, and it wasn't just developers running amok.
• Keeping things tidy when requirements change, both during the project and during ongoing maintenance afterwards:  if requirements change, you would like to know what the impact of the change will be--which pieces of the system will be impacted?  How many of them are there?

Negatively:

• So you can know who to blame: if something is missing in the product attributes.  Was that missing attribute somewhere in the requirements?  If not, blame the business!  (Or every team's favorite culprits, the BAs).  If it was there in the requirements, then figure out where along the chain the ball was dropped, and bingo! there's your culprit.  Did the designer forget it?  Did the developer disregard the design?  Did the tester fail to notice it was working wrong?  And so on.

From:  http://www.jacobsen.no/anders/blog/archives/2004/05/17/the_tire_swing_cartoon.html

Why Agile is Better than Waterfall at Traceability

The best way to put this is that where waterfall is a translation process, agile is a refinement process. 

Translation:  As suggested in the illustration above, a waterfall software development process takes a list of requirements from the customer, which may or may not be well explained, translates them into "official business requirements" language, then into "official functional requirements language," then into "design elements," then into "software components," then into "documentation chapters."  This process is exactly analogous to passing a requirement through a succession of online translations:

1. Original:  "I need a drop-down to select toppings on the web site for my pizza store."
2. Original->Bulgarian:  Имам нужда от падащото меню да изберете гарнитурата на уеб сайта за моята пица магазин.
3. Bulgarian->Portugese:  Eu preciso de um menu drop-down para selecionar as coberturas no site para minha pizzaria.
4. Portugese->Basque:  Goitibeherako menu bat behar dut nire web, pizza du toppings aukeratzeko.
5. Basque->Haitian Creole:  Mwen bezwen yon meni drop-desann sou sit entènèt mwen an, engredyan pitza elèv yo te chwazi nan.
6. Haitian Creole->English:  I need a drop-down menu on my website, the ingredients of pizza have to choose from.

Suddenly, the pepperoni has a seat at the monitor.  In which language did the customer get throttled by his own pizza?  We don't know now, but give us a traceability matrix showing word for word translations at each step, and we'll figure it out.

"That's absurd!" you say.  But is it absurd?  How does a customer follow the requirements inventory as it progresses through the development life cycle?  Would elements written in Haitian Creole be any harder for her to understand than page 37 of the Functional Requirements Document? 

Refinement:  In an agile process, requirements are refined in place, not translated.  In fact, for purposes of forward and backward traceability, you literally build out a requirements matrix from the beginning, and develop in a way that keeps traceability intact.  What is this matrix?  In agile, it is called "the backlog."  Each backlog item (called "a story") is a description of a piece of business functionality which the system will provide, described from the point of view of a person doing something in particular with the software.  So these steps look more like this:

1. Planning:  the whole team gets together and builds a high-level model of the drop-down for the pizza store page, including a frank discussion about the power struggle between people and food, and the need for people to win.
2. Story splitting:  since adding a drop-down to the page is a complex task that would take more than 2 weeks, the story would be split into smaller stories.  Let's say in this case that story 1 is "add the drop-down control to the screen with one choice, pepperoni", story 2 is "get everyone in the store to agree on what the rest of the toppings are, and put those choices in as well," and story 3 is "subdue the rebelling garnishes."  
3. In iteration 1, we might tackle the garnish rebellion, defining success as "pepperoni does not wield weapons observably in the kitchen."  We might also have a developer add a drop-down control on the web page showing pepperoni to customers as a thing they can order on their pizza, defining success as getting calls into the pizza store from 10 people who confirmed they had visited the web site and identified pepperoni pizza as something they could buy.
4. In iteration 2, we might spend the whole time getting a gigantic multi-lingual kitchen staff to agree to add pineapple, canned corn, and Spam as additional ingredients, and refine the system test criteria to say that customers should identify all four of these as choices from which they were picking.

Note that as we go, traceability doesn't need to be maintained as a separate activity, because each system feature is being built out from general description into detailed implementation, and okayed as "complete" when it meets pre-agreed acceptance criteria, and the customer herself nodding in agreement.  If requirements need to change, new stories would be introduced to override behavior not needed any more on the page (for example, sales of Spam and pineapple might be lower than expected), and through grouping in the requirements repository, stories related to the drop-down could be viewed in sequential order, to show that first one thing was implemented, then another. 

Fans of the classic Frances Hodgeson Burnett story for girls, The Little Princess, will remember the point in that book where Sara Crewe tries to explain to the evil Miss Minchin that she doesn't need to learn French.  Miss Minchin overrides her and gives Sarah a basic vocabulary book so she can get started.  When the French master arrives at the school, she conducts a lengthy conversation with him--in French--and he exclaims, "Madame, Sarah does not need to learn French because she IS French!" 

Agile is like that.  You don't need to expend effort in agile developing a traceability matrix, because the agile requirements repository IS a traceability matrix at its core.