The Agile Lumberjack: Why Batched Requirements Are Not As Sensible As You Think

Let's say you're a Business Analyst, and you're okay.  You've done your job for years.  You are a subject matter expert in your own right, and you get the job done. 

Suddenly, a sparkly-eyed Agile Guru (SEAG) shows up at your workplace and asks you to stop writing a "requirements document" that holistically describes the software your developer friends will be building.  Instead, the SEAG insists that you use a set of index cards to "hint" at what needs to be built, tape the cards to a wall, and plan to focus on the details of each card later, right before the developer starts to build it.  And maybe you won't even write those details down then, either.  You'll take the "hint" from the card, have a "conversation" with the developer, everyone will nod enthusiastically, and off you go to find out more about a different hinty card.

Really, when you put it that way, it sounds totally crazy.

This is why some analysts have a very hard time with agile at first.  Analysts are the conscience of an agile team.  They are the people who draw the big picture from tiny, precise details.  They get the details right, but they can always place the details into a larger context.  They see the forest because they can see the veins on the leaves of every tree.

http://en.wikipedia.org/wiki/The_Lumberjack_Song

Why in the world would you NOT want to develop software in such a sensible way?  Aim, then shoot, you might say.  Map, then deforest.  And so on.  Actually, no.

What Toyota learned in the 1950s competing with Ford is that you have to be very careful about building your "economies of scale."  It might seem to you that it's more efficient to "just sit down and write all the requirements at once" than to intersperse writing requirements with developing software to meet the requirements, but empirically this is not so.  Consider this schematic view of a simple development life cycle -- requirements, development, and test of 10 software widgets:

Diagrams based on a really wonderful blog at:  http://rasmusson.wordpress.com/2008/04/16/batch-vs-continuous-flow-processing/

Done in the "economy of scale" requirements batch, followed by a development batch, and then by a testing batch, the first widget rolls out to your production web site after 51 hours, or maybe you wait to do a release, and all 10 widgets roll out in a total of 60 hours.  Nice.

But what if instead of batching the work by "role," you instead batched it by "widget?"  Viewed from solely your perspective as an analyst, it seems uncomfortable.  But if you think in terms of the whole system, this mode of operating is amazingly efficient.

You do the requirements for just one feature and hand it to the developers.  Then you move onto the second.  You work continuously on just one requirement at a time.  Devs work on building just one widget at a time.  Testers work on just one set of tests at a time.  The regrouping makes a remarkable difference in terms of time to market for your widgets.

• Let's say you decide to release all of your widgets to market at once, once they're all tested, and let's say each widget earns your company $10K per hour once it is deployed.  Your "group by widget" strategy earns the company $1.8M more than the "group by role" strategy did ($10K times 18 hours times 10 widgets)
• But if you have the flexibility to release your widgets to your production site as they get done, the benefits are even greater.  The first widget gets done in 12 hours rather than 51.  If this is a money-making widget for you, and it's worth $10K per hour to your business, that difference is worth $390K for just the one widget, and you get an additional influx of revenue for each widget that comes in ahead of the eventual 60 minute mark.

You will notice that this system still isn't optimal.  What would make sense would be to see whether the development work could be split into smaller pieces and done in parallel, so that the elapsed time per widget would be 1 hour for requirements, 1 hour for development (because you would have 10 developers for each analyst), and 1 hour for testing.  Now we're talking about bazillions of dollars difference in revenue, more than enough to make up for staffing the additional developers.

This is one of the big reasons why big Web companies like Amazon.com are so excited about agile.  They have widgets that can make this much money.

And no matter what software you are building in your company, you can benefit the same way.  Even if you have to wait to release software on some regular cadence, like quarterly, each release can have that much more software in it, because you batched by feature, not by role.

Which brings me to one final point.  Often times, new agilists think that "agile is just doing waterfall over and over again, really fast."  A team new to agile may take the time they have to develop a piece of software, and divide it into 4-8 week iterations.  In each iteration, the analysts do requirements for the first week, the developers do development for the next three weeks, and the testers get everything at the last minute, and then get blamed for bad quality.  You know they do.  You've seen this before.  Only it used to take 12-18 months, and now it happens monthly.

"Rapid waterfall" is really just the worst of both worlds.  You still have people sitting idle part of the time, or else you evolve some kind of "overlapping rapid waterfall" where everyone is working on three iterations at once to "keep busy."  So don't do that!
 

http://commons.wikimedia.org/wiki/File:Stone_rapids.jpg

Do the stuff the SEAG tells you to do.  Break the work into small "stories," and work as a team to move the stories from "not started" to "done" as individual units, keeping the flow constant, and not waiting for some artificial whistle to blow somewhere to announce that "now it's time to start developing."  That's why we have a card wall.  The smaller your units of work, the more productivity you can squeeze out of your system.

You didn't want to be a lumberjack anyway!