logo design...copyright Elena Yatzeck, 2010-2015

Thursday, August 23, 2012

How to Do Project Cost and Time Estimation for Distributed Agile Teams

Danger Will Robinson!  Long post!

There is a chance this will be useful to you if some of the following are true:
  • You're an agile software development practitioner who needs to provide an overall "fixed cost" type estimate for your work.  This might be because the corporate entity paying your team insists on you providing such an estimate before it will commit a single penny to pay you, your team, or your "team room snacks" bill.  Or you're billing time-and-materials, which is great, but someone still wants a ballpark of what the total will be for what it currently seems the project subject matter is kind of about.
  • You would normally do your estimate by getting everyone into a room for a co-located "project inception" or "planning game" or "story mapping and release planning session" where you would do this estimating by moving physical cards around on a table, floor, or wall.
  • You don't actually have the whole team physically together, so any activity with index cards has to be reported to the "remote" people through shouting and a single digital camera built into someone's laptop.  Example:
"Preet!  Preet!  We took the third PURPLE Post-It and put it over to the FAR LEFT.  So widget 3 is now part of the "Useful Widgets" epic we're doing in Release 4.  I'll pan the camera over there for you to see it!"  
I can tell you from experience that the people in India joining your call by phone aren't getting as much out of this as you think, particularly if it is 3am their time.
Webcam-eye view of your team room as seen from far away from a good post on this topic, although I disagree with the author's conclusions:  http://www.gerrykirk.net/options-for-team-task-board-when-one-team-member-remote/
So before we start:
  • If you're a person who thinks agile should never involve planning (see my rant about the need for life before iteration 1 here), this is not for you.  
  • If you're a person who thinks agile must always be done in person by the whole team, using 3M products, this is not for you (see my rant about distributed agile being a reality we should all face here).  I absolutely agree it would be better if we could all collocate all the time, but we can't, so let's work with what we have.
  • If you're a person who thinks distributed agile requires specialized high-tech tools to be done right, I'm with you!   However, we don't all always have those tools available, so I'm providing this alternative suggestion with tools you can get for FREE.  But if you do have the right tools, and they're legally licensed and so on, you can read this and feel happy if you like.
The enviable TechSmithies at Agile 2012:  http://blogs.techsmith.com/wp-content/uploads/2012/08/agile2012_techsmithies.jpg
If for some reason you're still with me, let me walk you through the method.

Let's say your team has been hired to bring Loretta and Steve Paizano's Pizza Palace into the 20th century, if not the 21st.  Their grandson has convinced them that they need a web site.  They are willing to go for it, hearing that their nephew Scotty down the street has increased his bakery revenue by $545 per month ever since he put up his web page, which allows customers to order pierogies for delivery.  No, they don't have GrubHub there.

They want to know from you how much your team will charge to build the page for them.  You are in Oshkosh, Wisconsin, your developer friend Monica is in Svalbard, Norway, and you have a tester friend, Anupam, in New Zealand.  So you guys need to confer, but not in a way that involves Mom and Pop having to buy the three of you Second Life avatars and a high tech virtual team room with life-sized virtual planning poker cards.  You all have internet phones, luckily, so talk is cheap.  But what do you do for your estimation and release planning!?

Step 0:  Know what you want to do, and type it into a shared spreadsheet.  Why?  Have you ever been on a conference call?  Seriously--you need everyone to have a shared thing to look at.  This is helpful for keeping people focused on the same thing, and it's crucial so that if there are problems with the quality of the phone lines (or if some people primarily process visual data rather than spoken information), people can read it as well as hear it.  Choose a fast typist.  This is where you are relying on readily available virtually FREE tools.  You need:
  • Phone access (everyone has internet phones on your team, weirdly, even including the Paizanos).
  • A chosen spreadsheet format (Google has a free one, and there are others out there.  Often people have one they've paid for)
  • A chosen way to share your voices and one screen, including free options like Skype and the like.  
You don't need "video conferencing" or even a "webcam."   

If you can all look at the broadcasted screen together, and talk about it in real time on a shared audio line, you can do collaborative estimation and release planning.

Okay, so with everyone at their respective screen, and on a shared phone bridge, you start by asking your clients why they want a web site.  The mom and pop team pass the phone back and forth between each other, and eventually they tell you about their problems, the goals, their dreams, and so on.  As you talk, your fastest team typists records their motivations and potential solutions to their problems into the spreadsheet you are all looking at. 

The Paizanos are in Newfoundland, and they drop off the call a lot, but they can see that at the end of the discussion, you've written stuff down they agree with:

The Paizanos are impressed that you can type so quickly.  They agree this is what they want.  Now you work together to break the features (which some of the Cool Kids call epics, but some don't) into individual "stories."  As experienced agilists, the three of you suggest some stories in a flash.  You keep track of which feature each story belongs to, so if the Paizanos need to make some hard decisions later, you can use filtering features in the spreadsheet to bring up the stories feature by feature.

Step 1:  Now you're ready to focus on the stories.  Use the "hide column" feature to help everyone focus, and just bring up the story list:

You have much to say to the Paizanos about the INVEST principals and the need for a standard story format, but I'll suppress those details for right now, because once you fix the stories to meet your own high standards, they won't fit nicely on the page here.  They could be in extra columns on the spreadsheet which are hidden for the most part, though.

Step 2:  Identify your "Minimum Viable Product," and don't worry right now about stories beyond your first release.  Notice in this example that item six seem somewhat fanciful, and is all out of whack compared to the rest.  Perhaps a mom-and-pop pizza store which will be lucky to have a read-only web site should not tackle computer-driven pizza-production automation for the first release.  You get Mr. and Mrs. Paizano to agree that maybe you should get the site up first, and then deal with the automation piece in a later release.  So now, here's what your list looks like.

Technically, perhaps, you could have had this discussion at the feature level, without even getting to stories, but the Paizanos needed to be humored for a while before you could have that discussion.  Product Owners are sometimes like that.  Last year, your team literally had to estimate the effort for building a working miniature version of the Saturn V rocket, before the Product Owner admitted that the desired scope exceeded the available budget.  Even so, he kept shouting, "FAILURE IS NOT AN OPTION" a whole bunch of times.  So you're feeling okay that all you had to do here was show goodwill by typing out a story, and then push the story to release 2.

Step 3:  "Story Splitting."  Make sure the stories left in Release 1 are a reasonable size for the team to estimate.  Monica is only available to develop your software when she's on break from her job at the Global Seed Vault, so about ten hours a week.  Handily, Anupam is willing to test "whenever," because all he does in New Zealand is surf.  So Monica's time is your real constraint.

Your team decides to use a two week iteration for the work, which gives you 20 hours of Monica's time per iteration.

In order to give the Paizanos a really clear estimate, you all agree that no story should be tackled for estimation purposes which is larger than 10 hours, so that you know you'll be delivering at least two meaningful stories worth of work to the Paizanos per iteration.  To ensure this is the case, you do "story splitting," where you take stories that are too big, and divide them into sub-stories that are small enough to estimate.  You, Monica, and Anupam walk through the spreadsheet line by line and determine quickly whether anything seems bigger than 10 hours.

Monica observes that if all they want to do is let people see a "Buy Now" button she can implement through PayPal, that's no problem, but if the Momstress and Pop-Tart want for people to buy multiple pizzas in one order, she's going to have to implement a more complex shopping cart, which could take ten hours in itself.  Following similar logic, you break down the "Refund handling" request as well.  Now you have this new version of your list, post-Story-Split, where everything is the right size.  You've left the old stories in for reference, but crossed them out to indicate they are now fully superseded:

Step 4:  "Comparative Estimation"  Although you are all quite giddily certain that the stories are now all in the right ballpark, size-wise, you want a higher level of precision for iteration planning.  But Monica gets all freaked out if you ask her to provide time estimates.  She will take a story like "login to the system" and estimate it at 120 hours, just in case she has to stop and try to get a manicure in the middle, and Svalbard is out of nail varnish, and she has to get it FedExed from the mainland.

So rather than get into another argument about that, instead, you all agree to put time completely out of your mind and just do comparative sizing.  You start by hiding the stories you're not going to do in the first release, and you set all the stories you are going to do, to the default size of "Medium."

Now you march through the lines one at a time as a group on the phone looking at the spreadsheet together, and compare them.  Story 2 seems smaller to Monica than Story 1, so if 1 is "Medium," then 2 is "Small."  So is 3.  7 seems bigger.  Anupam chimes in about the regression test load on 10.  And so on.  In the end, you have stories comparatively sized something like this:

If you're familiar with the Agile "Fruit Estimation Game," it's as though the three of you have simulated a rating system whereby Story 1 was an apple, Stories 2, 3, and 9 were plums, Story 7 was a grapefruit, and stories 8 and 10 were pineapples.  But no expensive fruit was used--just a FREE spreadsheet!  But what now?  Neither your t-shirt sizes nor their fruit equivalent will convey useful information to the Paizanos about the cost and timeline of the project.  So why did you go to all this trouble?

New agile teams are always eager to start estimating in "points."  Points are what the Cool Kids use.  Often, however, for teams that haven't done agile before, the points are just disguised time estimates.  In Monica's case, for example, if you had started with points, she would have said the login page was 120 points, with 1 point = 1 hour.  So you purposely don't do that.  But now that you have safely estimated everything comparatively, you can enter the Cool Kid World of Story Points, and you do it really easily, circumventing Monica's crazy manicure fixation.  You can translate directly from t-shirt sizes to numbers using this scheme from Fibbonaci:  XS = 1, S = 2, M = 3, L = 5, XL = 8.

Applying this logic, your spreadsheet now looks like this:

But we still don't know what this means for time or cost (insofar as we will be paying ourselves for the time we spend in each iteration).  So now what?

Step 5:  Team Velocity Estimation.  "Velocity" is a fancy agile term for "the number of story points your team will complete in a single iteration."  The best way to calculate velocity is to have your team do a few iterations and see how much they get done.  But the Paizanos, like many funding authorities, prefer to get the estimate first, and then commit the funds.  So rather than determining velocity empirically, we use a set of simulations to put the team into a frame of mind where it predicts the amount of work it could get done in a single iteration.  To avoid the newbie agile impulse to equate points to hours, we temporarily hide the points, and ask the team to pick groups of stories they could get done (developed, tested, and approved by the Paizanos) in two weeks.  Here's what the spreadsheet would look like to start with:

You, Monica, and Anupam start by looking for a random set of work that could be done in 2 weeks, where Monica is only available when she's on break.  You carefully avoid any mention of "how long will each one take."  Instead, you focus on what combinations would add up to the right total amount of work, disregarding dependencies.  You are going to deal with dependencies in just a moment.  For now, you just chill, and get into the focused Zen state of picking combinations that add up to the right amount of work).  After a while, you decide as a group on three combinations that seem as though they could fit into two weeks:

Because you're totally Zen and chill, you don't mind that story 8 involves refactoring code that you haven't actually written yet.  For purposes of velocity estimation, you assume all dependencies are handled.  So maybe combination H is going to be iteration 10, where combination I would occur in iteration 3.  All you're doing here is figuring out the team velocity.  You're not actually assigning any story to any iteration.

So, voila, you can now use your three sample "chunks of work" to estimate team velocity.  Unhide the points column, and see whether there's a pattern in terms of how many points the team wants to tackle per iteration.

Column H represents 10 points worth of work, Column I represents 8 points worth of work, and Column J represents 10 points.  So if the simulation is accurate, this team expects to be able to do about 9-10 points of work per iteration.

Step 6:  Calculate minimum project duration and cost.  Even before you try to figure out which story should go in which iteration, you now have a lot of useful information to give to the Paizanos.  Since you know that:

Total points = 30
Estimated team velocity = 10 points/iteration

You know that all things being equal, it will take your team 3 iterations to complete the work, although if every story has to be done in a particular sequence, you may have an even longer time frame.  So you can go to the Paizanos and tell them truthfully that they will need to pay your team for at least 3 iterations, but possibly 4.  Do they want to move forward?  Your team charges $400 per iteration.

Step 7:  Adjust to fit desired budget.  It turns out the Paizanos only want to spend $800, and see whether they actually start getting the extra $545 that Scotty saw.  If things work out, they will come back for more work.  They ask you to fit the work into 2 iterations, not 3.  You tell them that now that the work has been broken down into these small pieces, you don't see any way to have the work take less than 3 iterations.  Then you suggest that maybe the Paizanos could look through the stories and see if some of them could come out of the first release, even beyond what you agreed to before.  You suggest that they prioritize the stories, so that everyone will know what the least important ones are, and it will be easy to cut 10 points worth of work off of the project schedule.

They prioritize and you resort.  Now it looks like this:

It appears that the Paizanos have significant pizza quality problems, so automating refunds is actually more important to them than enticing customers to buy a whole virtual cartload of pizza!  You're intrigued but a little sad.  You want to give them your mom's tomato sauce recipe, but instead you point out that if they give up on the shopping cart for now, you will be able to deliver the other six features in a little more than 2 iterations.  Maybe you could split the difference somehow.  They seem happy with the offer.  Now, if you unhide your "release" column and the row that you already designated for the second release, your list looks like this:

Reassured that you remember what they want for release 2, the Paizano's ask you to finish your estimation and actually let them know when they will have each feature available for their customers.  So you move quickly to:

Step 8:  Organize stories into actual iterations.  For this project, you now have six stories to put into two iterations.  You need to:

  • Deliver in priority order as much as possible, since the Paizanos ranked these by their expected return on investment for each feature
  • Make sure the iterations have about the same number of points in them
  • Make sure that you handle any dependencies between the stories.  You can't "refactor" something that doesn't exist, for example. 
  • If anything is a more technically risky, Monica would like to schedule that first, because if she can't figure it out, she doesn't want the Paizanos to have to pay for a second iteration, since the whole thing will be a bust.
Keeping those issues in mind, you find yourself with this plan:

  • Stories 1, 2, and 7 are scheduled for Iteration 1, and Stories 3, 9, and 10 are scheduled for Iteration 2.
  • Total expected velocity for Iteration 1 = 10 (add the points for stories 1, 2, and 7).  Expected velocity for Iteration 2 = 12.  So the work appears to be set up to be around the expected team velocity of 9-10, although you seem to have let your pity for the Paizanos lure you into over-committing.  Tut tut.
  • For the most part, the things the Paizanos ranked as important are happening more in Iteartion 1 than in Iteration 2.
  • Monica has been able to put two of her high risk items into Iteration 1.
  • Logical dependencies are observed--the system is set up for login before any user actions are added, for example.  And the refund page is created in Iteration 1, and only in Iteration 2 do you add features for automating the refund.
The Paizanos doubtfully write you an advance check for the first iteration, and you run out to buy snacks and then FedEx them to Svalbard and New Zealand.  They don't have string cheese in New Zealand.  You did it!  You did it!  You're ready for Iteration 0!  Huzzah!

Sunday, August 19, 2012

Ban Bullies From Your Teams

If any of you have dealt with a bully, you will be quick to tell me that the bullies we encounter day-to-day are not ON the team.  They are ABOVE the team, or parallel to the team, or in some position of authority, and that's how we notice them.  This is true.  A bully, or as Robert Sutton defines her, "an a**hole,"  is defined by the difference between how she "manages up" compared to how she "manages parallel and down."
From the excellent blog post at http://www.theregionalinternshipcenter.org/tag/workplace-bullying-statistics
 A bully uses her position of strength to behave badly towards people who can't stop her.  Some bullies simply lack empathy, and some actively enjoy inflicting pain.  This is a serious problem, not just a "personality detail" about an otherwise valued coworker or employee.  Sutton has a wonderful New York Times article about how, in contrast to the Donny Osmond position on the matter, "Bad Apples Infect the Tree."  Well, for Osmond, it was "the whole bunch, girl," not the "tree," but the thrust is the same.

Bullies are unfortunately very common in the working world, particularly at the top end of the executive chain.  Some of you may remember the 2011 Jon Ronson book which found that corporate CEOs are four times more likely to score as "psychopaths" on standard mental health exams than people in the general population.

Fair enough.  Part of the bullying picture is indeed that you need to take appropriate measures to avoid being victimized by a workplace bully.  As is so often the case, Europe is way ahead of the United States in enacting national laws to provide redress for employees against workplace bullies.  Even in the US, some workplaces have actual "No A**hole Rules," which spell out a corporate policy about bullying, and which help to make those corporations the type of places by which most of us would like to be employed.

But I'm not talking about that.  I'm talking about you, and about the teams you actually influence or directly control.  If you have jerks on your team, it's your responsibility to reform them or get rid of them.  And you need to take a hard look on who is setting the standard that creates the nastiness on your team.  Sticking with the fruit metaphor, the apple doesn't fall far from the tree.  (And by the way, as Adam Lashinsky's new book, Inside Apple, reveals, Apple isn't exactly nirvana as a work place either.  But I almost digress.)  Two important questions:

Are you hiring and encouraging bullies on your teams?  Quick self-test:
  1. Is everyone who reports to you routinely nice to you, and do they agree with all of your edicts decisions?  Seriously, you're always right?  You need to worry right now, and a lot.  Start by finding out:  are they equally agreeable to each other and to people below them in the pecking order?  Do you even know?  My fourth-grade teacher, Andrew F. Horan, always said, "flattery will get you somewhere," and it will.  Make sure when you are hiring that you include peer interviews in the hiring process--don't just make "an executive decision."  Include 360 reviews or "skip levels," where people talk with their boss's boss at least once per year, in your mix of self-improvement HR activities.  And for yourself as well.
  2. Beware of making "an executive decision," in general.  If you are a "servant leader," and even more if you are an ambitious go-getter executive seeking promotion in most normal US corporations, your role is primarily to understand what your teams are doing, to socialize it across your level in the organization, and up the executive chain, and to support it.  You must certainly be mindful about your delegation strategy, but in steady state, you should not need to insert "your touch" on every single thing produced by your team.  Don't criticize, don't nit-pick.  Offer improvements if something comes to mind, but otherwise just celebrate your team's achievements.  You should give ALL public credit for team accomplishments to your team.  You should never take credit.  Understand me--people will give you the credit anyway!  Unlike the "trust fall," which I can never learn to like, this leap of faith by you will always pay off immediately for you, and pay networking dividends in perpetuity.
  3. Include a "character screen" as a crucial part of your hiring decisions.  I can't reveal my patented character screen technique for fear one of you might one day apply to work for me, but the information is out there on the InterWeb.  Find it!
  4. Institute your own "no bullying" rule for the part of your world that you control.  Investigate and follow up on bullying reports involving your team.  Do not tolerate them.  Is it really okay to cut into the productivity of everyone Janet interacts with, just because she's the only one who understands the PowerPoint template?  Think this through.
Even more importantly, are you a bully?  What if, unbeknownst to you, YOU are actually a bully?  Sutton has a handy self-assessment you can take to check this out.  I can't believe most people I know would fail this test, but just to make sure, I recommend you take it just to make sure.  As Sutton says, all of us are jerks some of the time, so at minimum, take the test to see whether you might be inadvertently making someone else's life miserable just because you have more power than they do, and they're scared to tell you.  It will be impossible for you to stamp out bullying in your work place if you're the guy at the front of the group, and you're the one holding the pitchfork and the biggest burning torch.
From http://mat.usc.edu/students-give-us-precious-gifts/

Let's make the world a safer place for nice people.  Start today!  Ban bullies where you can, and start the process by taking a hard look at yourself.

Saturday, August 18, 2012

Be Proactive

I once spent an hour at a facilitated "C-level" meeting where we discussed what we could learn about leadership by considering the "Dancing Man at Sasquatch Music Festival" video.  Yes, the meeting was compulsory.  Excerpts from my meeting notes:  "Note the importance of the moment when the SECOND crazy dancer joins the first!"  "How does one man eventually become a crowd?"  "What is the tipping point?"  "What are YOUR day-to-day 'crazy dancer' moments?"  "Let's all go be crazy dancers!"

This type of meeting, along with any event in which adults need to play musical chairs or "self-organize" into groups using color-coded name tags, is what keeps Scott Adams in business with Dilbert.  If there's an important lesson to be learned from the crazy dancing guy, it's probably: "if you want your average executive to be courageously proactive, you will need to get him drunk and send him to Saskatchewan."

And yet, bucking this pragmatic wisdom, I, for one, prefer to work with proactive people.  Not glum naysayers, not snide peanut-gallery-mutterers, and especially not "assertive people" who position themselves for a promotion with no regard for their overall contribution to the business.  And also not people who go through the motions but block forward progress.

How much time do we waste every day in email conversations, meetings, and memo wars, in which there is never a problem identified, a solution proposed, or a plan agreed to?  How many times do we "follow the process" but never create anything in particular?  What did we deliver?  What value did it bring?  What did it cost to build, and what will it cost to maintain?  It's a sad, cynical world we live in.

What's the alternative?
  • Always have a plan.  You should make one up, or someone should give you one, but you need to have one.  Why are you going to do something?  What is it that you should do?  Will the plan work?  The thing you plan to do, supported by reasoning and tested by agreed-upon criteria, is your first plan.  Your plan is likely to grow from a vision to something bigger, and it is likely to "pivot," as the Lean Startup people say, but you need a plan.  If you don't have one, you are already lost, and you are certainly not in a position to be proactive.
  • Supply your own energy.  Are you an emotional black hole?  If so, it's highly unlikely that you will be a proactively contributing member of any team.  I don't care how brilliant you are--if you have to be propped up by your peers all the time, you're almost as bad as a bully.  Add energy to the gatherings you attend.  Do not subtract.
  • Have skin with variable thickness.  Sometimes it is appropriate to be sensitive to every nuance of every breath taken by everyone on your team, all day long.  That's how you notice trouble, and that's what puts you in a position to head off problems while they're still getting organized.  But sometimes it is appropriate to let a peer or a stakeholder attack you directly or in a back-ally, back-stabbing kind of way, and just shrug it off.  Know the difference, keep your balance, and remain graceful.
  • Take breaks.  Medieval medicine, while relying in a regrettable way on leeches, also considered recreation, in the form of sports or the arts, as necessary to human achievement.  The proverb was "the bow that is always strung does not shoot straight," or something like that.  If you don't relax sometimes, you will gradually wear yourself down to nothing and someone will put a leech on you.  Okay, no, that won't really happen.
My daughter worked on the kitchen staff of an immersion French camp in Minnesota this summer, and part of the staff indoctrination was to learn this cheer:  "qu'est-ce qu'il faut? DE L'ENTHOUSIASME! et de quoi encore? UN PLAN! et si on n'a pas de plan? PLUS DE L'ENTHOUSIASME! mais on devrait vraiment avoir un plan..."

If you aren't fluent in French, or have an actual fear of French as I do, she has provided this translation:
What do we need?  Enthusiasm!  What else?  A Plan!  What if we don't have a plan?  More Enthusiasm! ... But We REALLY SHOULD HAVE A PLAN.
That really sums it up for me.  Be enthusiastic, have a plan, and take yourself and your team forward. 

A plan with no heart is a waste of your time, and enthusiasm alone, while it makes for an inspiring music festival YouTube video, is just silly.  As we say in science, if you're not part of the solution, you're part of the precipitate.

Wednesday, August 15, 2012

Hands Across the Conference Table: Coordinating Simultaneous Agile Work Streams

Most of the people I interact with are unequivocally excited about the opportunity to try agile for the first time. Not.

Most people have doubts. And one of the top questions I often get is "can I seriously use agile for a very large project with multiple work streams, and with cross-impacts from other groups who aren't agile at all?" Yes! Yes, you can. In fact, agilists and the agile approach has a particularly strong set of tools for helping you manage cross-team interactions. These tools not only help you scale your "pure agile" project management approach, but also can help strengthen best practices in your "waterfall" PM approach, if you're working in a hybrid environment. So let's talk about that.
From https://eventioz.com/events/culture-and-projects-are-not-like-oil-and-water-th--2
First, a few myths that you might have heard, but I'm sure you agree aren't true.
  • Agile doesn't scale.  Um, wrong.  Agile Scaling has been around for almost as long as agile itself, discussed, for example, this year by Tom Grant, in his blog on Forrester, and supported for ages before that by well-known luminaries Scott Ambler (Agile@Scale) and Dean Leffingwell (Scaled Agile, Inc.).  Myths about non-scalable agile are just there to prevent poor, deprived enterprise IT people from enjoying their rightful standing in the world of agile.  You're falling prey to agile snobs who prefer to work on small teams for small companies with bean bag chairs, and they are successfully trying to make you feel bad for working in a big company.  Sometimes it's hard to be Goliath.
  • If you do planning before you start coding, in order to support agile at scale, that's not agile.  Wrong again, in my humble opinion, and of course that of many others.  
But let's say I concede the point, and agree that every single practical thing that you might do to make agile work at scale makes it not-agile.  Let's look at what you can do within a multi-stream project which has agile elements (that would not be possible in a classic waterfall implementation).

You can plan the optimum number of work streams/work teams for your project in the first few weeks after kickoff.
  • For the work you will be able to manage internal to the scope of your own project, and will therefore be able to handle in an "agile" manner, you have divided up the needed features into small enough pieces that you can see in a very granular way whether it's possible to create parallel work streams who won't step on each other by touching the same parts of the code base simultaneously, or ever.  
  • Agile release planning calls for partitioning the work across the teams after you understand the architectural direction and the full feature set, but before you even start detailed analysis.
  • If you have an insufficient number of people with analysis, development, or testing skills to keep development moving in multiple streams, you know that ahead of time, rather than hiding those details in the "batch" mentality of delivering all requirements before starting all development.  
You can plan integration strategy with all cross-impact partners proactively.   Effective agile teams will inventory their data interfaces during release planning, and for each interface, determine:
  • Is the integration partner already done?  In this case, we build to their specification.
  • Is the integration partner going to start work after we're completely done?  In this case, they will build to our specification, and we can document where we ended up later, so long as it's in time for their process.
  • If the partner is building in parallel with us, targeting the same release?  Then the next question is, is the integration partner doing their needed new work in an agile or waterfall methodology?  For waterfall partners, agile teams need to develop detailed requirements around the interface before starting subsequent work, so that the agile team can mock and stub the partner behavior correctly, even if the partner doesn't have time to collaborate beyond that.
  • But better, does the integration partner want to do proactive interface testing with you, and could that be scheduled ahead of the official "Integration Test" period for the release?  You and your partner teams can also make proactive arrangements for mutual stubbing or mocking of code, developing test sets reciprocally (you provide test strategies around your code as it interacts with their system; they provide the obverse to you).
You can still do meaningful program-level roll-ups showing progress across the disparate work streams, so long as you agree to use business-friendly metrics like "actual code complete" rather than "points burnt up."  And in fact, since the work streams which progress in an agile manner will get to the coding sooner, the program-level dashboard becomes a showcase for the delights of agile over waterfall.

Please don't let the agile-for-tiny-teams-only people get you down.  Corporate moguls running ginormous agile programs world-wide, unite!  You have nothing to lose but, hm, okay, well, it's actually quite good to be a ginormous corporation.  And agile scaling just adds to the benefits.

Tuesday, August 7, 2012

How to Control Scope Creep in Agile

Project managers and others new to agile often ask me "how do you control scope creep in the Agile software development life cycle (SDLC)?"

From http://m.odul.us/
Scope creep, for those of you reading this blog purely for the joy of it, is when a team has agreed to build a piece of software for a given price in a given time frame, and then the person who wants the software changes their mind about what they want, and they ask the team to do something outside the initial agreement, often without concomitant adjustments to the budget or the time frame.  Scope creep is nasty, and left unchecked may cause blindness or death.  Or at least loss of sleep for the team members, or perhaps loss of hygiene if they opt to skip showers in order to fit in a nap.

And yet business needs change.  Nobody buys VT-100 terminals these days, and only audio aficionados buy vacuum tube stereos.  The iPhone 4 gives way to the iPhone 4S (but still doesn't provide turn-by-turn spoken navigation).  If you're the person buying the software, you don't want to contract for a VT-100, and then have to take delivery on it ten years later when everyone has moved on to a VT-220.

In the "waterfall" methodology, you control scope creep through a thing called "Change Control" where everyone agrees that the original contract was written in blood.  Although you as a product owner can negotiate with the team for changes in the plan, to stay abreast of the market, it will cost you.  At the end of your painful and protracted negotiations, you will indeed get something different than what you requested plus some interesting new bruises.  But your bruises help to console the team and get them through the night on the Sunday of Memorial Day when they are doing the production deployment and you are out on your boat with vodka tonics and attractive companions of one or more genders in brightly colored swimwear.

In agile, we reserve the right to bruise you, but we have a slightly different perspective.  For one thing, we purposely structure the work so that many things that might have been scope creep under a waterfall SDLC aren't a problem.  Here are some agile rules of engagement:

  • It's not scope creep if you're changing something before the team has started to think about the details.  In an Agile SDLC, we lay out the planned work at a high level before we start, to make sure we're in general agreement about how much the team will get done, but once the general size of the effort is agreed, details are purposely left to be determined later.  At the "planning game" stage of the project, we might specify that we need a user interface with about 10 fields.  But we don't specify what the exact fields are, what the validations on the fields will be, nor where they will park in the database, until we are ready to do actual development on that story.  Why do we wait?  To avoid scope creep even coming into the conversation!  This method allows you to change your mind without penalty, and to give you and your company a competitive advantage over product owners in other companies who are kept to their original contracted scopeIf you are working with an agile team, all you've invested initially in that segment of your end product is a maximum of about 100 words on an index card.  If the team hasn't even done detailed analysis of the story, it's really no big problem to swap in a different 100 words.
  • It's not scope creep if it doesn't create additional work for anyone.  So, you might say, there's a big difference between 100 words that say "Build a drop down field to allow someone to order pepperoni on their pizza, etc., etc.," and 100 words that say "Build the Curiosity Mars Rover, etc., etc."  That's right.  So in Agile, we will let you swap one story for another without penalty, but only if the team can estimate that the new story is roughly the same amount of effort as the old one.
  • It's not scope creep if it's within the team's defined contingency allowance for "unknown unknowns."  Empirically, experienced agile delivery teams will tell you, new necessary scope will be discovered as you go.  When you put together your release plan, leave a 20% contingency completely open for this newly discovered scope.  You should still be careful not to introduce anything that isn't strictly needed, but if you discover as you go that to get what you need, you need some additional effort, you're still fine until you use up your contingency.
  • Fundamentally, it's not scope creep if the delivered product is exactly what project stakeholders need at release time, and the code quality is such that you can continue to keep the software matched to the business needs indefinitely.  So long as the software development team and the people watching the market are in touch with each other and doing their jobs correctly with attention to intrinsic and extrinsic code quality, and so long as the team is able to work at a sustainable pace, the issue of "what did I say I wanted six months ago" is pretty much the least of your worries.  The important thing is that the software meets current business needs and will continue to do so.
So what is scope creep in Agile, and how do you control it?
  • It would be scope creep if the stakeholders want to swap new work for work already completed.  If you have completed development and testing on a software feature, the project stakeholders don't get to say, "oh, whoopsie, the market has changed, and we don't need it any more--please go back and change it.  Instead of doing A, we now want to do B.  We're agile!  Yay!  It's not scope creep."  Wrong--it certainly is scope creep, if the team agrees to do it.  The team doesn't have a time machine, and the time to develop Feature A is already gone.  If the team needs to kill Feature A and swap in new Feature B instead, that is a new request which needs to be traded off against other requests that are still in the future for the team.
  • It would be scope creep if the stakeholders want to swap something big for something small.  Let's go back to the Mars Curiosity Rover for example.  If the team is budgeted to do a drop-down with choices of pepperoni, mushrooms, and sweet corn, and suddenly you want them to develop the Mars Rover in the same time frame, and they agree to do so, that would be scope creep.
And how do you control it?  If you're an agile team, you point to your pile of completed work, your team's track record for delivering every two weeks, and your projected delivery date for the next production deployment.  You say, "We have committed to this plan, and we are delivering to it with fully production-ready code every two weeks.  If you add additional work, we will not be able to deliver on time.  Here is our evidence."  Notice that you're not having a conversation about who said what six months ago.  You are all squarely focused on the delivery date and the probable contents of the software to be delivered on that date.  You are all securely committed to keeping quality constant, rather than agreeing to allow more work to creep into the schedule.  That's the responsible conversation to be having, so that's really nice.

Can scope still creep?  Yes, certainly.  If you have unreasonable product owners, an agile project is vulnerable to scope creep.  But if you keep product ownership reasonableness constant, an agile project is actually much less vulnerable to scope creep than a waterfall project, simply because it is built to support the need for change.  Agile requirements are pointed towards "what the business will need at delivery time," where waterfall requirements inevitably point back towards "what the business thought it needed at project charter time."  An agile team will not nickel and dime its product owner to death, and an agile team will be able to provide really good evidence to show why a request will or will not work with a projected budget and time line.

In the end, scope creep is actually easier to control in Agile.  So to me the real question should be:  how do you control scope creep in waterfall?