logo design...copyright Elena Yatzeck, 2010-2014

Saturday, October 22, 2011

Iteration Showcases for Backend Systems

Let's say you are in charge of the "services" operation within the IT department of a large enterprise.  You're a government entity, a telecommunications giant, or some other titan of industry.  Other IT organizations have grown up around you in the enterprise over time, and they're writing cute little front-ends that get information from customers to your services, and pass the results back.  They're doing iPods and tablets, and you're still dealing with Cobol.  Your colleagues are all concerned with "cascading style sheets" and "user experience" and color schemes and the like, but you're doing all the grungy, large-scale back-end work that actually causes the money to pour into your organization and keep you all paid.
Image courtesy of http://www.apolloamusements.com
Needless to say, your vast IT cube farm in the sub-basement is not equipped with a foosball table.

Suddenly, one day, you get an edict that your enterprise is going to "go agile!"  A perky trainer, most likely bringing with her stickers, pipe cleaners, and brightly colored balloons, explains that in the brave new world you are entering, you will now be demonstrating your software to your customers every two weeks.  It will be a "feel good" moment for you and for them.

This is where you break it to Ms. Perky that you have no user interface.  There is nothing to show.  If you do your work correctly, calculations no-one sees will now occur differently in some way, and sometime, some system, somewhere, probably running on the experimental browser of someone's 4-year old's Nintendo DS, is going to register that new calculation as an asterisk on the screen.  As ThoughtWorks project management guru Joe Zenovich says, "Backend Stories Make for Unexciting Demos," although of course he goes on to show you what to do about that, as does "Scrumwiz" in How To Demo Your Backend Software Increment.

So fear not, Backend System Vice President!  You too can experience the fun and excitement of agile software development.  There is, indeed, a way for you to structure development "stories" around your work which will be useful to customers, will keep your conversations interesting and can be demonstrated at a biweekly showcase!  Solutions fall into three categories:
  1. Pair up.  If your system has one or two main customers, and your teams are dependent on each other, then build and demo the stories together.  Sometimes what happens is that your team is doing "the big Oracle database" and their team is doing the "slick web interface," because you have different skill sets on your teams.  But from a business perspective, you're both producing exactly one experience for your customer--they interact with the screen, the screen interacts with your systems.  The best thing you can do for your customer is show them steady progress on actual software they can use, and that means structuring your work together and doing a joint demo.
  2. Create viewable mocks or stubs.  As my rockstar BA colleague @jenny_wong pointed out when I consulted her on this matter, your services do not work in isolation.  if your system serves as the back end to a lot of disparate customers, then you will still need to know what each client system is expecting your back end services to do, from a customer perspective.  This map of usage serves as the basis for the details of the interface between your system and all of its clients.  In this case, for functional automated testing purposes, you will likely need to create an all-purpose "mock" or "stub" which will mimic the front-end behaviors the real systems will give you (for the difference see Martin Fowler's "Mocks Aren't Stubs").  Seeing those tests run can be quite informative and inspiring for customers of the various front-end systems, particularly when accompanied by a powerpoint deck that explains what they are seeing.
  3. Just show powerpoint plus log files flashing across the screen.  At worst, you may want to show them a running batch file spitting out status messages at rapid pace to a screen, with some accompanying powerpoint showing progress this week on the architecture compared to progress last week.  Seeing the fast-paced letters fill the screen can produce a small amount of adrenalin in the most weathered front-end system veteran, particularly if this week's messages are different than last week's.
As Mike Cohn, so often the last word on all things scrum, says on his web site, what ties these story-writing and story-demo techniques together is that at the end of the day, you and your team of grungy back end system coders may never see a customer or even daylight, but you are in business because someone calls your service to do something, and someone calls back later to see what happened.  The key to making your work interesting to your funding authorities on a biweekly basis is to bring in the people who are in touch with your front end system or systems, and showing them the return they're getting on their investment.

Sunday, October 16, 2011

Business Analyst Corner: Write Later, Not Less

If you are a BA looking down the barrel of an agile adoption at your work place, you may feel worried that you will be switching from reams of paper stored in large binders to index cards. And not the big cards either. You're looking at the 3x5 ones.  You feel this plan is ridiculous.  You may murmur something to yourself about "insane fads!" or "damage control!" or "keep secret requirements locked in my desk and bring them out later to save the day!"  Take heart, dear friends.
card wall from http://www.scrumology.net/2011/09/15/kanban-kickoff/ (Mr. Yuck is public domain)
Things may be different at start-ups, in small shops, and in places already whirring like Kanban tops. But in most fair-sized enterprise IT shops I've worked in, your first efforts at agile implementation will not aim to reduce the quantity of requirements documentation significantly.  Instead, they change the timing at which the requirements are written, and with that change in timing, reduce the work you will spend as a BA filing and executing change requests, and reworking traceability matrices.  The overall work may also be reduced since you don't write details for features you don't implement, but the ratio of working software to the number of words and diagrams in corresponding documentation may not change at all.

That's right:  you will still end up with high quality requirements at familiar levels of bulk.  Let's compare what you do now with what you might do with a more agile process.

Requirements-first technique (you likely do this now):
1)  You and your group may write requirements for 3-6 months.  (This may be divided into separate "business" and "systems" requirements phases).  You may then have some "architects" do some further system requirements in the form of a detailed "systems architecture document."  (SAD)

2)  Once development starts, if anything changes, you file or update a change request and re-write all of the linked requirements (and the traceability matrix that links them together!) to match the actual behavior of the working software.  Changes can occur because:
  • the market changed, so now your business needs something different
  • you have learned more about what you're doing, so you don't want exactly the same implementation you wanted before
  • the development group ran out of time, so a bunch of requirements need to be removed.
  • or for other reasons.
If development takes 3 months, and system test takes 3 months, and UAT takes 3 months, you could end up spending 15-18 months first writing, then continuously re-writing the requirements.  And if you're in the kind of shop that has to "trace changes," your resulting documents may be very hard to read, because everything you ever wrote is still there, color coded or struck out.

"Just in time" requirements technique (typical enterprise-level agile teams do this):
1)  You, the SMEs, the developers, and the testers all get together for a "release planning" workshop to determine what the high-level goals are, and what the system will look like (also at a high level) to meet those goals.  You create a flexible outline of the pieces of the system using the index cards (sometimes called "story" cards), arranged in the rough order you plan to build those pieces.  You can think of this as just writing the table of contents for what would have formerly been your requirements documents.

2)  Just before development starts, you work with SMEs and other BAs to write fully detailed requirements for each of the stories you intend to tackle in the upcoming 2-week iteration.  You may start an iteration ahead, or you may just start a few days ahead.  The fully detailed requirements attached to a story are called "narratives" at ThoughtWorks, or sometimes just "detailed requirements."  As you go, you keep writing the details, the stuff that adheres to your table of contents, just before the developers actually start coding.  If changes occur to the big picture or any of the implementation that's going to occur down the line, that's fine.  You let developers complete the work for the current iteration, but you change the remaining "release plan" cards to reflect the team's new learnings and business needs.

If your workshop took 2 weeks, and development takes 3 months, you have now completed exactly one outline of the work (the release plan), and one set of detailed documents for what you actually did (the narratives attached to the stories you did).  Roughly speaking, 18 months turns into a little over 3.


So if someone tries to tell you that agile lacks rigor, or that you don't need BAs to facilitate SME discussions and record the results any more in an agile project, you may want to take their hysteria with a pound or so of salt.  Ask them:  are they a theory-drunk zealot, or have they actually tried it in your environment?

I was originally going start the title of the post with "BA Corner," but I didn't, because my teenage daughter has confided in me that whenever I say "BA," she thinks "Bad A**."  You can imagine the resulting dinner conversations, when I attempt to talk shop.  At any rate, if you're feeling particularly fierce today, please feel free to re-read this post substituting her "BA" definition.  Power to the BAs!  

Thursday, October 13, 2011

Next-Gen Agile: Demote the Non-Coders?

Mike Gualtieri threw down an enjoyable gauntlet this week with his Forrester blog post, "Agile Software is a Cop-Out, Here's What's Next"  Gualtieri put some provoking words around two sentiments I've agreed with for some time, namely:
  • 'Using "working software" as the measure of progress is narcissistic,' since it focuses on what developers are interested in (the software), not what the business is interested in (the value the software brings to the business)
  • It's a further cop-out to think 'that great software can be developed through a process of dead-reckoning with business people.' (the hyperlink is Gualtieri's).  Incremental design by committee may be democratic, but it's not going to create any iPods.
I found myself very entertained by this blog post.  Particularly the "dead-reckoning" reference.  Anyone who knows Johnny Cash's famous song, "One Piece at a Time," has probably thought of that song when first encountering evolutionary design.
From http://lyricsdog.eu/lyrics/505397
But what are we to aspire to, if we use the "cop-out" label on "working software," "customer collaboration," and "responding to change?" That would make us one for four, Agile Manifesto-wise.  Here, I found myself less in agreement with Gualtieri.  He proposes four new pillars to replace the four old ones:
  • Parallel (implement things in parallel, not serially)
  • Immersive (developers should know the business, so they can build an experience, not just write some loops)
  • Software (needed for completeness, since this is a new software manifesto)
  • Studio (re-imagine software development as producing a customer experience)
I think it's actually three pillars, not four, plus as Gualtieri points out wryly, the pillar names together build to an unfortunate acronym.  But that's okay.

What I was surprised by is that when Gualtieri says "agile is a cop-out," he doesn't mean that the original manifesto was too focused on the development point of view (at the expense of other team members and stakeholders), but instead, that developers should aim for more.  They should stop being part of the cast, and take the heroic central roles they've always meant to have.  The problem: '[d]evelopers often blindly rely on businesspeople, business analysts, user experience designers, and customers to tell them what will make a great user experience.'  Gualtieri's solution?  Developers should do user experience themselves and cut out the middle man!

Great software talent are renaissance developers who have passion, creativity, discipline, domain knowledge, and user empathy. These traits are backed by architecture, design, and by technical know-how that spans just knowing the technology flavor of the day.
Yikes, and measuring progress by "working software" is narcissistic?  Moreover, how many people out there can actually be "best" at passion, creativity, discipline, domain knowledge, user empathy, architecture, design, technical know-how, all simultaneously, and while still holding a day job?  I think I know one person who has six out of seven of these, but he can't design his way out of a paper bag, and gets help with that. 

I had actually expected Gualtieri to take this somewhere different.  Having agreed with his diagnosis that agile needs to be more than coding, and needs to take more interest in delighting customers, I had thought he might take this more in the direction of Jez Humble's Continuous Delivery or Eric Reis's Lean Startup, where small teams build towards compelling "delightful" customer visions by breaking down and testing ideas against actual customer use, and modifying to fit.  The "definition of done" in this case isn't "deployed" but rather "deployed with an ability to do A/B testing and measure usage patterns to determine next steps."

Or, alternatively, I thought he might invoke Steve Jobs, so much in our minds these days, to say that having a single person of genius impose a vision with an iron hand is more important than everyone having a say all the time, and developers should have some respect for that concept.

Or indeed, I thought he might be going in the direction of Stephen Denning, whose appealing "Radical Management" corporate vision seems to meet Gualtieri's needs for Manifesto tweaking by spreading "the Wisdom of Crowds" all the way up into senior management, empowering small teams to come up with delightful products by getting executives out of the way.

Of course I agree with Gualtieri that a focus on "customer delight" appears to be an engine which can drive corporate success quite vigorously.  And of course design by committee and an over-reliance on consensus leads to least-common-denominator solutions which maximize team peace and at the expense of measurable customer value.  And of course coders should take some responsibility for entering into discussions about design visions or user interactions, and not require minute written instructions.  But I would picture getting to "customer delight" as being a process of refactoring the way teams work together, and perhaps elevating our joint respect for user experience designers as people with distinct skills not available to the entire general population.  A coder can be an experience designer but need not be (and vice versa).  Let's just make sure there's always at least one on the team.

Tuesday, October 4, 2011

Life Before Iteration 1

It was a bright and sunny day, and suddenly an agile software development project began.

Scott Ambler started his classic 2008 Dr. Dobbs article "Iteration Minus One" this way.  And of course he went on to drive home the point that projects don't just emerge fully staffed like Aphrodite from the waves. Somebody has to do the logistics.

Botticelli's Venus, from http://www.paleothea.com/Gallery/VenusSea.html
But how much planning and setup are you allowed to do on a project before they revoke your agile badge and change the secret handshake?  Although the Agile Manifesto celebrated its tenth anniversary this summer at the Agile 2011 "Return to Snowbird" conference, people seem as disagreeable as ever about what the runway should look like for an agile project, and how long it should be.

Just before that conference, Vikas Hazrati's InfoQ article urged readers to put their teams straight to work: "every iteration needs to produce working software."  He cited George Dinwiddie, Alistair Cockburn, and Ken Schwaber, concluding "there seems to be a general consensus that iteration zero could be best avoided, if possible."  Hazrati conceded that "chartering" could still occur, but not "building a detailed backlog and creating the infrastructure."

Are you kidding me? If we're working with software teams that currently think nothing of spending a whole quarter producing nothing but requirements, do we seriously have to be all macho about "showing the business side some working software" after an hour or even a week?

In most real-life corporate teams I have seen, "jump in and scrum" creates a situation where software is prolific and predictability is a four-letter word.  (Who's counting letters?  Pencil pusher!)  Sure, the old estimates were off, but the "just scrum" people are now telling the stakeholders that they are wrong to even ask for specific estimates around their return on investment.  Questions like "what will I get by the end of the summer?" are waved away with "gotcha--we're agile now."  Throw in concepts like the "overflow scrum," (where you do testing of the stuff you wrote over the last quarter or two to make it actually work), and IT has just used agile vocabulary to really screw over the businesses that fund them.  No--this vileness isn't "pure agile," but it certainly seems to happen a lot.

In non-theoretical situations I recommend would-be newbie corporate agilists to take deep breaths, take their fingers off the keyboard, and do three things before starting Iteration 1, the sum of which will more than pay for itself in project accountability, transparency, predictability, quality, and actual value to someone in particular:
  1. Build a lean business case, comprised of a hypothesis that the business will gain more than it loses by investing in a software project with a particular direction.  The business case should include the specific measures the funding authority will use to test that the investment was worthwhile along the way.
  2. A chartering, quick-start, discovery, or inception process, in which all relevant project stakeholders from business and technology (including "operations" people) meet together in person for a short amount of time (2 weeks should be more than enough for most projects--ThoughtWorks has even seen this work for a project lasting a year) to agree on the big picture from both a business and technical perspective.
  3. An "iteration 0," "sprint 0," or "tech setup time," during which business-side people (product owner/business analyst/tester/UAT tester) begin working out details of the stories for iteration 1 (talking across the table as needed to their development and testing buddies), and technical-side people (devs and automated test experts) set up work stations, build, test, UAT, and production environments, along with a functioning continuous integration and deployment environment that does nothing more than build "hello, world!"
The Lean Business Case
Far too few businesses actually think through their project portfolio in terms of what the likely actual ROI will be for each project (increased revenue, decreased cost, increased customer satisfaction, faster speed to market, quality, or achievement of other corporate non financial goals).  Even if you did nothing else whatsoever to become more agile, really great things would happen at your company if someone wrote down ahead of time what the claimed ROI was for each project, and then compared it to what actually happened when the project was delivered.  And the "lean" business case would go further--calculate a minimum product to test in the market, actually measure how it does, and evolve from there, always measuring as you go.  But take baby steps.  Try for just ONE testable hypothesis for the whole project and see how you like it.  Success is contageous.

I've long outed myself as a fan of in-person kickoffs that go longer than the typical executive pep talk you may be used to.  I've now discovered that as usual, my "discovery" was covered in a book even before the Agile Manifesto--please read Kent Beck and Martin Fowler on "Planning Extreme Programming" for advice which is as true now as it was in 2000.  There are things you need to do even before you grudgingly start your "Iteration 0" concept, whatever that might be.  Get the whole team together, business and technical stakeholders alike.  Establish the relationships that will carry into phone and email conversations for months to come.  Let everyone understand what's going on.  And to give your business more predictability and control than they ever dreamed possible, go ahead and put together a "span plan" that shows what users will do with your software, divided into logical coding segments of no longer than an estimated 3-5 days apiece.

During this time, the group should put together a small charter document, a backlog of high level epics/features which comprise the current project vision, a high level set of scenarios for use of the software, a high level architecture, and a detailed story backlog for the first release where all stories have been sized at about 3-5 days, estimated in nebulous units of time.

Despite now-conventional-wisdom about "just scrum," please lay out a sensible and fairly complete initial "master story list" or "product backlog." up to the first release.  Allow some additional room in the plan for stakeholders to change their mind (say, 20%), figure out what the expected number of iterations will be and lay out which stories you'll do when.  The plan will change, but now everyone is looking at the backlog with a fixed cost and fixed time release in mind, and everyone understands that scope is the only thing you can play with.

Iteration 0
Once your release plan is done, plan for some time to prepare before you start to measure your velocity in Iteration 1.  You need some requirements to be well understood so developers can hit the ground running with those first stories on the first day of real coding.  You may very well do technical investigation work at this time ("spikes"), as well as setting up environments and detailing out initial stories and acceptance criteria.

Again, don't plan to do this for a whole quarter--if you're doing a 2-week iteration, start the project with a 2-week iteration 0 to get ready.  Don't belabor, don't buy all of your production hardware before it goes on clearance, and certainly don't lay out "the whole data layer" in Iteration 0 

NOW you're ready to jump in and scrum!  Go for it.

Saturday, October 1, 2011

The Full Monty: the Continuous Delivery Business Case at Scale

My ThoughtWorks colleague Rolf Andrew Russell recently pointed out on our company intranet that there are implicitly two definitions out there of the agile manifesto's principle of "continuous delivery of value":
  • "'business' CD - what we sometimes call the full monty.  minimizing lead time from idea to production and then feeding back to idea again."
He further observed, "When a business person or executive hears the term 'continuous delivery' they naturally interpret it as the latter.  And this makes sense because 'business' CD speaks to the true business value and encompasses 'technical' CD.  But 'business' CD is way more than just the devops stuff.  It is changing the way XD, PMO, analysis, etc, work, and minimizing their lead times."

And yet it's still hard to find a list of the practices a CD-oriented person should follow if she is not a developer or QA, but is instead an Agile user experience designer, program manager, product owner, or business analyst.

Technical CD

But let's start with "technical CD."  What is that?  We've been very excited lately at ThoughtWorks about the concept of technical continuous delivery, championed by Rolf as well as Jez Humble, Martin Fowler, and TW alum David Farley.  Supported by a practice called "Continuous Deployment," this concept captures the imagination especially if you're in a corporate IT department, because you picture a world where not only can you have software teams WRITE software quickly, but you can also have them DEPLOY quickly and safely to production.

Perhaps you live in a world where you've managed to set up a hyperperforming agile team, and you've started delivering small chunks of tested software to your pre-deployment environment every two weeks.  Your users are delighted, even though your well-tested software has to sit in a QA holding bin for a week before it moves laboriously to the UAT testing server, and from there, in ponderous splendor to production.  Months have passed, and a small startup in Albuquerque is already making money on your concept, except in version 3.0.

Now picture that you can put that software into production at will, even ten times per day.  And it's perfectly safe, because you can decide later whether you want to turn new functionality on or not.  And you can even be ITIL compliant.  That is a beautiful dream vision for many of us, and CD is consequently and justifiably quite popular.  Jim Highsmith built this wonderful diagram to show how you might progress from agile development practices to "continuous integration" (frequent check-ins to main and builds), and from there to "continuous deployment", and the accompanying strategic impact to be derived from this progression:

From http://www.jimhighsmith.com/2010/12/22/continuous-delivery-and-agility/strategic_continuous_delivery/
But note that introducing "technical CD" into your environment only gets you from development kick-off to operational deployment.  Although your IT department is now building software "right" in a big way, how do you know that you're building the right thing?  That's what brings us to:

Business CD

What would the "continuous delivery business case" look like?  I think you can look at this from (at least) two perspectives:
  1. CD powers the lean startup.  As Jez Humble points out in this InformIT article (and elsewhere), continuous deployment enables what Eric Ries calls "the Lean Startup."  Within the development cycle, agilists have speeded things up considerably by eliminating "Big Upfront Design" (BUFD) in favor of a general design plan whose details evolve to fit the customer need.  Analogously, the Lean Startup calls for eliminating the Big Upfront Business Case (BUFBC) in favor of a general vision and strategy, followed by a series of small experiments which lead to modifications in both the strategy and the resulting software as you go.  Because the Lean Startup comes equipped with CD developers and QAs, it could be 20 minutes from the time a software behavior is requested to the time it is deployed in production to a controlled subset of the software's user base.  What this means is that:
  2. Business-side counterparts of CD technical practitioners must therefore be skilled entrepreneurs:  Ries's book provides this diagram for the business side of CD:
    Image taken from: http://gumption.typepad.com/blog/entrepreneuria/
    Business-side CD practitioners need to be people who can identify what Ries identifies as the value and growth assumptions in their business cases, devise experiments to test those assumptions, starting with the kind of very small experiments developers can build and deploy quickly, the ability to interpret the data, and an ability to pivot the strategy to allow for the next round of experiments.
What does that mean for today's corporate PMO director, product owner, business analyst or user experience designer?  That answer isn't obvious, but let's take some solace from the fact that the best "traditional" software architects were delighted to descend from their ivory towers and start making software prototypes instead of writing vast theoretical documents.  The best build and release technologists were happy to become specialists in designing and building deployment pipelines instead of sweating with hardware and firmware "surprises" under pressure.  In the same way, the best business-side people who interact with software development have an opportunity to stop making huge, unjustified bets based on risky assumptions, and instead build up excellent and well-grounded business cases. 

So as you put your next agile pilot together, think about your reforms starting at the idea phase, and your business case evolving right along with the software--cost of change is now quite low.  You can and must do experiments that fail in order to feel confident about the strategy that succeeds.  Sure, it's scary, but it's also tremendously exciting, and it can't be done by technology alone.  This is management, analysis, and design stripped to their essence.  (But you can leave your hat on).