...copyright Elena Yatzeck, 2010-2017

Wednesday, February 19, 2014

Return of the Matrix: The Organization Around the Self Organization

Do you have a group of, say, 100 people, whom you want to organize optimally for a software delivery program?  Martin Fowler is famously on record saying that "scaling agile is the last thing you should do."
A better approach is to try to scale down your project. ...an unscientific straw poll revealed that most projects could lose about half the people of the project without making things go slower. Time and time again I hear of success occurring when a team is cut significantly in size. Large teams carry a big overhead in communication and management. Using smaller teams staffed with more able people is usually faster and cheaper, even if the everyone is more individually expensive.
This brings to mind one of Tom Waite's "non-lethal weapons," the Shrink Ray, in the movie Mystery Men,.  Aficionados of the movie (or comic) will remember that the ray is "based on simple dry-cleaning technology."
From the Blu-ray site:  http://www.blu-ray.com/movies/Mystery-Men-Blu-ray/42756/
You are left with the (perhaps not completely false) impression that in reality, you should do a completely different project, (a more interesting one), and lay everyone off (and hire Martin Fowler peers). 

Perhaps you don't have that option.  It can happen.  Moreover, if you're pragmatic, you want to keep everyone in your current work force around to figure out how they will fit into future teams, rather than starting with a clean slate (and zero subject matter expertise about the specifics of your operation).  Some people may choose to leave, and you may eventually find responsibilities shifting dramatically, leading to other staff reorganization and reduction, but you should let that happen later.  Don't start there

So if starting over from scratch isn't your first and best option, here are some questions you should ask, and some options which will vary based on the answers to those questions.

You really should read this article!  http://www.unspecial.org/UNS649/t37.html
Things you already know, but may be questioning, now that you're "going agile:"
  • You still need a line management structure of some kind, yes, seriously, you do.
    • Even before you consider organizing people for delivery, you need to consider organizing them into some kind of reporting structure which allows you, as the employer, to support them and ensure their overall well-being.  
    • The line management structure does not need to be the same as the delivery structure.  In fact, it may be better if the structure is different.
    • After witnessing many bizarre varients of "flat" or "wire-archy" structures, I have come to believe strongly that you are best off with an old-fashioned "matrix" where people with like skill sets report into a line manager who stays in touch with them regardless of what delivery team they are working on.
    • So if you are a company of 100, and you are wondering how to organize your whole set of employees in an agile way, then you should figure out what your line management structure will look like (shallow or deep), and subtract people who do line management tasks from delivery.  I suppose this point is controversial, but all I am saying is that if you leave "line management" tasks to take care of themselves, you are conceding to a seemingly random system of hiring, annual review, and other work allocation negotiations where politics will take over.  Your team will never survive the ensuing chaos.
  • You need a delivery structure of some kind. 
    • The remainder of your 100 people are going to have to be divided up into:
      • Some team groupings, "teams," "pods," "work streams," or whatever you decide to call them locally.
      • Some leadership positions.  You cannot self-organize a program of 100.  You will need servant leaders to think about coordination and the big picture.  Examples:  program manager, software lead or architect, lead quality person, overall product owner.
    • You will need people with the appropriate skill sets:  some leaders, some people who can develop, people who can wrangle stakeholders until coherent requirements emerge, people who can design tests (and the data to drive the tests) which will allow the team to detect whether the software continues to work as it is developed.
Some rules of thumb to start with for agile delivery teams--some "patterns," if you will:
  • Teams larger than 15 grow unwieldy.  10 is a good number, and 6-8 is better even than that.
  • A single servant leader (program manager, software lead, lead quality person, lead product owner) will have trouble with creating vision and facilitating the work of more than 10 teams.  Create a scaling structure where no manager needs to provide servant leadership to more than 6 people at a time.
  • A grouping ratio that often works is 1 analyst: 2 pairs of developers: 1 tester.  
  • Putting it all together, a starting assumption about the 100-person delivery organization might be:
    • 1 Program Manager + staff as needed to handle
      • External reporting/compliance
      • Scrum of scums team coordination
    • 1 Lead Developer/Architect + staff as needed to handle
      • Solution tech stack architecture visioning and hands-on evolution
      • Delivery tech stack architecture visioning and hands-on evolution
      • Librarians/curators to keep things tidy without becoming a bottleneck (refactor to remove duplication, rather than creating one "shared tools" team that everyone has to depend on.
    • 1 Lead Tester + staff as needed to handle
      • Solution automated test architecture (functional + non functional) visioning and hands-on evolution
      • Delivery data and test harness architecture visioning and hands-on evolution
    • 1 Product Owner + staff as needed to handle
      • Consensus building in the business hierarchy
      • Business architecture visioning and hands-on evolution
      • SME identification and outreach
    • Other extended team members (DBA, Security, Trainers, etc.)
    • Remainder allocated to teams of 6-14 people:
      • Scrum Master/Iteration Manager
      • Product Owner
      • 1-2 Analysts
      • 2-8 Devs
      • 1-2 Testers
And finally, some basic questions to consider when you design your staffing plan:
  1. How do we allocate our current workers to teams while moving towards teams of "T-shaped" individuals who can do more than one job?  In final state, in a large enterprise, you are likely to have people who CAN do anything, but who may PREFER to do one kind of work more than another.  So don't worry too much if, at first, your testers specialize in "how to test" but don't automate, or your developers specialize in development plus test automation, but don't know how to do analysis or rigorous test scenario development.  Allow people to grow, but take advantage of their current skills and interests.
  2. Are we staffing tactically or strategically?  If you have a diverse set of technologies, or over-specialization in your current workers ("I only do the UI."), you may choose to staff to people's current strengths ("all Pega work goes to team A"), or you may want to distribute people with specialized skill sets across multiple teams, to make each team more flexible and content-agnostic.  In general, the more you stretch people towards being able to handle complete stories from UI to DB and back, the slower things will be at first, but the better off you will be in the 6-12 month time frame.
  3. How do we design a set of parallel efforts that can be internally cohesive and loosely coupled with each other?  See Mike Cottmeyer's excellent post on this point.  Concretely, this means if you are building a workflow, you may want to cluster functionality in "stages along the workflow" to allow each team to build the full stack of one closely-related set of functions, and reduce the interdependence between teams to small APIs which can be virtualized by the first side to be done, until the second side gets to it.  You may NOT want to set things up so that "all common things" are owned by a single agile team, since that team will become a bottleneck to everyone.  Instead, think about a design curator/librarian to keep things clean, while allowing parallel efforts to proceed unfettered.
  4. Do we already have an automated continuous delivery pipeline which accommodates automated functional regression testing plus manual exploratory testing as needed?  If not, you will need many more people and probably a different schedule, in order to avoid killing the teams under the load of functional regression testing.
  5. How diverse is the technical stack?  If you have many different platforms, each set up differently, with, perhaps, a few vendor packages in the mix, then your staffing will be more rigid, because you will need people who are expert on each of those platforms, and hardly anyone can be expert on all of them.

    Thursday, February 13, 2014

    Patterns for Agile Transformation Success: Agile Enterprise Adoption Without a Gut Rehab

    Mike Cottmeyer has re-emerged in the blogosphere, after a bit of a lull, with an electrifying series of posts about enterprise agile, one of which is entitled How to Structure Your Agile Enterprise.  I love the whole series, and especially this post, anchored as it is in Cottmeyer's real life experience:
    First of all… let me share that I have NEVER worked on a small agile team. I’ve coached many of them, but my introduction to agile was in the context of large enterprise class financial systems… things like online banking and bill payment. The kinds of systems where the company makes a penny or two on every transaction and does millions every year.
    Is this your enterprise business flow reality??  From http://thedailywtf.com/Articles/The_Customer-Friendly_System.aspx
    Cottmeyer provides some really crucial insights about agile at scale that may run counter to what you believe is "fundamental" to agile:
    • "Feature teams" are not practical, if any given feature involves a workflow where data passes through many different application silos, end to end.
    • In contrast, "Governance Teams" and "Product Owner Teams" are needed at scale, not just a single scrum master working with a single product owner who knows everything about every system, and a single self-governing team. 
    In fact, Cottmeyer says, what is practical is a delivery team structure analogous to SOA systems architecture, where "services [are] loosely coupled and highly cohesive. When building Scrum teams, we want each team to be loosely coupled and highly cohesive… just not necessarily responsible for an end-to-end slice of the entire product."
    A wonderful diagram from http://www.planetgeek.ch/2011/07/08/presentation-agile-code-design-how-to-keep-your-code-flexible/ -- applies both to code and to agile delivery team structure!

    A question which may come to your mind as you read this, however, especially if you don't have Cottmeyer's organization currently coaching you, is "how do I get there from here?"  How feasible is it to morph from your current executive and vendor management strategy, from your current HR strategy, from your current policies about software delivery, from your current location strategy, from your current staff, and so on, to your optimal organizational end state?
    From http://ethicsalarms.files.wordpress.com/2011/05/you-cant-get-there-from-here.png

    The good news is that as an organization which is already making money through some kind of strategy and tactics, you are very likely to be set up very well for these changes already, and without having to throw away all your binders of regulations and policies (or requirements, if you were worried about that).  And, in fact, by leveraging your corporate goals, and using agile techniques to steer your strategy and tactics better to align with those goals, an agile adoption is likely to be the fastest way to make the most of your strong points.

    From http://www.incredibleart.org/lessons/middle/tessell.htm
    Let's start to talk about patterns of what is feasible, because, as Cottmeyer says, there are techniques and tools that work that can be described architecturally without descending into a coaching process that micromanages everything about a team's day to day operation.  And, just as Cottmeyer does, let's borrow from the now-standard concept of architectural design patterns to talk about things we may want to establish in some locally appropriate way with the business architecture which surrounds our software delivery.  Here are some ideas I've thought about already, some that I've blogged about already and some that I will build out in the coming weeks. 

    Creational patterns
    Structural patterns
    Behavioral patterns
    Concurrency patterns
    The secret to agile transformation is that your enterprise already has patterns in place which promote the values of the manifesto:  working software, valuing people, keeping up with the market.  Further transformation can be aided by a coach that can help you reflect on what will get you closer to your corporate goals, and that you act on your findings.  Use what you're good at, and fix the rest as you go, based on the ROI of further investment in change.  And please let me know if you've seen other patterns that have worked--I would love to hear your ideas as well!