Return of the Matrix: The Organization Around the Self Organization

Do you have a group of, say, 100 people, whom you want to organize optimally for a software delivery program?  Martin Fowler is famously on record saying that "scaling agile is the last thing you should do."

A better approach is to try to scale down your project. ...an unscientific straw poll revealed that most projects could lose about half the people of the project without making things go slower. Time and time again I hear of success occurring when a team is cut significantly in size. Large teams carry a big overhead in communication and management. Using smaller teams staffed with more able people is usually faster and cheaper, even if the everyone is more individually expensive.

This brings to mind one of Tom Waite's "non-lethal weapons," the Shrink Ray, in the movie Mystery Men,.  Aficionados of the movie (or comic) will remember that the ray is "based on simple dry-cleaning technology."

From the Blu-ray site:  http://www.blu-ray.com/movies/Mystery-Men-Blu-ray/42756/

You are left with the (perhaps not completely false) impression that in reality, you should do a completely different project, (a more interesting one), and lay everyone off (and hire Martin Fowler peers). 

Perhaps you don't have that option.  It can happen.  Moreover, if you're pragmatic, you want to keep everyone in your current work force around to figure out how they will fit into future teams, rather than starting with a clean slate (and zero subject matter expertise about the specifics of your operation).  Some people may choose to leave, and you may eventually find responsibilities shifting dramatically, leading to other staff reorganization and reduction, but you should let that happen later.  Don't start there

So if starting over from scratch isn't your first and best option, here are some questions you should ask, and some options which will vary based on the answers to those questions.

You really should read this article!  http://www.unspecial.org/UNS649/t37.html

Things you already know, but may be questioning, now that you're "going agile:"

• You still need a line management structure of some kind, yes, seriously, you do.
• Even before you consider organizing people for delivery, you need to consider organizing them into some kind of reporting structure which allows you, as the employer, to support them and ensure their overall well-being.  
• The line management structure does not need to be the same as the delivery structure.  In fact, it may be better if the structure is different.
• After witnessing many bizarre varients of "flat" or "wire-archy" structures, I have come to believe strongly that you are best off with an old-fashioned "matrix" where people with like skill sets report into a line manager who stays in touch with them regardless of what delivery team they are working on.
• So if you are a company of 100, and you are wondering how to organize your whole set of employees in an agile way, then you should figure out what your line management structure will look like (shallow or deep), and subtract people who do line management tasks from delivery.  I suppose this point is controversial, but all I am saying is that if you leave "line management" tasks to take care of themselves, you are conceding to a seemingly random system of hiring, annual review, and other work allocation negotiations where politics will take over.  Your team will never survive the ensuing chaos.
• You need a delivery structure of some kind. 
• The remainder of your 100 people are going to have to be divided up into:
• Some team groupings, "teams," "pods," "work streams," or whatever you decide to call them locally.
• Some leadership positions.  You cannot self-organize a program of 100.  You will need servant leaders to think about coordination and the big picture.  Examples:  program manager, software lead or architect, lead quality person, overall product owner.
• You will need people with the appropriate skill sets:  some leaders, some people who can develop, people who can wrangle stakeholders until coherent requirements emerge, people who can design tests (and the data to drive the tests) which will allow the team to detect whether the software continues to work as it is developed.

Some rules of thumb to start with for agile delivery teams--some "patterns," if you will:

• Teams larger than 15 grow unwieldy.  10 is a good number, and 6-8 is better even than that.
• A single servant leader (program manager, software lead, lead quality person, lead product owner) will have trouble with creating vision and facilitating the work of more than 10 teams.  Create a scaling structure where no manager needs to provide servant leadership to more than 6 people at a time.
• A grouping ratio that often works is 1 analyst: 2 pairs of developers: 1 tester.  
• Putting it all together, a starting assumption about the 100-person delivery organization might be:
  
  • 1 Program Manager + staff as needed to handle
  • External reporting/compliance
  • Scrum of scums team coordination
  • 1 Lead Developer/Architect + staff as needed to handle
  • Solution tech stack architecture visioning and hands-on evolution
  • Delivery tech stack architecture visioning and hands-on evolution
  • Librarians/curators to keep things tidy without becoming a bottleneck (refactor to remove duplication, rather than creating one "shared tools" team that everyone has to depend on.
  • 1 Lead Tester + staff as needed to handle
  • Solution automated test architecture (functional + non functional) visioning and hands-on evolution
  • Delivery data and test harness architecture visioning and hands-on evolution
  • 1 Product Owner + staff as needed to handle
  • Consensus building in the business hierarchy
  • Business architecture visioning and hands-on evolution
  • SME identification and outreach
  • Other extended team members (DBA, Security, Trainers, etc.)
  • Remainder allocated to teams of 6-14 people:
  • Scrum Master/Iteration Manager
  • Product Owner
  • 1-2 Analysts
  • 2-8 Devs
  • 1-2 Testers

And finally, some basic questions to consider when you design your staffing plan:

1. How do we allocate our current workers to teams while moving towards teams of "T-shaped" individuals who can do more than one job?  In final state, in a large enterprise, you are likely to have people who CAN do anything, but who may PREFER to do one kind of work more than another.  So don't worry too much if, at first, your testers specialize in "how to test" but don't automate, or your developers specialize in development plus test automation, but don't know how to do analysis or rigorous test scenario development.  Allow people to grow, but take advantage of their current skills and interests.
2. Are we staffing tactically or strategically?  If you have a diverse set of technologies, or over-specialization in your current workers ("I only do the UI."), you may choose to staff to people's current strengths ("all Pega work goes to team A"), or you may want to distribute people with specialized skill sets across multiple teams, to make each team more flexible and content-agnostic.  In general, the more you stretch people towards being able to handle complete stories from UI to DB and back, the slower things will be at first, but the better off you will be in the 6-12 month time frame.
3. How do we design a set of parallel efforts that can be internally cohesive and loosely coupled with each other?  See Mike Cottmeyer's excellent post on this point.  Concretely, this means if you are building a workflow, you may want to cluster functionality in "stages along the workflow" to allow each team to build the full stack of one closely-related set of functions, and reduce the interdependence between teams to small APIs which can be virtualized by the first side to be done, until the second side gets to it.  You may NOT want to set things up so that "all common things" are owned by a single agile team, since that team will become a bottleneck to everyone.  Instead, think about a design curator/librarian to keep things clean, while allowing parallel efforts to proceed unfettered.
4. Do we already have an automated continuous delivery pipeline which accommodates automated functional regression testing plus manual exploratory testing as needed?  If not, you will need many more people and probably a different schedule, in order to avoid killing the teams under the load of functional regression testing.
5. How diverse is the technical stack?  If you have many different platforms, each set up differently, with, perhaps, a few vendor packages in the mix, then your staffing will be more rigid, because you will need people who are expert on each of those platforms, and hardly anyone can be expert on all of them.