Organizing at Scale: A Basic Scrum Framework for Enterprise Teams

Recently, I was working with a smaller, but distributed, team that was trying to establish a basic framework for working together in a lightweight, Big Picture, enterprise pattern. We were rolling out a basic scrum practice for the individual teams, but one that lends itself to scaling well into enterprise release planning, and is also based on sharing and communicating sprint objectives, rather than just a prioritized backlog . (After all, what team or stakeholder wants to sort through another teams backlog just to understand what they are doing in a sprint?).

Based on a little background with the team, I assumed we were all working from the same playbook. Well surprise, surprise, there were many misunderstandings about the basic practices I thought we had standardized. That led to a variety of interesting conversations.  To clarify intent, I put this brief framework together and thought perhaps others might be able to use it as well.

One caveat though, while this is what I use to good effect, I make no claims that it would be a perfect fit in any other “Scrum team in the enterprise” context. Your mileage will vary, but here you go:

Sprint Pattern

Sprints are two weeks in length. Sprints for all teams start and stop at approximately the same day. Each sprint has a standard pattern. Plan the Sprint. Execute the Sprint. Demo the Sprint. Retrospective.

Plan the Sprint

1)   Presentation of objectives and stories by the product owner.

2)   Breakout. Team members work independently or in collaboration to understand intent and draft an acceptance test for each story. Teams use task/hour breakdown, or not, at their discretion.

3)   Commitment. Teams reconvene and commit to negotiated objectives and supporting stories for the Sprint. Commitments are made only by the team. Final objectives are stated in a sentence or two, in plain English, and are communicated to all stakeholders.

Execute the Sprint

Teams collaborate on achieving the objectives of the Sprint. Stories are discussed, elaborated, completed and demo’d to the product owner as available. Story (not task) status is maintained in “big visible chart”. Typical story states Backlog, Defined, In Progress, Complete, Accepted).
However, the objectives for the Sprint trump any particular story. Team (but not management) may refactor, add, or delete stories where necessary to achieve the Sprint objectives.

Demo the Sprint

Results of the Sprints for each team are demo’d to all team-aligned stakeholders. In the larger enterprise, an additional, aggregate, higher-level demo or “Feature Progress Review” may be necessary for program stakeholders.

Retrospective

Team conducts a short retrospective in two parts:

Objective (metrics) Review: Determine whether or not the team met the objectives of the sprint (yes or no). Collect any metrics team has agreed to, but teams must understand velocity, both for new development and that devoted to maintenance.

Subjective (process) Review. A typical format is: What Went Well? What Didn’t? What One Thing Will We Do Better Next Time?

Typical Sprint Ceremonies

In a two-week sprint, a typical meeting (ceremony) set is as follows:

Standard meeting set

Release Cadence

Releases (or Potentially Shippable Release Increments) are at approximately 90 day intervals (approximately 6-7 sprints per Release Increment). Teams may include stabilization (hardening sprints) or not based on quality, system level integrations, continuous integration capability and technical debt issues.

Release Planning

Release planning occurs at the beginning of every 90-day cycle. Release planning is typically one or two days, face to face wherever possible. All team members participate. A standard format is as follows:

• Review and retrospective on accomplishments of prior release.
• Executives communicate current business context
• Management presents solution Vision, including a prioritized list of features
• Teams break feature into stories, plan the Sprints based on known velocities, negotiate commitments with stakeholders.
• Teams identify risks and impediments. Each is discussed and addressed into categories as follows: (Resolved, Owned, Accepted, Mitigated.
• Teams (including management) restate (in plain language) team and aggregate objectives for Release, so that all stakeholders understand all objectives
• All present commit to release objectives, or planning continues until a commitment can be achieved.

Release Planning Output

The output is a set of clear objectives, along with a prioritized feature list, for the upcoming release increment.

Organizing at Scale: Feature Teams vs. Component Teams – Part 3

As the title indicates, in the last two posts (Part 1 and Part 2 – and also be sure and check the comments from others) I’ve described the conundrum of organizing agile teams at scale, and said I’d provide some additional input from others along with some recommendations.

Should the Agile Enterprise Lean to the Feature Team Approach?

Given the advantages and disadvantages to each approach – essentially the balance of immediate-value-delivery vs. architectural focus and potential for reuse, the agile enterprise clearly leans towards the feature team approach, as there can be no question on the immediate value delivery velocity benefits and the focus of the teams.

However, as we’ll see below, the answer is not that simple, and it’s likely that a more balanced approach may be required.

Mike Cottmeyer opined:

” Our first really important decision when thinking about how to scale agile is determining if the feature team approach will work or if we should consider a component team model.  Feature teams are THE starting place for agile organizations.  They are loosely coupled and highly cohesive and their inherent independence from the rest of the organization makes product development significantly faster and easier to forecast.

That said… many organizations are dealing with issues that make the feature team approach difficult to implement at scale.  These companies are developing products that are actually systems of systems.  Integrating disparate systems… disparate technologies… disparate build environments… disparate locations… disparate cultures… and disparate organizational politics into a single integrated feature team is not always advisable for even a moderately sized development shop.  For these organizations… the component team model represents a viable scaling alternative.

Yes, there are increased coordination costs associated with running large projects through component teams. Hopefully those costs are offset by the value of multiple products leveraging those commonly shared components.  Many organization will have to take a hard look at the feature team approach and examine at what scale the feature team approach will break down… because at some scale they WILL break down. “

Even then, the Line is Blurry

Even in light of this advice, we must also recognize that features and components are both  abstractions and the line is not so clear. One person’s feature may be another’s component.

For example, TradeStation Securities builds an on-line trading system where “charting market data” is a key capability for the trader. A few collocated agile teams work together on the charting function. They are some of the world’s foremost experts in charting real time data of this type. On the surface, that looks like an excellent example of a feature team, as charting certainly is a major feature of the system.

However, when new online trading capabilities are developed, such as “trading foreign exchange currencies (Forex),” new chart functionality must be added. However, driving this new chart functionality are major components such as streaming data, account management and interfaces with Forex exchanges. Is the new feature value stream described as “trading Forex all the way through the specialty chart function?” If so, that would make an obvious vertical feature stream and the teams might reorganize by taking some members of each component team and create a new vertical feature team for Forex trading.

Or,  is the feature “trading of Forex” plus “charting Forex”, in which case the charting team is already organized appropriately? Is the charting capability a feature set or a component? Both? Does it matter what you call it?

Even in the case where it is clear what you call it, is a feature team always the best choice? Keith Black, VP of TradeStation Technologies, notes:

“When we first interviewed Scrum coaches their view of Agile was solely focused on feature teams, this threatened to kill the initiative. Online trading requires a great depth of expertise and industry knowledge at many different levels. We could not reasonably form feature teams that included members from every component area. For one we’d have to increase our staff, and secondly the feature teams would be too large if they covered every facet of the trading process end to end.

Therefore, for our transition to Agile we organized around component teams and through maturity we are now in special cases putting together feature teams where it makes sense. This works for us because feature teams are excellent at driving an initiative through completion.

However, in some cases they still don’t make sense. For example, if you have twenty feature teams and they all rely on a common component, such as a time-critical online transactional processing engine, it may be inadvisable to have 20 different teams sticking their hands into this critical component. Rather, you might choose to have these changes controlled by a single Agile component team that can broker the needs of the 20 teams and make sure they don’t step on each other’s toes, or jeopardize areas they don’t understand by making changes for their particular features.”

So The  Answer is Likely a Mix

In the larger enterprise where there are many teams and many, many features, and systems that are in turn composed of system, the answer will likely be a mix of feature teams and component teams. Indeed, even in the modest-sized agile shop, a mix is likely to be appropriate.  Ryan Martens, founder and CTO of Rally Software (as agile as an organization as any that I am aware of) sketched the following, five-agile-team org chart and its “feature paths” for me:

Figure 1: Five-team agile organization with various feature paths noted.

Ryan went on to note:

“While we don’t think of it in these terms as such, three of these teams (ALM1, ALM2 and PPM in the top) would be readily identifiable as feature teams. One, (Infrastructure and Ops,  on the bottom)” is clearly a component team. I don’t know what you’d call the one (Platform and Integration) in the middle, because it sometimes originates it’s own features, and sometimes is simply a supportive component for other features.”

He concluded with some sound advice:

“What I would recommend is that you don’t get too hung up on the labels and the organization at the start. We think of maturity in stages: Flow, Pull and Innovate. In Flow, it’s about getting organized into agile teams and getting the value stream flowing in an agile manner. For Pull, its about leaning the work in process and having backlog maturity so that teams can pull from the backlog and implement in an order that makes the most technical and value sense. In Pull, I’d think a feature team could be a logical evolution for many. However, it’s getting started in Flow that is most critical and hardest step, and if you have to reorganize the enterprise before you begin, you may never begin at all.”

Recommendation: Lean to Features, Accept a Mix and Optimize Near Term for Collocation

Based on all this input and well-articulated arguments from both sides of the feature teams vs. component teams debate, what is my recommendation?

Interestingly enough, at real scale (think 30-40-50 teams), features are such spanning items, that this whole argument can become almost silly, and those cases, a broader “program based” (groups of teams collaborating) organization model (not unlike the charting teams feature/component pod above) must be considered.

But in the more typical, modest size enterprise or business unit (5-10 teams),  my recommendation is to apply feature teams wherever practical, but to optimize neither for features nor components per se, but to optimize around collocation. Pick feature teams if they are already collocated. Pick component teams if they are already collocated. The communication, team dynamics, continuous integration support,  and velocity benefits of collocation are likely to far exceed the benefits of the perfect theoretical organization, especially when the benefits of one approach vs. the other are not always so clear, as the various opinions indicate.

As your agile organization matures, you’ll be so competitive that you’ll have lots of opportunity to inspect and adapt your model and evolve your organization in a way that makes sense to you. But in the meantime, maybe nobody will have to move!

Organizing at Scale: Feature Teams vs. Component Teams – Part 2

In my last post, I reintroduced this topic, describing the conundrum of organizing large number of agile teams to better address value delivery for new end user features and services. I described two basic approaches, feature teams and component teams, and some arguments for each.

I’m still waiting for some feedback from a couple of others before I conclude with my personal recommendations, which I’ll post in the next day or so. In the meantime, Craig Larman courteously responded with his comments on my comments on his work in Larman [2009]. Since wordpress, as a capable (and free!) as it is, tends to bury comments at a level where you really have to want to see them to find them (not sufficiently “in your face” from my perspective), I thought I’d elevate Craig’s comments to this post, as he provides some clarifications on his work and my interpretation, as well as some more expansive thinking on the topic. Craig’s Comments from the prior post below are verbatim below.

Note: please also see additional comments from others on this and the prior post.

========================

“hi dean,

(someone pinged me that this topic was under discussion). thx for raising the topic.

bas and i did not write or intend to write that feature teams are the only rational way to organize teams. rather, the choice of any organizational design (including team structure) can be evaluated in terms of the global goal for optimization (e.g., maximizing fast value delivery, minimizing disruption, etc). from a toyota or lean perspective and the first agile principle, the global goal relates to early/fast delivery of value to real customers.

in the context of that measure, if one has design 1 with more delay and handoff and design 2 with less delay and handoff, design 2 is ‘better’ — with respect to that particular measure.

the ideal cross-functional cross-component feature team minimizes delay and handoff, whereas (not an opinion, just by pure systems logic..) component teams (which per definition are doing only part of the work for a customer feature) have more delay and handoff and create mini-waterfalls with single-specialist groups passing WIP items to each other (analysts, architects, programmers, testers, …).

but feature teams are only the ‘better’ model depending on the global goal — which is worth identifying in any discussion. and as you point out, we note that an org design such as feature teams may optimize flow of value, but it raises other problems (which are solvable). from a systems thinking perspective however, optimizing around addressing one of the other issues (e.g., minimizing org disruption) is a sub-optimization from the the perspective of value delivery… but not a suboptimization if the group agreed minimizing disruption was the system’s global goal.

if one visits an org that was organized around component teams and then they successful transition to feature teams, and then one asks if delivery of value is faster, the answer will be “yes.” it is also common when investigating such an org that they will say that planning and coordination are simpler, and that the multi-skilled learning that arises when working in feature teams has been useful to increase their degrees of freedom and to reduce the “truck number” of bottleneck specialists.

i look forward to your future comments, and those of others.

regards, craig

=============

===================
References:
Highsmith, Jim. Agile Project Management. 2004. Addison-Wesley.
Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises. 2007. Addison Wesley.
Larmon, Craig and Bas Vodde. 2009. Scaling Lean and Agile Development. Addison Wesley.

Organizing at Scale: Feature Teams vs. Component Teams – Part 1

While continuing my work with a number of larger enterprises facing the cultural change, new practice adoption and organizational challenges of a large scale agile transformation, the topic has again come up as to how to organize large numbers of agile teams to effectively implement agile and optimize value stream delivery.

For the smaller software enterprise, it’s usually no issue at all. Teams will naturally organize around the few products or applications that reflect the mission. The silos that tend to separate development, product management and test in the larger enterprise  (hopefully!) do not exist. The teams are probably already collocated, rather than being distributed across disparate geographies. Creating an agile team is mostly a matter of deciding what roles the individuals will play, and rolling out some standard training.

At scale however, like most other things agile, the problem is dramatically different and the challenge is to understand who works on what and where. How do we organize larger numbers of teams in order to optimize value delivery of requirements? Do we organize around features, components, product lines, services or what? While there is no easy answer to this question, the question must be explored because so many agile practices – how many backlogs there are and who manages them – how the vision and features are communicated to groups of teams – how the teams coordinate their activities to produce a larger solution  – must be reflected in that decision.

Organize Around Components?

In Scaling Software Agility,  I described a typical organizational model whereby many of the agile teams are organized around the architecture of a larger-scale system. There, they leverage their technical skills and interest and focus on building robust components – making them as reliable and extensible as possible, leveraging common technologies and usage models, and facilitating reuse. I even called the teams define/build/test component teams, which is (perhaps) an unfortunate label. However, I also noted that:

“We use the word component as an organizing and labeling metaphor throughout this book. Other agile methods, such as FDD, stress that the team be oriented around features, still others suggest that a team may be oriented around services. We use component rather liberally, and we do not suggest that every likely software component represents a possible team, but in our experience, at scale, components are indeed the best organizing metaphor.”

I wasn’t particularly pedantic about it, but noted that a component-based organization is likely to already exist in the enterprise, and that wasn’t such a bad thing, given the critical role of architecture in these largest, enterprise-class systems.

In any case, in a component-based approach, development of a new feature is implemented by the affected component teams, as the figure below illustrates.

In this case, the agile requirements workflow would put backlog items for each new feature on each of the affected component teams. They minimize multiplexing across features by implementing them in series, rather than parallel. Moreover, they are able to aggregate the needs of multiple features into the architecture for their component and can focus on building the best possible, long-lived component for their layer.

Perhaps this is reflective of a bias towards an architecture-centric view of building these largest-of-all-known software systems. For there, if you don’t get the architecture reasonably right, you are unlikely to achieve the reliability, performance and longer-term feature velocity delivery goals of the enterprise. If and when that happens, the enterprise may be facing a refactoring project that could be measured in hundreds of man-years. A frightening thought for certain.

In addition, there are a number of other reasons why component teams can be effective in the agile enterprise:

1. Because of its history, past successes and large scope, the enterprise is likely already organized that way, with specialists who know large scale data bases, web services, embedded operating systems and the like, working together. Individuals – their skills, interests, residency, friendships, cultures and lifestyles – are not interchangeable. It’s best not to change anything you don’t absolutely have to.
2. Moreover, these teams may already be collocated with each other, and given the strong collocation benefits in agile, organizing otherwise could potentially increase team distribution and thereby lower productivity.
3. Technologies and programming languages typically differ across components as well, making it difficult, if not impossible, for pairing, collective ownership, continuous integration, testing automation, and other factors critical to high performing agile teams.
4. And finally, at scale, a single user feature could easily affect hundreds of practitioners. For example, a phone feature like “share my new video to utube” could affect many dozens of agile teams, in which case organizing by feature can be a nebulous and unrealistic concept.

Organization Around Features?

However, our current operating premise is that agile teams do a better job of focusing on value delivery, and this creates a contrary vector on this topic. Indeed, as traditionally taught, the almost universally accepted approach for organizing agile teams is to organize around features.

The advantages to a feature team approach are obvious: teams build expertise in the actual domain and usage mode of the system, and can typically accelerate value delivery of any one feature. The team’s core competence becomes the feature (or set of features), as opposed to the technology stack. The teams backlog is simplified, just one or two features at a time. That has to promote fast delivery of high value added features!

Other authors support the Feature-focused approach as well. For example, in Agile Project Management, Highsmith [2004] states:

“Feature based delivery means that the engineering team build features of the final product.”

Of course, that doesn’t mean to say that the teams themselves must be “organized by feature”, as all engineering teams in the end build features for the final product, though perhaps that is a logical inference. Others, including Larmon and Vodde [2009] have more directly (and adamantly) promoted the concept of feature teams as the only rational (well, perhaps, see Craig Larman’s comments below) way to organize agile teams. They note:

“a feature team is a long lived, cross-functional team that completes many end-to-end customer features, one by one.. advantages include increased value throughput, increased learning, simplified planning, reduced waste…”

Larman and Vodde state specifically that you should “avoid component teams” and indeed, spend an entire chapter with good coverage of the aspects of the  topic. However, the authors also point out several challenges with the feature team approach, including the need for broader skills and product knowledge, concurrent access to code, shared responsibility for design and difficulties in achieving reuse and infrastructure work. Not to mention the potential for dislocation of some team members as the organization aligns around these boundaries.

So What is the Conclusion?

So what is an agile enterprise to do in the face of this conundrum? I’ve asked a few additional experts for their opinion, and I’ll be providing my opinion as well, in the next post.
===================
References:
Highsmith, Jim. Agile Project Management. 2004. Addison-Wesley.
Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises. 2007. Addison Wesley.
Larmon, Craig and Bas Vodde. 2009. Scaling Lean and Agile Development. Addison Wesley.

Lean and Scalable Requirements Information Model Published at Modern Analyst

The latest version of the Lean and Scalable Requirements Information Model for the Agile Enterprise is now published in the July ejournal of the ModernAnalyst.com. For those of you interested in trends in requirements practices, this is a good source for all things requirements and business analyst related, agile or not.

With respect to agile development, the agile methods category there has a plethora of good content.

Ten Suggested Practices for Applying Agile/Lean Software Management Principles to Other Knowledge Work

Introduction
Recently, I’ve been asked to help some teams apply lean/agile/Scrum/XP-like project management practices to non-software development knowledge work.
These organizations have seen agile methods produce huge benefits in visibility, productivity, quality, empowerment and motivation in their software teams. They naturally want to understand whether these techniques can be effective in other knowledge work activities such as managing IT projects, developing user documentation, managing and administering data server farms, implementing marketing communications, managing HR programs, and the like.

According to Wikipedia,

“a knowledge worker is an individual that is valued for their ability to interpret information within a specific subject area. They will often advance the overall understanding of that subject through focused analysis, design and/or development. They use research skills to define problems and to identify alternatives. Fueled by their expertise and insight, they work to solve those problems, in an effort to influence company decisions, priorities and strategies.”

Clearly, software development is a core knowledge work activity, in that it both results from, and creates, new knowledge. This knowledge emerges in the valuable, tangible form of code and test assets, as well as in the invaluable, intangible asset of the new knowledge gained and carried in the minds of the knowledge workers themselves.

Agile development methods have been developed primarily to unleash software development knowledge workers from the constraints and bounds of the existing, deficient software process, management and governance models. Given that they are proving to work so effectively, could these models, while designed for a different purpose, apply equally well in other knowledge work activities?

The answer is YES, these techniques can indeed be applied. However, it isn’t a simple, off-the-shelf application, as many of the agile practices, such as those around software coding practices – continuous integration, coding standards, test automation, pair programming and the like – may not be relevant in these contexts.

Therefore, in order to effectively apply these practices, we must make some modifications and simplifications. However, before we do so, it is important to first return to the underlying agile principles. If they are relevant, we will have a proper foundation for the agile/lean project management practice set that we can hopefully apply to general knowledge work.

Intended Benefits

Lets start with the motivation for making such a change – nothing less than a step-change increase in productivity, quality and morale of the teams with a resulting increase in market competitiveness. What enterprise wouldn’t want to do that?

Principles of Agile – The Agile Manifesto

I must admit that a few years back, I got bored with the mom-and-apple pie nature of always beginning with the agile manifesto (www.agilemanifesto.org) in any orientation to agile and I started to skip over it. That was a mistake, as later in the project I would come to learn that my clients and I were not working from the same set of assumptions and core beliefs. That caused unnecessary friction and roadblocks, based on unexpressed differences in philosophy.

These principles are still the foundation for all agile development today and I believe they apply equally well to other knowledge work. If we start with the manifesto itself  and make only a one-word substitution (solution for software), we get:

“We favor:

•    “Individuals and interactions over processes and tools
•    Working solutions over comprehensive documentation
•    Customer collaboration over contract negotiation
•    Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.”

Certainly the manifesto itself appears directly relevant to other knowledge work, but what about the important principles behind the manifesto? Again, no problem, for with a couple of minor word substitutions (noted in italics), we get the following:

“We follow these principles:

•    Our highest priority is to satisfy the customer through early and continuous delivery of valuable solutions.
•    Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
•    Deliver working solutions frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
•    Business people and developers must work together daily throughout the project.
•    Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
•    The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
•    Working solutions are the primary measure of progress.
•    Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
•    Continuous attention to technical excellence and good design enhances agility.
•    Simplicity–the art of maximizing the amount of work not done–is essential.
•    The best architectures, requirements, and designs emerge from self-organizing teams.
•    At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”
Again, no problem whatever for general knowledge work, assuming of course that the subject enterprise really believes in these principles!

Principles of Lean

As if the agile principles weren’t enough to drive knowledge work, we have the entire body of lean to apply as well. Based on my experiences  in lean manufacturing as well as lean software development, I suggest that the following lean principles apply directly to managing other knowledge work.

Ten Suggested Practices for Applying Agile/Lean Software Management Principles to Other Knowledge Work

Based on this background in agile and lean development, I’d suggest the following ten practices that can be applied directly to knowledge work development.

# 1 – Develop and empower, self-organizing, self-managing teams. Energize with vision, mission and time pressures. Provide requisite creative chaos, care and support.

# 2 – Plan work in short (one or two week) iterations. Teams plan together and commit to value delivery objectives in these increments.

# 3 – Focus on value delivery. Apply user story form (“as a <user role>, I can <do something> so that I can <business value>) to help assure value delivery.

# 4 – Develop, single prioritized backlog of work for the team. Establish product owner (or product owner team equivalent) roles to manage and prioritize backlog.

# 5 – Apply, daily 15 minute standup meeting as a primary form of communication and commitment. (What I did yesterday. What I’m doing today. Whether I am blocked or not.)

#6 – Minimize work in process to increase productivity. Develop work in process limits to task and stories. Minimize multiplexing within time boxes.

#7 – Plan for delivery of larger enterprise initiatives in larger (release) time boxes. Engage stakeholder in periodically (approximately quarterly) planning, visioning and commitment setting. Make vision, objectives and commitments visible and public.

# 8 – Provide total, real time visibility. Build big visible chart to show work in process and individual and team responsibilities. Speak directly to facts and issues. Publish iteration and release objectives.

#9 – Develop shared knowledge. Institutionalize knowledge by pairing on tasks, projects and stories. Avoid over-specialization so work force can flex to backlog and resource bottlenecks.

#10 – Apply work physics and agile planning. Estimate and track time completion by story and task. Establish and apply team velocity (capacity of work achievable in a time box).

Summary

In this post, I’ve posited that most of the core principles of lean and agile software development apply equally well to other enterprise knowledge work projects and activities. As a basis, I recapped many of the agile and lean principles with a bit of a knowledge work-spin. To get things started, I’ve also provided a set of ten suggested lean and agile practices that an enterprise can apply to non-software development, knowledge work activities. I believe that you will find them to be unambiguously good, and will help you achieve many of the benefits – including improved productivity, quality and morale – that every enterprise covets. Feel free to “try these at home”.