The Agile Release Train, Strategic Alignment and Product Development Flow

In this blog, in Scaling Software Agility, and in my forthcoming book on Agile Requirements, I’ve been writing fairly extensively on the implementation of the Agile Release Train (also see whitepaper derived from the book)  as a means of achieving both strategic alignment and product development flow in the larger software enterprise. It is hard to overstate the importance of the Agile Release Train in how it helps the enterprise achieve product development flow. I spend a lot of my time helping enterprises achieve the benefits of this model.

Recently, I’ve been reading (and rereading) Don Reinersten’s new book The Principles of Product Development Flow: Second Generation Lean Product Development. This book may be the best book on product development I have read in a decade. I strongly recommend it to anyone who finds my blog or the topic of scaling software agility of interest.

In addition to the depth of content, and its simultaneously light, yet rigorous, (is that really possible?) economic and mathematical treatment of flow, I like the way the book is organized around eight major themes of product development flow. They are:

1. Take an economic view

2. Actively manage queues

3. Understand and exploit variability

4. Reduce batch sizes

5. Apply WIP constraints

6. Control flow under uncertainty – cadence and synchronization

7. Get feedback as fast as possible

8. Decentralize control

For a meaningful discussion of these themes, I refer you directly to Reinertsen’s book.

Mapping The Agile Release Train to Reinertsen’s Themes

Perhaps not surprisingly, the Agile Release Train maps pretty directly to Reinertsen’s eight primary themes. Understanding this mapping is the key to understanding the criticality and motivation for the ART itself. So to make it easy for you, I’ll map them in this post:

Product Development Flow Theme 1 – Take an economic view.
The ART’s incremental releases substantially improve the ROI of software development by accelerating the release of value to the customer. This helps capture early market share and drives gross margins by delivering features to the market at the time when the market values them most highly.

Product Development Flow Theme 2. Actively manage queues.
The short, frequent planning cycles of the Agile Release Train actively manage queue lengths:

Team backlogs (queues of waiting stories), (see the mildly controversial post on this topic ) are generally limited to about the amount of work that can be accomplished in a single release increment (Or Potentially Shippable Increment). Planning much beyond that is generally not very productive for the teams, because teams know strategic priorities could change at any release boundary.

Release backlogs (queues of waiting features) are typically limited to those features that can realistically be implemented in the next release or two. Beyond that, product managers understand that they may be over-investing in feature elaboration for features that will never see the light of day.

Portfolio backlogs (queues of waiting epics and future projects) are typically limited to those epics that could likely find their way to release planning in the next six months or so. Too early, or too in-depth investment in business cases for those projects that will not be implemented is a form of waste.

Product Development Flow Theme 3. Understand and exploit variability.
Since a high degree of variability is inherent in software development, frequent re-planning provides the opportunity to adjust and adapt to circumstances as fact patterns change. New, unanticipated opportunities can be exploited by quickly adapting plans; critical paths and bottlenecks become clear and resources can be adjusted to optimize throughout through the bottlenecks and better avoid unanticipated delays.

Product Development Flow Theme 4. Reduce batch sizes.
Large batch sizes create unnecessary variability and cause severe delays in delivery and quality. The ART reduces batch sizes by releasing only those features to the development teams that are prioritized, elaborated sufficiently for development, and are sized to fit within the next release cycle. This avoids overloading the development teams with significant number of longer term, parallel development projects, which may or may not ever see the light of day, and in any case cause multiplexing, overhead, and thrashing. The transport (handoff) batch delay between teams is minimized as well, as face to face planning facilitates high bandwidth communication and instant feedback.

Product Development Flow Theme 5. Apply WIP constraints.
In release planning, teams plan their work and take on only the amount of features that their velocity indicates they can achieve. This forces the input rate (agreed-to, negotiated release objectives) to match capacity (what the teams can do in the release). The current release timebox prevents uncontrolled expansion of work; so the current release does not become a “feature magnet” for new ideas. The global WIP pool, consisting of features and epics in the enterprise backlog, is constrained by the local WIP pools, which reflects the team’s current backlog driven by the current release increment. When WIP is too high, lower value projects either a) never make it into development, or b) are purged during, or just prior to, release planning.

Product Development Flow Theme 6. Control flow under uncertainty – cadence and synchronization.
Planning – The release train planning cadence makes planning predictable, and lowers transaction costs (facilities, overhead, travel.) Planning can be scheduled well in advance, allowing participation by all key stakeholders in most planning events, making face to face information transport reliable and predictable.

Periodic re-planning (resynchronization) allows us to limit variance and misalignment to a single planning interval.

Releasing – The cadence and synchronization of regular, system-wide integration provides high fidelity system tests and objective assessment of project status at regular intervals. Transaction costs are lowered as teams invest in infrastructure necessary for continuous integration, automated testing, and deployment.  Since planning is bottom up, (performed by the teams and based on teams actual known velocity) and short term, delivery becomes predictable. Most all that has been planned should be reliably available as scheduled.

Product Development Flow Theme 7. Get feedback as fast as possible.
The fast feedback of the iteration and release cycle allows us to take fast corrective action. Even within the course of a release increment (or PSI), feedback is no more than two weeks (or the iteration length) away. Small incremental releases to customers allow us to track more quickly to their actual, rather than anticipated, needs. Incorrect paths can be abandoned more quickly (at worse, at the next planning cycle).

Product Development Flow Theme 8. Decentralize control.
Release plans are prepared by the teams who are doing the actual implementation, rather than by a planning office or project management function. Commitments to the plans are bottom-up, based on individual’s commitment to their teammates and team-to-team commitments reached during the planning cycle. Once planned, the teams are responsible for execution, albeit subject to appropriate lightweight governance and release management. Agile project management tooling automates routine reporting; management does not have to slow down the teams to assess actual status.

Summary

That’s it for this post, though I expect I’ll visit the topic again, as it’s certainly going to be a major theme of the next book. In the meantime, you might want to read Reinertsen’s new book, as it’s available now.

Agile Requirements Book: Chapter 1: A Brief History of Software Requirements Methods

To you software method historians out there (you know who you are, even if we can’t admit it to your  family and friends), I’ve just posted the first Chapter to Agile Requirements on the book resource page. It’s mostly context for what follows in the book, but I think it contains a pretty good summary of where we’ve been for the last 20-25 years, and where we will be likely going next.

More on Lean, Backlog and Little’s Law

In a recent post, An Agile Illusion: How That Nice Backlog is Actually Decreasing Your Team’s Agility, I described how long, well-formed backlogs can actually decrease a team’s ability to respond to the market. This post generated some lively comments – some supportive, some more critical – but all quite perceptive and representing a pretty high degree of sophistication and agile maturity.  These comments were refreshing (there may be hope for our industry yet…) and thought provoking  and I wanted to respond further in this post.

What drove me to write that lengthy post wasn’t the wait at Starbucks on that dreary Saturday morning. Rather, I wanted to make a few, fundamental points about agile and lean, and address a few things I have seen in practice that continue to bother me.

In rethinking that post, and now in summary, I want to make three key points:

Point #1 – Little’s Law Doesn’t Lie.

Little’s Law tells a fundamental truth – the longer the queue and the slower the processing time, the longer the (mathematically predictable) wait. This core law of lean makes intuitive, logical, and mathematical sense (actually, it was Little that did the formal proof, rather than inventing the law).

What it says, and what we understand from lean, is that long queues of work in process are fundamentally bad. Whether it be elaborated user stories in a backlog that won’t see the light of day, code that has been written but not tested, hundreds of manual tests awaiting automation, blobs of new code that haven’t been checked back into the baseline, a long set of portfolio level projects awaiting resources, etc. etc. etc., they are ALL Universally BAD, as they all decrease agility and time to market. (It’s hard to respond quickly to an order for red fenders with a metallic finish, if you have a boatloads of fenders primed for regular finish stacked up in front of the paint shop).

However, we also understand that, at least when it comes to backlog, Little’s Law isn’t a perfect analogy as a story can jump the queue in agile (I’d recommend that you do not try that at Starbucks, as those people in front of you likely really need that caffeine) and a system can decide not to process all items in the queue (Starbucks could do that to decrease queue length, but the long term economics would be quite unfavorable).

So when it comes to the backlog post, as some readers pointed out, we are not forced to assume that all backlog items will make their way into processing in an orderly fashion. We are smarter than that. And therefore Little’s Law cannot be blindly and universally applied. However, the fact that we are smarter than that does not prevent some undesirable behaviors that I often witness in practice. This brings me to points #2 and #3.

#2 – Don’t be Hostage to Your Own Backlog.

In many sprint reviews, I ask teams how they are doing with respect to the release objectives. Surprise, many times they are behind (and that’s one of the reasons we like agile – we are likely behind less than we would have been, we actually know it and its not too late to take corrective action).

When I ask them how they can “jump the queue” and meet the release objectives though they are behind, they often say – “well, we have all these backlog items we have to finish first. We have elaborated them, we understand them, we have invested in them, we have committed them to others, we believe them important and they are ready to go”.

However, I then comment something like , “as true as that may be, those are sunk costs and that doesn’t make those backlog items necessary, per se.  If the backlog you have isn’t the one you need to meet the release objectives, ignore the sunk costs and flush it”!

#3 – The Enterprise Can’t Be Held Hostage to the Team’s Backlog Either.

At enterprise scale; the problem is compounded. Often,  we approach release planning boundaries with 10-20 agile teams already having detailed and well structured backlogs for the existing work in process.  In that case, the enterprise has little or no agility, as nothing can really be changed. After all, if the teams must first work through their committed backlog, then Littles’ Law is in your face and the result is immutable – you can’t do squat at the enterprise level unless you do one of two things:

1)   wait the infinite amount of time it takes teams to work their local backlogs down to near zero (it’s infinite because other laws – the law of “teams don’t like having empty backlogs” the law of “we have too much technical debt” and the law of “work will expand to consume the time allotted to it” –  come into play)

2)   The enterprise must override the teams, or kill entire projects, in order to move forward with more globally aligned objectives.

Either choice will be painful and sub-optimal.

Wouldn’t it be far better if the system were leaner, and if the team’s backlogs were really short and lightweight, and they didn’t represent fixed commitments based on too-long-a-term thinking?

Backlogs are great, so long as you keep them short, extremely lightweight, and negotiable. Remember, never overinvest in some stories you may just not do. And never, ever, be help hostage by your own backlog.

Finally, if you haven’t yet read Don Reinertsen’s new book, Product Development Flow, I would sure recommend that you do so.

Lean Agile Enterprise Leadership Workshop

I continue working with a number of software enterprises in the throes of large-scale agile/Scrum rollouts. Whether it be a new rollout, or one where the next set of potential achievements and impediments rest at the door of management, one thing is increasingly clear: these rollouts will not reach their full potential until first, mid, and upper-level management is fully on board.

Since Scrum often starts bottom-up, or at least the training focuses almost exclusively on the team level, perhaps our expectation has been that awareness of the initiative at the management levels was enough. Perhaps we thought that that mangers would naturally fold into the mix and provide the requisite support and leadership needed for success at this next level. However, how we expected them to know how to do that was not so clear. Moreover, the popular chicken-pig Scrum story does not help. After all, who is the chicken and what is their role if it is not to be engaged actively in the operation of the teams? Even the role of the ScrumMaster, which in so many ways is a proxy for more effective management leadership of the team, can be as much of a conceptual barrier as it is a breakthrough. If the ScrumMaster mentors the team, what’s does the manager do? Wait patiently outside the room for a full report?

To this end, I have personally had to reset my expectations for managers in these large-scale lean/agile rollouts. It is simply not sufficient to be supportive. Rather they must be engaged, empowered and sufficiently knowledgeable to be able to lead, coach and drive the transition.

Doing, so however, requires some orientation and training which does not appear off-the-shelf from the Agile or Scrum community. To this end, I’ve created a two-day course that is designed specifically for managers and executives in such a transition. You can tell from the learning objectives that this is a serious undertaking:

1. Provide a principled, lean and flow-based product development foundation for improving competitiveness, economics and return on investment in product development

2. Introduce basic and advanced agile principles and practices as a contextual reference for a large-scale agile software development transformation

3. Introduce and experience Scrum as a potential mechanism for implementing team-level software agility

4. Introduce and explore the Agile Release Train as a means to provide strategic alignment and visibility across the enterprise

5. Introduce the agile technical and quality practices necessary for software teams to reach their highest productivity and quality potential

6. Provide a leadership framework that helps management facilitate an effective, large-scale, lean and agile transformation

Here is the abstract for the course:

Lean Agile Enterprise Leadership Workshop

It is a serious course, for serious people who understand the potential challenges, impact and competitive benefits of a lean agile transformation at enterprise scale. If this is potentially of interest to your enterprise, ping me.

Upcoming Public Lean|Agile Leadership Workshops in Australia

Most of my workshops are delivered at a client’s site. However, I do have two public workshops scheduled in Sydney (February 21-22) and Melbourne (Feb 28-29) in February 2012. You can register for these workshops through Agile University here: