I’ve been using a “Release Predictability Metric” (RPM) on a number of larger agile programs. The goal of this key metric is to help the teams achieve a level of program predictability that their enterprise stakeholders can count on, while still allowing for taking on the reasonable risks that are necessary for innovation , as well as stretching to the maximum potential achievements in a PSI time box. In describing it to a colleague, I realized I had never blogged about it, so I’m posting this excerpted content from the new book to address that oversight.
Measuring Release Predictability
If you ask top executives what they would most like to see out of the software development process, many will answer “predictability”. And that is one of the many challenges in agile. We can reliably predict quality (by fixing it and adopting effective technical practices) and date (by fixing it) and cost (by fixing the team size and the PSI date) – but we can’t actually predict functionality – at least in the longer term. If we did, we’d be right back in the iron triangle that has served us so poorly in the past. Moreover, if we predicted and controlled functionality long term, then we’d have to temporarily ignore the variability and new opportunities the market presents. That isn’t agile.
However, as professionals, we must be able to provide our enterprise with a reasonably reliable predictor of upcoming events, at least near term, as well as some sense of the future product Roadmap that we intend to execute.
When implemented properly, the Agile Release Train can provide just such a predictability measure, at least for the next PSI (or maybe two). That gives the enterprise from 3-6 months of visibility into upcoming release content – enough to plan, strategize and support with market communications, release launches, etc. The release objectives that we established during release planning are our primary means to do this.
At release planning time, each objective is given a business value, assigned by the business owners for that program or subdomain. During each release retrospective, the teams meet with their business owners to self-assess the percentage of business values they achieved for each objective. This can be done both at the team and program level. For example, a program might rate its accomplishments as follows:
Actual Vs Plan Release (PSI) Objectives
Release Objectives Process Control Band
In the example above, the program accomplished 79% of its release objectives. The questions arises – how good, or bad, is that? To answer this, we must return to our lean principles and the context for the enterprise program itself.
On the surface, at least, it might appear that accomplishing 100% of release objectives is the only worthy goal. Closer analysis, however, tells us differently. In order for a team to routinely accomplish 100% of its release objectives, they must either
1) drive all risk out of the plan by eliminating or curtailing innovation and risk taking
2) back off so far on objectives so as to assure completion
Neither of these optimizes the economic impact of our efforts. To achieve that, we need to operate successfully in some acceptable process control band, so that the program has reasonable predictability, and yet allows for the variability, “optionality”, and stretch goals inherent with software development.
In our experience, a program that can reliability achieve most of its release objectives is a trusted program that is an extraordinary asset to the enterprise. In this case, the release predictability measure should fall in a process control band something like that in the following figure over time.
Managing uncertainty with a release (PSI) objectives process control band
In this figure, Team B is predictable, even though they do not routinely hit 100% of their release objectives. They can’t, otherwise there would be no room for innovation and risk taking. However, the enterprise can depend on the fact that they will do most of what they said they would do. Team A, however, is all over the map. It’s hard to manage any program or enterprise with the characteristics of Team A. You simply can’t depend on them to deliver anything like what they predicted.
By creating and measuring this predictability measure at every PSI, the enterprise can eventually achieve the right balance of predictability and risk taking, thereby achieving the optimum economic outcomes.