In the last post, I described a system for estimating features that will be in my upcoming book. This post, prioritizing features, is that post’s evil twin. Together, it should be provide a reasonable strategy to estimate cost and schedule for a feature, and a systematic way to pick which features will deliver the highest value first, based on the cost of delay. For context, you may want to browse the Big Picture category and the Agile Requirements category.
Introduction
One of the largest challenges that all software teams face is prioritizing requirements for implementation and delivery to the customers. This is certainly a challenge for every agile team at iteration boundaries, and it rises to even greater importance when prioritizing features at the program level. Here, small decisions can have big impacts on implementation cost and timeliness of value delivery.
There are a number of reasons why prioritization is such a hard problem:
- Customers are seemingly reluctant to prioritize features. Perhaps this is because they simply “want them all”, which is understandable; or perhaps they are uncertain as to what the relative priorities are; or perhaps they cannot gain internal agreement.
- Product managers are often even more reluctant. Perhaps this is because that if they could only get them all, they wouldn’t have to prioritize anything and more importantly, they would be assured of receiving all the ultimate value[1]
- Quantifying value is extremely difficult. Some features are simple “must haves” to remain competitive or keep market share. How does one quantify the impact of keeping market share, one feature at a time?
To assist us in our efforts, we often attempt to provide a Return on Investment (ROI) per feature, by predicting the likely increase in revenue if a feature is available. Of course, investment in determining ROI is most likely a false science, as no ones crystal ball is an adequate predictor of future revenue, especially when you attempt to allocate revenue on a per-feature basis. This is compounded by the fact that the analyst that does the work is likely to develop a vested, personal interest in seeing that the feature is developed. Plus, any product manager or business analyst worth their salt can probably make a case for a great ROI for their feature – otherwise they wouldn’t have worked on it to begin with.
In agile, however, the challenge of prioritization is immutable – we admit up front that we can’t implement (nor even discover) all potential requirements. After all, we have typically fixed quality, resources and delivery schedule. Therefore, the only variable we have is scope. Effective prioritization becomes a mandatory practice and art; one that must be mastered by every agile team and program.
Of course, prioritizing requirements is not a new problem. A number of authors have described reasonable mechanisms for prioritization. Our favorites include
- Agile Estimating and Planning [Cohn. 2006]
- Software Requirements [Wiegers. 1999.]
- Software by Numbers [Deene and Cleland-Huang [2004].
For those for whom this topic is potentially a determinant of program of success or failure of the program, we refer you also to these bodies of work. While we will take a different approach in this post (and upcoming book), there is certainly no one right way to prioritize and teams will benefit from differing perspectives on this unique problem.
Value/Effort as an ROI proxy – A First Approximation
As we alluded to above, we have traditionally prioritized work by trying to understand the relative Return on Investment, which is the relationship between potential return (value) divided by the effort (cost to implement) for a feature. At least the model was simple:
If we could simply establish value and cost – if not in absolute terms, then at least relative to other features – then we have a way to prioritize based on economics. After all, who wouldn’t want to deliver a higher ROI feature before a lower ROI feature? That seemed to make totally intuitive and (apparently) economical common sense.
What’s Wrong with our Value/Effort ROI Proxy?
However, based on more complete economic framework, it turns out that relative ROI (as we’ve too simply defined it above) is not an adequate proxy for prioritizing value delivery. Recently, one of the more thoughtful and rigorous economic views to prioritizing value delivery (sequencing work in flow terms) is Reinertsen’s Principles of Product Development Flow [Reinertsen 2009] that we introduced in an earlier post. These principles describe methods of sequencing work based on the mathematics and underlying economics of lean product development.
We’ll use those principles to describe an enhanced method for prioritizing features. As we do so, we’ll discover a deeply seated flaw in our first assumption – the assumption that a high relative ROI project should naturally have precedence over a lower ROI project.
Instead, what need to understand is the way in which the economics of our program may be dramatically affected by sequence. For example, the potential profit for a particular high ROI feature could be less sensitive to a schedule delay than a lower ROI feature. In this case, the lower ROI feature should be implemented first, followed by the higher ROI feature. This may not make intuitive sense, but we’ll see that it does make economic sense.
Prioritizing Features Based on the Cost of Delay.
Since prioritizing value delivery is the key economic driver for a program, we’ll need a more sophisticated model to produce better returns. To build the model, we’ll make a small detour through some of the fundamentals of queuing theory. After all, as we described in another post, prioritizing value delivery in software development is just a special case of queuing theory. So applying those principles will should create a solid foundation for critical decision-making.
Introducing Cost of Delay
As Reinertsen points out “if you only quantify one thing, quantify the cost of delay.”[2], so we’ll need to be able to estimate the CoD as part of our more economically-grounded approach. (More on that shortly). Fortunately, however, we don’t have to only quantify one thing, and as we have already outlined an estimating strategy, we’ll actually be able to quantify two things, the feature effort estimate, as we have described in the last post, and the Cost of Delay. Together, we should have what we need.
In achieving product development flow, Reinertsen describes three methods for prioritizing work based on the economics of Cost of Delay.
Shortest Job First
When the cost of delay for two features is equal, doing the Shortest (in our case, smallest) Job First, produces the best economic returns, as is illustrated in the Figure below.
Shortest Job First
Fro
Indeed the impact is dramatic, as doing the smallest job first has economic returns that are many times better than doing the larger job first. So we arrive at our first conclusion:
If two features have the same Cost of Delay, do the smallest feature first.
High Delay Cost First
If the size of the jobs are about the same, then the second approach, High Delay Cost First, illustrates the effect of prioritizing the jobs with the highest cost of delay. Again, the economics are compelling as the figure below illustrates.
High Delay Cost First
Of course this makes intuitive sense in this case as well (not that intuition has always led us to the correct conclusion). For if CoD is a proxy for value, and one feature has more value than another and it’s the same effort, we do it first; we knew that already from our ROI Value/Effort proxy. So we have our second conclusion:
If two features have the same effort, do the job with the highest cost of delay first.
Weighted Shortest Job First
Now that we have seen the impact, we understand that these two conclusions are quite sensible when the size or CoD of two features are comparable. Of course, we are not manufacturing widgets here. The Cost of Delay and implementation effort for different software features is highly variable. Plus, often they have weak or no correlations (i.e., some valuable jobs are easy to do, some are hard); that’s just the way it is with software. In the case where the CoD and effort for a feature are highly variable, then the best economics is achieved when we implement them in order of the Weighted Shortest Job First.
In this case, we calculate the relative priority weighting by dividing the CoD by the size estimate. If the CoD and job sizes vary greatly, then the differential economics can be dramatic, as illustrated in the figure below.
Weighted Shortest Job First
This, then, is our preferred approach for software development:
If two features have different size and CoD, do the weighted shortest feature first.
Estimating the Cost of Delay
This seems like a promising decision model for prioritizing features. It is based on solid economics and is quite rational once you understand it. However, we have excluded one small item – how does one go about calculating the cost of delay for a feature? If we aren’t careful, we could fall into the analysis trap we mentioned above – overinvesting in calculating size estimates for features – plus overinvestment in calculating CoD – could lead to too much overhead plus a potential bias by those doing the work. We need something simpler.
We suggest that CoD – so critical to our decision-making criteria – is in turn, an aggregation of three attributes of a feature, each of which can be estimated fairly readily, if compared to other features. They are user value, time value, and information discovery value.
- User value is simply the potential value of the feature in the eyes of the user. Product managers often have a good sense of the relative value of a feature, (“they prefer this over that”) even when it is impossible to determine the absolute value. And since we are prioritizing like things, relative user value is all we need.
- Time value is another relative estimate; one based on how the user value decays over time. Many features provide higher value when they are delivered early, and differentiated in the market, and lower value as features become commoditized. In some cases time value is modest at best (“implement the new UI standard with new corporate branding”). In other case, time value is extreme (“implement the new testing protocol prior to the school year buying season”), and of course there are in between cases as well (support 64 bit servers).
- Information discovery value adds a final dimension – one that acknowledges that what we are really doing is software research and development. Our world is laden with both risk and opportunity. Some features are more or less valuable to us based on how they help us unlock these mysteries, mitigate risk, and help us exploit new opportunities. For example, “move user authentication to a new web service”, could be a risky effort for a shrink-wrapped software provider that has done nothing like that in the past, but imagine the opportunities that such a new feature could engender.
With these three value factors, user value, time value, and information discovery value, we have the final pieces to our prioritization puzzle.
Feature Prioritization Evaluation matrix
Now, we can integrate all this information into an evaluation spreadsheet that we can use to establish the relative priorities for a feature, based on both effort and CoD. An example is shown in the table below below.
|
Cost of Delay |
|
Effort |
|
WSJF |
|
User |
Time |
Info. |
Total |
|
|
|
|
| Feature A |
4 |
8 |
8 |
20 |
|
4 |
|
5.0 |
| Feature B |
8 |
4 |
3 |
15 |
|
6 |
|
2.5 |
| Feature C |
6 |
6 |
6 |
18 |
|
5 |
|
3.6 |
|
|
|
|
|
|
|
|
|
| Legend: |
|
|
|
|
|
|
|
|
| Relative weighting Scale: 10 is highest, 1 is lowest. |
|
|
| Weight is calculated as Total Cost of Delay divided by effort |
|
|
Table 1: Spreadsheet for calculating weighted shortest job first
In our example, it is interesting to note that that Feature B – the job with the highest user value (8) and the highest feature/effort rating (1.3) – is actually the job that has the lowest weight, and therefore should be implemented last, not first. The job that has the lowest feature value (Feature A) actually produces the highest actual return on investment, so long as it is implemented first. So much for our intuition!
All Prioritizations are Local and Temporal
Reinertsen points out another subtle implication of WSJF scheduling. Priority should be based on delay cost, which is a global property of the feature, and effort, which is a local property of the team that is implementing the feature. In other words, a job with a lower relative feature value may require little resource from a specific team, and therefore should be implemented ahead of another higher priority feature, if that feature requires more resources for that team. This means that all priorities are inherently local[3].
This occasionally flies in the face of the way we often do things, whereby management sets a global priority for a project, which is to be applied across all teams. In that case, a lower priority task for a high priority project may take precedence over a high priority task for a lower priority feature. And availability of scarce resources doesn’t even enter into the equation. We see now that this approach simply doesn’t make economic sense.
In addition, we note that our model is highly sensitive to the time element – priorities change rapidly as deadlines approach. For example: “implement the new testing protocol in time for the school year buying season” could have a time value of “1-2” on the winter prior to the next school year start, but could easily be a “10” in May of the next year.
One conclusion of the above is that priorities have to be determined locally, and at the last responsible moment. That is the time when we can best assess the cost of delay and the resources available to work on the feature.
Fortunately, in our big picture model, we prioritize features on Release ( or PSI) Planning boundaries. Therefore the cadence we established for our release train will serve us well here, so long as we take the time to reprioritize at each such boundary.
When we do so, we can be confident that our priorities are current – taking into account then-current availability of resources and then-current cost of delay, as well as being based on solid economic fundamentals.
Achieving Differential Value – the Kano Model of Customer Satisfaction
Along with his colleagues, Noriaki Kano, an expert on the field of quality management and customer satisfaction, developed a model for customer satisfaction that challenged many traditional beliefs. Specifically, the Kano model challenges the assumption that customer satisfaction is achieved by balancing investment across the various attributes of a product or service. Rather, customer satisfaction can be optimized by focusing on differential features, those “exciters” and “delighters” that increase customer satisfaction and loyalty beyond that which a proportional investment would otherwise merit. The Kano model is shown in the figure below.
Kano Model of Customer Satisfaction
The model illustrates three types of features
- Basic (must have) features – features that must be present to have a viable solution.
- Linear features – features for which the investment is directly proportional to the result. The more you invest, the higher the satisfaction.
- Exciters and delighters. These are the features that differentiate the solution from the competition. They provide the highest opportunity for customer satisfaction and loyalty.
The primary insight from the Kano model is the position and shape of the lower and upper curves.
The shape of the basic curve is telling. Until a feature is simply “present”, the investment is proportional to the result, but satisfaction remains low until a threshold is achieved. Thereafter, however, additional investment produces a less than proportional reward. The center point of this curve gives rise to what is often described as the Minimum Marketable Feature (MMF). In other words, in order for solution to be considered viable, it must contain the requisite set of MMFs. However, enhancing or “gold plating” any MMF will not produce a proportional economic return.
The position and shape of the exciters and delighters curve tells the opposite story. Because these features are unique, compelling and differentiated, even a small investment (the area on the left) still produces high customer interest and potential satisfaction. Additional investment produces still more, and proportionally more investment produces still higher satisfaction. This is where we get the greatest leverage for our investment.
Prioritizing Features for Differential Value
Given that we have already described a full-fledged Weighed-Shortest-Job-First prioritization model, the question arises as to what additional benefit we can derive from Kano’s thinking? There are three strategic takeaways:
First, when competing on features in an existing marketplace, teams should place relatively high user value (and therefore a relatively high cost of delay) on features required to reach minimal parity. This leads us to rule #1:
Differential Value Rule #1 – Invest in MMFs, but never overinvest in a feature that is already commoditized.
Thereafter, the strategic focus should move to placing higher user value on differentiating features – those that will excite and delight the users, those for which competitive solutions have no answer, and those for which an incremental investment produces a nonlinear return. This leads us to rule #2:
Differential Value Rule #2 – Drive innovation by having the courage to invest in exciters.
Finally and most subtly, when we are forced to engage in feature wars with competitors that may already be ahead of us, it may not make sense to put all our investment into MMFs. After all, our competitors will keep investing too; what makes us think we can catch up? Instead, it may be better to slight some narrow category of MMFs and, instead, focus some amount of investment on exciters – even if we have not reached full basic feature parity.
Experience has shown that customers can be relatively patient with developers when they can a) reasonably anticipate the appearance of adequate MMFs and b) see the promise of the differential value of the exciters that the team is bringing forward from the backlog. This leads us to our third and final rule of feature prioritization:
Differential Value Rule #3 – If resources do not allow you to compete with the current rules of the game, play a different game.
Summary
In this post we’ve provided a method for prioritizing features based on a weighted Cost of Delay. It’s not so different from prior models, but it delivers superior economic returns, and it includes attributes of estimate size (cost), potential return and adjustments for risk (information discovery). We’ve also described the Kano Model which leads to an appropriate bias for investment in exciters and delighters. It may just be common sense; or perhaps its common sense buttressed by the real, yet subtle, economics of product development flow.
Comments welcome.
[1] There was once a Dilbert cartoon on this topic. The developer says to the customer, “please tell me what your priorities are so I’ll know what I don’t have to work on.”
[2] Principle of Product Development Flow E3: if you only quantify one thing, quantify the Cost of Delay
[3] Principle of Product Development Flow:F18: Prioritizations are inherently local.
Reference: Reinertsen, Don. The Principles of Product Development Flow: Second Generation Lean Product Development. Celeritas Publishing. 2010.
Scaling Software Agility 
