Agile Architectural Epic Kanban System: Part 3 – State Machine View

In the last few posts, we’ve been describing an architectural epic kanban system which addresses the objectives we described in this post. Last week, I met with system architects (and agilists) Santeri Kangas, James Cooper, Kuan Eeik Tan and Gabor Gunyho of F-secure Corporation to discuss the model further. We decided that while we liked the big kanban system graphic, it left many ambiguities as to how an epic transitions from state to state, including (for example) how one can be deleted.

To this end, we put together a state machine model to further elaborate the activities and decision properties of the system. It appears in the graphic below.

The state machine table follows:

In turn, this causes a few minor modifications to the kanban system graphic, which appears here:

By the way, for an interesting perspective on architecture in agile in general, you might want to check out this Architecture in an Agile World presentation by James Cooper.

An Agile Architectural Epic Kanban System: Part 2 – The Model

In the last post, I described the context and objectives for a kanban system for identifying, analyzing and implementing architectural epics in the agile enterprise. This work is being developed in collaboration with Santeri Kangas and others of F-Secure Corp., who are contributing to the model and applying it in their agile development shop. In this post, we’ll introduce the proposed model.

For additional context for this work in process, architectural epics are first class citizens of the agile enterprise in the Big Picture Series (see below)

And the epics themselves are described in the Lean and Scalable Requirements Model:

In the last post, we described the role of system architects and “intentionally emergent architecture” in enterprise class agile development. We also described four primary business factors that drive new, cross cutting, architectural epics:

New product or service opportunities.
Changes in technologies that affect multiple products and services.
Some architecturally-centric problem within the existing solution set.
Common infrastructure and avoidance of over-investment.

In order to assure that we address the development of architectural runway in a lean and agile manner, we proposed using a kanban system. We also described the three primary objectives for such a kanban system:

To make architectural work in process (AWIP) visible
To establish AWIP limits to help assure product development flow
To drive an effective collaboration with the development teams

Introducing the Architectural Epic Kanban System

Of course, we probably wouldn’t have invested this time if we didn’t have any idea how such a system might work, and indeed, we have been developing it for the last few months. And, as readers of this blog already know, I like to express concepts visually first, so the reader doesn’t have to pour through a ton of text just to build their own mental model for what it is, and how it works.

So, here is another big picture graphic that highlights the relevant elements of the proposed system.

Architectural Epic State Descriptions

The underlying assumption for the kanban system is that epics go through a series of four states, each characterized by different activities on the part of the architecture and development teams and correspondingly increasing levels of investment. They are as follows:

Problem/Solution Needs Identification: The Funnel

The Funnel state is the “capture” state. In this state all new “big ideas” are welcome. They can come from any source. They need no business case or estimates. Tooling is trivial – a document , spreadsheet or simple list on the wall will typically suffice. Since investment is minor, this state is not WIP limited; all ideas are captured for consideration. Funnel epics are discussed periodically. Based on decision criteria, epics in this state can be a) promoted to the next state, b) left in this state, or c) deleted.

Analysis: Backlog

The Analysis:Backlog state is more meaningful; epics that reach this state warrant further investment of time. These epics are roughly sized (we’ve indicated t-shirt sizes to avoid overinvesting in estimates). Investment is controlled to discussion level, and perhaps some very preliminary and lightweight investigation. The epic may be elaborated to a paragraph or two.

Since the investment is increasing, this state is WIP limited, primarily by the capacity of the architecture team.

Analysis:Backlog epics are discussed periodically. Based on decision criteria, epics in this state can be a) promoted to the next state b) demoted back to Funnel state, or c) deleted.

Analysis: Evaluation

Epics which are promoted to this next state deserve a more rigorous analysis, and require further investment. An architect is typically assigned as the epic owner. An active collaboration with development is initiated. Design alternatives are explored. Options for internal development and/or outsourcing, including 3rd party solutions acquisition, are considered. The affected development teams may estimate the epic. A lightweight business case, with a go or no-go recommendation, is developed. If 3rd party solutions are evaluated, the recommendation is given in priority order.

This state uses scarce resources, so it is WIP limited based on capacity of the architecture and development teams.

Epics that meet the go criteria are promoted to Implementation; otherwise, they are deleted from the queue.

Implementation

In this state, the epic is moved to implementation and the primary responsibility for the epic is passed to the development teams. This hand-off is effected when the epic appears in release planning. Architect resources remain available on a “pull” basis. (i.e. the responsibility for implementation rests with the development teams, but the architect assists the teams and shares responsibility until the team has developed an understanding of the work required.

This state is WIP limited by the capacity of the development teams and architect’s.

Promotion from State 3 to state 4 is an important economic decision for the enterprise that can be made only by the appropriate authority, based on the developed business case.

Summary

In this and the prior post, we’ve introduced the motivation and the model for an architectural epic kanban system. In follow on posts, we’ll develop the model further.

An Agile Architectural Epic Kanban System: Part 1 – Introduction

I’ve been building up an internal “head of steam” and some new content for the architecture chapter for my upcoming book on Agile Requirements. It’s one of the later chapters, but an important one as the topic of building ever-larger scale systems with agile methods is becoming increasingly important in the enterprise setting.

I’ve written about the need for some amount of intentional “architectural runway” in agile development in the SSA book and in this blog, so I won’t repeat it here. For that, you can look at the Agile Architecture category, or the whitepaper (Principles of Agile Architecture) I wrote with Mauricio Zamora and Ryan Martens.

By some almost-intentional-serendipity (yes, an oxymoron, but apropos nonetheless), I’ve been working with an enterprise who is puzzling on the same issue. The company is F-Secure, from Helsinki Finland, and they have been implementing and extending agile methods across their enterprise (hundreds of developers) for over four years. Fortunately, they also are inclined to talk publicly about their agile experiences and that means that we can both work together, and publish together – a perfect opportunity for each us to develop and implement some new lean-agile practices.

At F-Secure, Chief Architect Santeri Kangas leads a small team of system architects who are responsible for analyzing and addressing architectural concerns that are cross cutting, big vision things. These are the “architectural epics” that span development teams – consumer smart phone and PC point-application products, corporate server and gateway products, hosted security as service offerings and online storage services operated by operations teams – as well as a growing suite of web applications.

As they describe it, new architectural epics are driven by four primary business factors:

New product or service opportunities. New, ideas, from the portfolio backlog (big things they’d like to do) that provide opportunity for growth of revenue and market share.
Changes in technologies that affect multiple products and services.
Examples: new platforms and operating systems, mobile technologies, 64 bit chip sets, etc.
Some architecturally-centric problem within the existing solution set. Examples: performance, size, security, usability, upgradability, compatibility, etc.
Common infrastructure and avoidance of over-investment.
This last factor is driven by the need to make sure that certain components are built only once and reused by many teams, thereby avoiding having multiple teams invest in the same code. Even if “time-to-market trumps potential-for-reuse”, as it often can in agile, some common, reusable components can and do provide substantial long-term user (usability, fitness for use) and business (economic) benefits.

Clearly, this is an important charter for the F-Secure architecture team. Moreover, since the development teams are agile (indeed they are now implementing an Agile Release Train at the product line level), the identification, analysis and implementation of large-scale architectural epics must be agile too. Otherwise, at best we’ll have a clash of mental models; at worst, the company might not achieve the full benefits of top-to-bottom enterprise lean and agile practices.

Therefore, we’ve agreed to collaborate on a set of architectural practices that bring their system architectural efforts to a “lean and agile par” with that of the development teams. We’ll agree on a common framework, implement it, and eventually, report on the results. The output will likely be a whitepaper that we’ll submit to one or more US and Scandinavian agile and lean conferences. The tentative title is: A Lean Method for Analyzing and Implementing Architectural Epics in the Agile Software Enterprise

In a series of blog posts, starting with this one, I’ll be describing our initial thinking. Eventually, we’ll be able to report on the actual results.

Objectives of the Method

A lean, flow-based model for moving from architecture to implementation would accomplish three objectives:

Make architectural work in process (AWIP) visible.
Lean thinking drives us to make sure that all work is visible. As Reinertsen points out, invisible, development work in process is WIP nonetheless. Worse, since it can’t be seen, it has “no natural predators” and therefore there is natural tendency to overload those involved in such work. (Since we can’t see it or quantify it, why not do some more of it?). Our kanban system must make AWIP visible so that it can be owned and managed responsibly. Architectural backlogs, queues and analysis work in process must be made visible, creating a shared understanding of current and future work and workload
Establish AWIP limits to help assure product development flow. Limiting wip helps achieve flow by making sure that input matches output, thereby avoiding the turbulence and trashing caused by overload. First, we’ll limit local AWIP to only that work that the architecture teams can actually do, thereby assuring that the architects are not thrashing across too broad a workload – starting many projects – but finishing far fewer. That will increase the efficiency, productivity and quality outputs of the architecture team. Secondly, in doing so, we will also be consciously limiting global WIP. This includes upstream, portfolio WIP (projects that drive new architectures) and downstream, development WIP. (Projects that consume new architecture). In this manner, we’ll match input objectives to implementation constraints, all across the enterprise.
Drive an effective collaboration with the development teams. The tension between architecture and development (implementation) is obvious. Eventually, all approved architectural epics are going to be implemented by the development teams. The last thing we want to do is to either surprise them with new stuff (“if we would only have known that sooner, we wouldn’t have spent all this time ….”) or hold them accountable for implementing plans that they don’t feel are actually workable. If our model drives effective communication between these teams, things will naturally flow more smoothly.