My Agile Denver Presentation from Last Night

Last night I presented to the local Agile Denver group. This was a variant of my Agile 2010 presentation, but one with a heavier focus on the scalable requirements (backlog) model. The title is:

Scaling Software Agility:
Agile Software Requirements

The content and flow is:

1.Lean and Scalable Requirements Model

Applying the Model

2. The Agile Release Train

3. Rearchitecting with Flow

4. Agile Portfolio Management

I promised to post it, so here it is:

Agile Software Requirements (Agile Denver).pptx

Thanks to Brad Swanson and Bob Hartman for arranging this opportunity.

Presentation: Rearchitecting Enterprise Class Systems

I’ve been using this presentation to provide overview context for some larger enterprises who have interest in the topic of rearchitecting  large scale systems in an agile manner. Of course, it isn’t a trivial topic as the problem itself is quite complex, and this solution requires a “trifecta”  of three advanced practices:

① Lean and Scalable Requirements Model (more)

② The Agile Release Train (more)

③ An Architectural Epic Kanban System (more)

The presentation is here: Rearchitecting Enterprise Systems.pptx

This presentation pulls these three elements into a more cohesive overview, albeit it one that still requires some voice over explanation.

Rearchitecting with Flow

I’m happy to report that the final chapters of the agile requirements book are largely converging. It now appears as it might actually be finished someday. In the meantime, for those of you who have been interested in the topic of agile architecture, I’ve completed the next draft of Chapter 21 – Rearchitecting with Flow and posted it here for public review and comment.

I think it provides some innovations that might help  larger agile enterprises  reason about and implement system-level architecture in a lean and agile manner.

I’d like to thank the architects and enterprise agilists at F-Secure Corporation, including KE Tan, Gabor Gunyho, Santeri Kangas, Ari Inki and James Cooper,  for their substantial contribution to this Chapter. Even more importantly, they are putting a variant of this system to work right now so we’ll have more feedback over the next few months  However, any errors in thinking, logic, design or description are solely my own.

Agile Architectural Epic Kanban System: Part 3 – State Machine View

In the last few posts, we’ve been describing an architectural epic kanban system which addresses the objectives we described in this post. Last week, I met with system architects (and agilists) Santeri Kangas, James Cooper, Kuan Eeik Tan and Gabor Gunyho of F-secure Corporation to discuss the model further. We decided that while we liked the big kanban system graphic, it left many ambiguities as to how an epic transitions from state to state, including (for example) how one can be deleted.

To this end, we put together a state machine model to further elaborate the activities and decision properties of the system. It appears in the graphic below.

The state machine table follows:

In turn, this causes a few minor modifications to the kanban system graphic, which appears here:

By the way, for an interesting perspective on architecture in agile in general, you might want to check out this Architecture in an Agile World presentation by James Cooper.

An Agile Architectural Epic Kanban System: Part 2 – The Model

In the last post, I described the context and objectives for a kanban system for identifying, analyzing and implementing architectural epics in the agile enterprise. This work is being developed in collaboration with Santeri Kangas and others of F-Secure Corp., who are contributing to the model and applying it in their agile development shop. In  this post, we’ll introduce the proposed model.

For additional context for this work in process, architectural epics are first class citizens of the agile enterprise in the Big Picture Series (see below)

And the epics themselves are described in the Lean and Scalable Requirements Model:

In the last post, we described the role of system architects and “intentionally emergent architecture” in enterprise class agile development. We also described four primary business factors that drive new, cross cutting, architectural epics:

1. New product or service opportunities.
2. Changes in technologies that affect multiple products and services.
3. Some architecturally-centric problem within the existing solution set.
4. Common infrastructure and avoidance of over-investment.

In order to assure that we address the development of architectural runway in a lean and agile manner, we proposed using a kanban system. We also described the three primary objectives for such a kanban system:

1. To make architectural work in process (AWIP) visible
2. To establish AWIP limits to help assure product development flow
3. To drive an effective collaboration with the development teams

Introducing the Architectural Epic Kanban System

Of course, we probably wouldn’t have invested this time if we didn’t have any idea how such a system might work, and indeed, we have been developing it for the last few months. And, as readers of this blog already know, I like to express concepts visually first, so the reader doesn’t have to pour through a ton of text just to build their own mental model for what it is, and how it works.

So, here is another big picture graphic that highlights the relevant elements of the proposed system.

Architectural Epic State Descriptions

The underlying assumption for the kanban system is that epics go through a series of four states, each characterized by different activities on the part of the architecture and development teams and correspondingly increasing levels of investment. They are as follows:

Problem/Solution Needs Identification: The Funnel

The Funnel state is the “capture” state. In this state all new “big ideas” are welcome. They can come from any source. They need no business case or estimates. Tooling is trivial – a document , spreadsheet or simple list on the wall will typically suffice. Since investment is minor, this state is not WIP limited; all ideas are captured for consideration. Funnel epics are discussed periodically. Based on decision criteria, epics in this state can be a) promoted to the next state, b) left in this state, or c) deleted.

Analysis: Backlog

The Analysis:Backlog state is more meaningful; epics that reach this state warrant further investment of time. These epics are roughly sized (we’ve indicated t-shirt sizes to avoid overinvesting in estimates). Investment is controlled to discussion level, and perhaps some very preliminary and lightweight investigation. The epic may be elaborated to a paragraph or two.

Since the investment is increasing, this state is WIP limited, primarily by the capacity of the architecture team.

Analysis:Backlog epics are discussed periodically. Based on decision criteria, epics in this state can be a) promoted to the next state b) demoted back to Funnel state, or c) deleted.

Analysis: Evaluation

Epics which are promoted to this next state deserve a more rigorous analysis, and require further investment. An architect is typically assigned as the epic owner. An active collaboration with development is initiated. Design alternatives are explored. Options for internal development and/or outsourcing, including 3rd party solutions acquisition, are considered. The affected development teams may estimate the epic. A lightweight business case, with a go or no-go recommendation, is developed. If 3rd party solutions are evaluated, the recommendation is given in priority order.

This state uses scarce resources, so it is WIP limited based on capacity of the architecture and development teams.

Epics that meet the go criteria are promoted to Implementation; otherwise, they are deleted from the queue.

Implementation

In this state, the epic is moved to implementation and the primary responsibility for the epic is passed to the development teams. This hand-off is effected when the epic appears in release planning. Architect resources remain available on a “pull” basis. (i.e. the responsibility for implementation rests with the development teams, but the architect assists the teams and shares responsibility until the team has developed an understanding of the work required.

This state is WIP limited by the capacity of the development teams and architect’s.

Promotion from State 3 to state 4 is an important economic decision for the enterprise that can be made only by the appropriate authority, based on the developed business case.

Summary

In this and the prior post, we’ve introduced the motivation and the model for an architectural epic kanban system. In follow on posts,  we’ll develop the model further.

An Agile Architectural Epic Kanban System: Part 1 – Introduction

I’ve been building up an internal “head of steam” and some new content for the architecture chapter for my upcoming book on Agile Requirements. It’s one of the later chapters, but an important one as the topic of building ever-larger scale systems with agile methods is becoming increasingly important in the enterprise setting.

I’ve written about the need for some amount of intentional “architectural runway” in agile development in the SSA book and in this blog, so I won’t repeat it here. For that, you can look at the Agile Architecture category, or the whitepaper (Principles of Agile Architecture) I wrote with Mauricio Zamora and Ryan Martens.

By some almost-intentional-serendipity (yes, an oxymoron, but apropos nonetheless), I’ve been working with an enterprise who is puzzling on the same issue. The company is F-Secure, from Helsinki Finland, and they have been implementing and extending agile methods across their enterprise (hundreds of developers) for over four years. Fortunately, they also are inclined to talk publicly about their agile experiences and that means that we can both work together, and publish together – a perfect opportunity for each us to develop and implement some new lean-agile practices.

At F-Secure, Chief Architect Santeri Kangas leads a small team of system architects who are responsible for analyzing and addressing architectural concerns that are cross cutting, big vision things. These are the “architectural epics” that span development teams – consumer smart phone and PC point-application products, corporate server and gateway products, hosted security as service offerings and online storage services operated by operations teams  – as well as a growing suite of web applications.

As they describe it, new architectural epics are driven by four primary business factors:

1. New product or service opportunities. New, ideas, from the portfolio backlog (big things they’d like to do) that provide opportunity for growth of revenue and market share.
2. Changes in technologies that affect multiple products and services.
   Examples: new platforms and operating systems, mobile technologies, 64 bit chip sets, etc.
3. Some architecturally-centric problem within the existing solution set. Examples: performance, size, security, usability, upgradability, compatibility, etc.
4. Common infrastructure and avoidance of over-investment.
   This last factor is driven by the need to make sure that certain components are built only once and reused by many teams, thereby avoiding having multiple teams invest in the same code. Even if “time-to-market trumps potential-for-reuse”, as it often can in agile, some common, reusable components can and do provide substantial long-term user (usability, fitness for use) and business (economic) benefits.

Clearly, this is an important charter for the F-Secure architecture team. Moreover, since the development teams are agile (indeed they are now implementing an Agile Release Train at the product line level), the identification, analysis and implementation of large-scale architectural epics must be agile too. Otherwise, at best we’ll have a clash of mental models; at worst, the company might not achieve the full benefits of top-to-bottom enterprise lean and agile practices.

Therefore, we’ve agreed to collaborate on a set of architectural practices that bring their system architectural efforts to a “lean and agile par” with that of the development teams. We’ll agree on a common framework, implement it, and eventually, report on the results. The output will likely be a whitepaper that we’ll submit to one or more US and Scandinavian agile and lean conferences. The tentative title is: A Lean Method for Analyzing and Implementing Architectural Epics in the Agile Software Enterprise

In a series of blog posts, starting with this one, I’ll be describing our initial thinking. Eventually, we’ll be able to report on the actual results.

Objectives of the Method

A lean, flow-based model for moving from architecture to implementation would accomplish three objectives:

1. Make architectural work in process (AWIP) visible.
   Lean thinking drives us to make sure that all work is visible. As Reinertsen points out, invisible, development work in process is WIP nonetheless. Worse, since it can’t be seen, it has “no natural predators” and therefore there is natural tendency to overload those involved in such work. (Since we can’t see it or quantify it, why not do some more of it?). Our kanban system must make AWIP visible so that it can be owned and managed responsibly. Architectural backlogs, queues and analysis work in process must be made visible, creating a shared understanding of current and future work and workload
2. Establish AWIP limits to help assure product development flow. Limiting wip helps achieve flow by making sure that input matches output, thereby avoiding the turbulence and trashing caused by overload. First, we’ll limit local AWIP to only that work that the architecture teams can actually do, thereby assuring that the architects are not thrashing across too broad a workload – starting many projects – but finishing far fewer. That will increase the efficiency, productivity and quality outputs of the architecture team. Secondly, in doing so, we will also be consciously limiting global WIP. This includes upstream, portfolio WIP (projects that drive new architectures) and downstream, development WIP. (Projects that consume new architecture). In this manner, we’ll match input objectives to implementation constraints, all across the enterprise.
3. Drive an effective collaboration with the development teams. The tension between architecture and development (implementation) is obvious. Eventually, all approved architectural epics are going to be implemented by the development teams. The last thing we want to do is to either surprise them with new stuff (“if we would only have known that sooner, we wouldn’t have spent all this time ….”) or hold them accountable for implementing plans that they don’t feel are actually workable.  If our model drives effective communication between these teams, things will naturally flow more smoothly.

Summary

So there you have the context and objectives for this blog series. Stay tuned for content.

Agile Requirements Book: First Chapter on Architecture

The Agile Requirements book chapter originally entitled “Epics and Enterprise Architecture” has been mostly written but has been split into two chapters. Chapter 19, now titled Agile Enterprise Architecture is now available in draft form and has been pushed to the book resource page. The second chapter on architecture, Achieving Architectural Flow, will likely follow in a month or so.

Comments are of course, welcome, and I’d be particularly interested if anyone has input or opinions on some of the newest content, particularly the later sections on Implementing and Splitting Architectural Epics.

New Whitepaper: A Lean and Scalable Requirements Information Model for Agile Enterprises

For any who have been following the requirements blog series and category, my collaborator, Juha-Markus Aalto and I have just completed a whitepaper which integrates the lean and scalable requirements information model with the Big Picture Series in a narrative format. It’s not quite as deep as the blog series and some content (like Use Cases) has been left out to make it shorter and hopefully, more understandable. It is posted on the resource page.

This will be the topic of my keynote at the upcoming Lean and Kanban 2009 Conference.

Enterprise Agility-The Big Picture (12): Architectural Runway

Note: In the post Enterprise Agility: The Big Picture, we introduced an overview graphic intended to capture the essence of enterprise agility in a single slide. In prior posts, we’ve discussed Agile Development Teams, Agile System Teams, Iterations , Agile Product Owner, Backlog, User Stories and the Iteration Backlog , Release , Vision and Release Backlog , The Roadmap, Agile Product Manager, and the Release Management Team. In this post we’ll describe Architecture Runway [callout 12] below.

Big Picture 12-Architectural Runway

For any familiar with this blog or my book (Chapter16 of SSA – Intentional Architecture), the topic of agile, intentional architecture is not a new topic. The blog series on this topic is here. A whitepaper in this topic, Principles of Agile Architecture: Intentional Architecture in Enterprise-Class Systems, coauthored by myself, Ryan martens and Mauricio Zamora can be found here. Also, I recently presented a Rally webinar on this topic and you can see that webinar here.

In Scaling Software Agility, I defined architectural runway as:

A system that has architectural runway contains existing or planned infrastructure sufficient to allow incorporation of current and near term anticipated requirements without excessive refactoring.

Continuous build out and maintenance of new architectural runway is the responsibility of all mature agile teams. Failing to do so will call cause one of two things to happen, either of which is very bad:

1. Release dates will be missed as large scale, just-in-time, in-situ infrastructure refactoring adds substantial risk to scheduling
2. Failure to address the problem systemically means that the teams will eventually run out of runway. New features cannot be added and the system becomes so brittle and unstable that it has to be rewritten from the ground up.

Architecture in the Big Picture

With respect to the Big Picture, Intentional Architecture appears as a “red thread” through various levels of the hierarchy.

• Level 3 – At Level 3, Architectural Runway is represented by Infrastructure initiatives that have the same level of importance as the larger scale requirements epics that drive the company’s vision forward. While many architectural initiatives will be addressed routinely by the teams over time, others require elevation to the portfolio level to assure awareness and communication of these important initiatives. They will consume substantive resources and failing to implement them will compromise the company’s position in the market. They must be visible, estimated and planned just like any other epic.
• Level 2 – At Level 2, Product Managers, System Teams and other stakeholders translate the architectural epics into features that are relevant to each release. They are prioritized, estimated and resourced like any other feature. At each release boundary, each architectural initiative must also be conceptually complete, so as to not compromise the new release, though it may or may not surface itself to the user.
• Level 1- At level 1, refactors, design spikes, evaluations etc. that are necessary to extend the runway are simply a type of story that is prioritized like any other story. Architectural work is visible, accountable and demonstrable at every iteration boundary. This is accomplished by agreement and collaboration with the system architects (Level 2) and agile team and tech leads (Level 1) who determine what spikes need to happen when, and who work with the Product Owner to prioritize the iteration backlog. (see picture below).
  

  Architecture is a role collaboration

In this manner, architecture is a first class citizen of the Big Picture and is a routine portfolio investment consideration for the Agile Enterprise.

In the next Post, we’ll elaborate on the Agile Portfolio Vision.

Upcoming September 10, 2008 Webinar – Principles of Agile Architecture

For those interested in the Agile Architecture Series, I’ll be presenting a webinar on this topic on September 10, 2008 at 1:00PM Eastern time. The webinar Principles of Agile Architecture: Intentional Architecture in Enterprise Class Systems is described below:

“As Agile development practices cross the chasm to the enterprise, many interesting debates continue about the role of systems architecture. As the continuous refactoring of emerging-only architectures becomes less practical as the system size and complexity grows, there is a natural desire to minimize unnecessary refactoring as well as to build systems from components that work in a like manner. This defines a role for “Intentional Architecture” in agile, enterprise-class systems development. In this Web seminar, learn about seven core principles that can be applied to architecting these large scale systems without sacrificing the principles or benefits of true software agility.”

This is part of a series sponsored by Rally Software, in collaboration with Sticky Minds and Better Software Magazine, designed specifically for the Enterprise Agilist. This webinar series is designed around specific topics of interest to the larger enterprise adopting agile methods. The description from the website is below:

“Join leading Agile experts, coaches, and authors—including Jean Tabaka, Dean Leffingwell, Stacia Broderick and Ryan Martens—as they paint a picture of successful Enterprise Agile adoption from a team’s first project to enterprise-wide transformation. During this Web seminar series, you will learn proven organizational transition approaches and best practices around Agile project and product management, Agile architecture and quality management. Development, management, and product teams will gain both an understanding of what it means to transition to Agile as well as specific best practices for their role in the organization.”

The series is now available for registration at: http://www.bulldogsolutions.net/SQE/SQE09242008/calendar.aspx?bdls=15887.

And it is FREE!