New Agile Enterprise Webinar Series

Rally Software, in collaboration with Sticky Minds and Better Software Magazine, has been building a free, fall webinar program specifically for the Enterprise Agilist. This webinar series is designed around specific topics of interest to the larger enterprise adopting agile methods. The description from the website is below:

“Join leading Agile experts, coaches, and authors—including Jean Tabaka, Dean Leffingwell, Stacia Broderick and Ryan Martens—as they paint a picture of successful Enterprise Agile adoption from a team’s first project to enterprise-wide transformation. During this unprecedented Web seminar series, you will learn proven organizational transition approaches and best practices around Agile project and product management, Agile architecture and quality management. Development, management, and product teams will gain both an understanding of what it means to transition to Agile as well as specific best practices for their role in the organization.”

My session, Principles of Agile Architecture: Intentional Architecture in Enterprise Class Systems, to be held on Wednesday, September 10, 2008 at 1:00 PM Eastern, is described below:

“As Agile development practices cross the chasm to the enterprise, many interesting debates continue about the role of systems architecture. As the continuous refactoring of emerging-only architectures becomes less practical as the system size and complexity grows, there is a natural desire to minimize unnecessary refactoring as well as to build systems from components that work in a like manner. This defines a role for “Intentional Architecture” in agile, enterprise-class systems development. In this Web seminar, learn about seven core principles that can be applied to architecting these large scale systems without sacrificing the principles or benefits of true software agility.”

I participated in building the series and I know the seminar presenters so I’m confident that this will be a quality series. The series is now available for registration at: http://www.bulldogsolutions.net/SQE/SQE09242008/calendar.aspx?bdls=15887.

Did I mention it is FREE!

Enterprise Agility–The Big Picture (4): Backlog

Note: In the post Enterprise Agility: The Big Picture, we introduced an overview graphic intended to capture the essence of enterprise agility in a single slide. In prior posts, we’ve discussed Teams, Iterations and the Agile Product Owner. To see the full series, select the “Big Picture” blog post category. In this post, we’ll describe three types of Backlog [callout (4)].

Big Picture 4 - Backlog

One of the difficult and interesting aspects to working with software process models is their lack of semantic precision. Based on my experiences in working with the UML team in the late 90s and my responsibilities for the RUP thereafter, I’m familiar with the difficulty in establishing a common semantic structure for software things. It takes collaboration, consensus and process maturity as well as some sort of governing authority to establish agreement. Little such infrastructure exists today in the agile community. That’s a bit of handicap in describing these methods and perhaps a measure of their relative immaturity (not to confuse maturity with efficacy, after all, the waterfall model is really mature!). At the same time, that’s probably one of the reasons we like agile – it’s not too late to explore and develop new methods and new meaning.

In this context, I would describe the word “Backlog” as the simplest and yet most overloaded term in agile. It’s not intended to be complicated and it really just means “stuff that needs to be done”. In its simplest form, it is primarily a repository for un-done user stories, that agile invention of value-added objects that we use to describe things our system needs to do for the user. (See Chapter 17 of Scaling Software Agility or Mike Cohn’s User Stories Applied, for a deeper treatment of user stories.).

However, in enterprise-class development things get more complicated and effective agile gets more complicated too. In one very large scale software development case, I saw a schema that had about six or seven different backlog types and while that seems complicated, it seemed to work ok in that company’s increasingly agile practice.

In the Big Picture, I’ve taken a stab at some simple clarifying labels for three common types of backlog that scale to the enterprise. However, the reader should be aware that there is no “backlog label consensus” in the industry that I am aware of, and I’m simply using these labels as I and some others have applied them. In any case, the labels matter less than the principles and it’s the hierarchical principle that I hope to make clear. In the Big Picture, I’ve indicated a three-level requirements hierarchy to hold three types of backlog. They are:

The Iteration (or Story) Backlog – holding user stories and the like that are generally intended to be implemented by a component team in their code baseline in the context of an iteration

The Release (or Feature) Backlog – holding features that deliver system-level value that typically affect multiple component teams and typically span iterations

The Portfolio (or Epic) Backlog – a placeholder for capturing and discussing the larger scale initiatives that an enterprise has, or intends to have, underway. Epics often affect multiple component teams, multiple systems, and even multiple products. Epics typically span iterations, releases, and sometimes years!

I’ll describe the first and hardest working of these, the Iteration (User Story) Backlog, in the next post.

Paul Beavers’ Agile Acid Test: Variability, Incremental Delivery, Empowerment

As agile moves across the chasm into the mainstream software enterprise, it becomes important to separate truly agile behavior (some call it Agile with the capital A) from a general approach to being flexible, not overly rigid in software practices and being open to changing market demands. Pete Behrens commented on the danger of “everybody being agile” in a post Agile Adoption on the Rise? Problems ahead? A few months back. Moreover, in today’s market, no intelligent executive would claim to NOT be leading an agile enterprise, so some discrimination in applying the term to teams and enterprises that have truly adopted and applied the basic agile software development principles seems warranted.

I’ve seen a number of attempts at simplifying the test of what it means to be truly agile but I’ve never attempted one myself. Now I don’t have too. Paul Beavers of BMC captured my opinion in a pithy manner in his recent post: Is it Possible to be Half Agile?

In that post, Paul describes three quintessential agile characteristics, and he does so in only four words (it would have taken me twenty or thirty!). Paul describes the three key principles as:

1. Variability
   
2. Incremental Completion
   
3. Empowerment
   

Personally, I’d edit that to become Variability, Incremental Delivery and Empowerment, and I’ll say why in future posts, but first, read the post from Paul to understand his thinking. Thanks Paul!

Successes in Release Planning: An Agile Enterprise Tipping Point

In the “Release Planning” category and series on this blog, I described an enterprise best practice designed to coordinate the efforts of some number of agile teams in order to deliver a larger scale system in a somewhat predictive and systematic manner. I also described this process from a practitioner’s perspective in the Agile Journal.

The first time I remember seeing such a session in largely this format was an effort led by Jean Tabaka of Rally Software at BMC in about 2005. Since then, I’ve seen this best practice employed to good effect in a number of larger-scale agile transformations and I have facilitated a number of such sessions myself. As can be seen in the series, Release Planning is an intense, face-to-face process where team members meet to establish a common vision, set objectives for a near-term internal or external release, identify and coordinate their interdependencies, plan the iterations that constitute the release, and finally, commit to release objectives.

In one instance, we combined a couple of days of agile training and an initial release planning session in an “all in” kickstart for about 100 people. I recently noted a blog post on this topic from some of the participants and it looks like the process, and more importantly, agile in general, is really taking hold in that enterprise. The post is ” The Tipping Point: Release Planning – #3″ , and I’ve taken the liberty of providing a few highlights of “what success starts to looks like”, below:

• “Vision – We were able to convey enough of a roadmap to declare a common long term goal. Every team had an equal focus on this goal and they understood their purpose. In past release planning sessions, some teams weren’t working on the highest priorities. This caused frustration and doubt.
  
• Resources – We kicked off release planning with a known set of teams and resources. We didn’t spend valuable hours trying to figure out who would be on what team. We still had a few resource shifts that weren’t ideal but it was limited to one area.
  
• Product Owners – We went into release planning with each iteration team having their most effective product owner …. we hired a few very promising product owners and we elevated a few past iteration team members to high potential product owners.
  
• Dedicated Agile Build Team – We finally convinced folks that it was important to have a dedicated build team. Up until this release planning session, builds did not receive the valuable resources they needed.
  
• Pre-Planning Preparation – We went into the release planning session with each team’s PM, PO and SM having a good understanding of what needed to happen. The teams had a backlog and they even spent time with the entire iteration team validating their thoughts before the planning session started! In addition, each iteration team brought along at least one of their architects to further improve our results.
  
• No Major Surprises – We completed our release planning session without encountering any major surprises. No last minute major misses that could be used by the Agile Naysayers as reasons for why Agile can’t work. The old “Agile doesn’t plan long term” myth.”
  

This is a pretty insightful post, so I recommend reading it in its full context.

Enterprise Agility–The Big Picture (3): Role of the Product Owner

Note: In the post Enterprise Agility: The Big Picture, we introduced an overview graphic intended to capture the essence of enterprise agility in a single slide. In prior posts, we’ve discussed teams and iterations. To see the full series, select the “Big Picture” blog post category.

In this post, we’ll describe the Agile Product Owner [callout (3)].

Big Picture 3 - Product Owner

One of the nice things about blogs is that the usage statistics show where peoples interests (at least those who read your blog) lie. In this blog, there is typically a high degree of interest in the Product Owner and Product Manager posts and they appear as the most frequent search terms as well. So as I endeavor to put my summary thoughts down here, I promise myself to return to this topic for some future, more in-depth, posts on this critical topic. For my other current posts on this topic, check here.

For now, however, since this series is about the Big Picture and the Product Owner is just one element, we’ll have to be content with the highlights. Relative to the Big Picture, I’ll describe three facets of the product owner role here- responsibilities, attributes and organization.

Responsibilities

Manage the Iteration Backlog – The primary responsibility of the product owner is the development and maintenance of the iteration backlog. The iteration backlog is the repository for all user stories that are currently envisioned to address the user’s needs. The Product Owner owns this key artifact. That means they can add, delete or update as necessary to control the scope and content of the implementation. At the end of each iteration, the product owner reprioritizes the backlog for the next iteration and this continues for as long as the product (or application) serves its marketplace. (The iteration backlog itself is the topic of the next post).

Own the Iteration – In the Big Picture graphic, the Product Owner is located next to the iteration backlog. This illustrates the “near total immersion” of the product owner in the day-by-day, week-by-week delivery of core functionality via iterations. In addition, the product owner is typically the customer and Q/A proxy at iteration demos and is empowered to accept individual stories and the iteration as whole. So the product owner owns the iteration.

Collaborate with Product Management– As implied in the Big Picture, the strategic direction for the product line is typically driven by the enterprises Product Managers, who are aligned most closely with the customers and the market. So in addition to the team-based responsibilities, the Product Owner engages in a continuous collaboration with the Product Managers, determining priorities and tradeoffs. In many organizations this creates a “bold dotted line” of management reporting and this larger team steers the enterprise to its destination. In other organizations, the Product Owners are the Product Managers or report into the same organization. (see below).

Attributes of the Product Owner

Ability to communicate– the product owner is the glue chip that seamlessly binds the product management function and the development team. Doing so requires the best possible communication skills as the product owner translates, via the user story metaphor, user and business objectives into the technical terms the teams understand. Moreover, the product owner will almost certainly be involved in customer demos, sales support and the like, so a bit of polish and some customer friendly skills go a long way to enhanced performance in this role.

Technical Skills – effective scope triage requires the constant evaluation of technical, functional, performance and user-value tradeoffs. This in turn, requires a reasonable degree of technical competence as the language of discussion and the foundation for effective decision making is driven by the technology of the implementation. In addition, continuous refactoring is integral to agile, and since the product owner owns the backlog, prioritizing refactors vs. value stories is a critical skill that requires a technical foundation.

Trust – in many enterprises, the product owner is a new role, as the company discovers that the high velocity of elaboration and the need to be bound to the team for story and iteration acceptance outstrips the product management organization’s ability to keep up. Given the nature of this pivotal role, most agile enterprises end up promoting people from the development teams to fulfill this role. Since the primary responsibility for prioritizing and managing the backlog (i.e. what will and will not be done) falls to this role, the most essential quality is trust. The teams have to trust the product owner to make the hard calls on scope triage and to defend their interest in quality and functionality; the product managers have to trust the product owner to faithfully represent their feature priorities to the team.

Organization

This one is easy. In order the prevent the organizational schism (read: dysfunction) that can be so easily created in translating market requirements into technical requirements and, even more importantly, to assure that the accountability for timely delivery rests with the team, the primary “binding” of the product owner in the organization is to the development team. This means that the APO should be 1) collocated with the team and 2) held to the same performance measures as the team. In most enterprises that mean that the product owners also 3) reports into the same line management as the team. However, in my experience 1) and 2) are mandatory, 3) is only convenient. In any case, I intend to leave no ambiguity to my opinion that the agile product owner is part of the development team.

In the next post, we’ll discuss the Iteration Backlog in more detail.

Enterprise Agility–The Big Picture (2): Iterations

Note: In the post Enterprise Agility-The Big Picture, we introduced an overview graphic intended to capture the essence of enterprise agility and the Agile Release Train in a single slide. In this post, we’ll describe the iterations [callout (#2)] that are the fundamental building block of enterprise agility. For elaboration on earlier numbered callouts, select the “Big Picture” blog post category.

In this post, we’ll discuss the Iteration, the heartbeat of agility. While it isn’t appropriate here to elaborate in detail on how teams achieve the iteration, Chapter 11 of Scaling Software Agility, Mastering the Iteration
is posted right here on the blog for easy access to that discussion. But for now, we’ll just focus on the Big Picture, repeated with updated callouts below.

Big Picture 2-Iterations

Iterations and the Agile Release Train

The graphic illustrates an example of the Agile Release Train concept that I described in Chapter 18 of SSA. To summarize, the Agile Teams teams are organized around a standard iteration length, and share start and stop boundaries so that the code maturity is comparable at each iteration boundary system integration point. Of course, agilists always recommend continuous daily integration (Chapter 14 of SSA) and this graphic doesn’t mean to imply that isn’t happening. However, it does gives credence to the fact that it may be quite difficult to achieve continuous integration, (including full automated regression testing) across all teams (at least early on in the agile adoption lifecycle) and at least it is forced at the iteration boundaries in this model.

Iteration Length

Iterations have a standard pattern of plan-implement stories-review (demo and retrospective). In the Big Picture,  , the iteration lengths for all teams are the same as that is the simplest organizational and management model (Chapter 18 of SSA). As for length, most have converged on two weeks as a standard length. The logic is as follows: Realistically, there are only four natural, calendar-based choices – 1 week, 2 weeks, 4 weeks, and a calendar month. Due primarily to iteration overhead (planning, demo, retrospective, etc.) and the occasional holiday, one week is too short for many teams (though I am aware of large scale systems being built in one week iterations and that length is also quite typical for XP teams). Four weeks or one month is simply too long, as there are not enough times to fail before some major release commitment. In other words, the lessons learned are too far apart to make the necessary mid-course corrections necessary to land the release on time. That leaves just one choice: two weeks.

However, you’ll also note that the graphic doesn’t call out an iteration length. I am aware of different teams using this basic Agile Release Train model that apply iterations of one, two and four weeks, respectively, so if you don’t like the two week choice, after you and I are done arguing, you can still apply the model.

Number of Iterations per Release

You may also note that we’ve illustrated four development iterations (indicated by a full iteration backlog) and one hardening iteration (indicated by an empty backlog). This is somewhat arbitrary as well and there is no fixed rule for how many times a team iterates prior to an internal or external release boundary. On the other hand, if you assume standard two week iterations and the multi-iteration pattern shown here, you’ll arrive at an internal release boundary of about ten weeks. In practice, many teams apply this model with five development iterations and one hardening per release, creating an internal release cadence of a fully shippable increment about every 90 days, which is perhaps a bit more natural and corresponds to the likely maximum external release frequency for some enterprises.

In any case, the length and number of iterations per release are up to each enterprise and in point of fact, the differences are not all that critical, so the Big Picture model is largely immune to the team’s actual selection.

We’ll discuss the Role of the Product Owner and the iteration backlog in the next posts.

Enterprise Agility–The Big Picture (1): Agile Teams

Note: In the post Enterprise Agility-The Big Picture, we introduced an overview graphic intended to capture the essence of enterprise agility and the Agile Release Train in a single slide. We repeat it here with an annotation callout.

Big Picture (1): Teams

In this post, we’ll describe the Agile Teams [callout (#1)] that are responsible for creating all that high quality software.

You can see from the graphic that we’ve indicated some number (<=10) of agile teams that collaborate on building the bigger system.  Since it’s an agile team, it has a maximum of ten or so members, and includes all the roles necessary to Define/Build/Test (see Chapter 9 of Scaling Software Agility) the software for their component or feature. The roles are Scrum/Agile master, Product Owner and the Team itself. The Team includes fully dedicated developers, test and test automation experts, tech leads, and support from component level architects, SCM/build/infrastructure support personnel, or indeed, whomever it takes such that the team is fully capable of defining, developing, testing and delivering working and tested software for their component into the system baseline.

These teams are most typically organized around a software component or a feature. Most enterprises will have a mix of both, some component teams focused on infrastructure and architectural components, and some feature teams focused on larger scale, current, value-delivery initiatives. Agile Teams are self-organizing and tend to reorganize themselves continuously based on the work in the backlog so the teams are more dynamic than static. (That’s one of the reasons the open space environment works more effectively).

We’ve indicated on this slide that there typically a maximum of 5-10 or so such teams cooperating on building a larger system or subsystem (the system, application or product domain). However, this isn’t a hard or fast rule, but experience has shown that even for VERY large systems, the logical partitions defined by system or product family architecture tend to cause “pods” of developers to be organized around the various domains. This implies that perhaps 50-100 people must intensely collaborate on building their “next bigger thing” in the hierarchy. And as we’ll discover later, this is also about the maximum size for face-to-face, collaborative Release Planning at the system or subsystem domain level. So while this construct and the numbers are somewhat arbitrary, it does track real world examples and is a fairly natural way to organize for large scale agile development.

Of course, even that’s an oversimplification for a really large system, as there are likely to be a number of such larger domains, each contributing to the portfolio (application suite, larger system, or whatever). From a purely mathematical scalability standpoint, ten such domains, each consisting of ten component teams each could be the agile organization containers for up to 1,000 practitioners. In practice, this is about the largest, highly cooperative, development project I’ve personally seen. While there are certainly much larger numbers of practitioners in some really large software enterprises, it is not typically the case that all, or even most, must contribute cooperating code. Rather, it is far more likely that the teams are organized around somewhat stand-alone product lines or  applications with relatively thin interfaces.

So this big picture model appears to serve us well in describing an organizational and process model that could work for teams of many hundreds of software practitioners, who must collaborate frequently and intensely to deliver a large scale system in an agile manner. You’ll also notice a Release Management Team a little higher in the graphic. This is covered in Chapter 18 of SSA, and it will also be discussed in a post later in this series.

In the next post (Big Picture #2, later this week), we’ll discuss the heartbeat of the agile enterprise, Iterations.

Enterprise Agility–The Big Picture

(Note: The Big Picture graphic below has been revised during this series of posts. I try to keep the latest version here so you may see some differences if you follow the series from front to back).

I spend a fair amount of time working with executives trying to give them the “big picture” of what their software organization, requirements, process and delivery model would look like after they adopt enterprise agility and implement the Agile Release Train (Chapter 18 of Scaling Software Agility). My thinking is that if I can paint the “after” picture in their minds without spending all day at it, then it will be easier to for them to understand the types of work they need to do to achieve it.

For some time, I’ve struggled to come up with a graphic that communicates the high-level model of enterprise agility as simply as the waterfall graphic communicated its stage-sequenced model. I’ve made a number of attempts at it, including a few in the book, but I don’t feel that I ever really captured the essence in a single graphic. (Though I’m a writer at times, I’m a far more visual learner).

Recently,  I worked with Matthew Balchin and Richard Collins at Symbian Software Ltd. , ( and later, with Juha-Markus Aalto of Nokia, Nov-Jan 2009) to build a “single slide” big picture that we could use in a number of presentation contexts. Here is a modified result.

The Big Picture (updated Mar 2009)

I tested an earlier version of this graphic recently with a group of software executives contemplating an agile transformation. We spent a solid hour just discussing the graphic. It seemed pretty effective in helping them understand the intended result, with far fewer words and slides.

Since it seemed to work but it’s not quite self explanatory, I will annotate it extensively in some upcoming posts in the hope that it might help others struggling with the same communication challenge.

Agile Architecture – The Whitepaper Now On Line

For those of you following the agile architecture series, you may know that I’ve been working with Ryan Martens, Rally’s CTO and founder, and Mauricio Zamora, from CSG Systems, to put our thoughts on agile, intentional architecture into whitepaper form. Rally has given a big assist to the production and publishing process and I’m happy to announce that the final whitepaper, Principles of Agile Architecture: Intentional Architecture in Enterprise-Class Systems: is now on line at the Rally Knowledge Portal. There’s lots of other enterprise-level content there as well so it’s well worth the simple registration.

For context, here’s an extract:

“In this paper we’ll discuss the role of “Intentional Architecture” in the development of enterprise-class systems built using agile methods and techniques. In order to gain a better understanding of this important practice, we’ll describe:

• The context and challenges of large-scale system architecture in Agile development

• The need for Intentional Architecture to buttress emerging architecture

• How the traditional systems architect can contribute to Agile teams”

The paper goes on ……

” There are a number of governing principles teams can apply to the challenge of architecture development in large-scale systems. These governing principles are:

Principle #1 The teams that code the system design the system.

Principle #2 Build the simplest architecture that can possibly work.

Principle #3 When in doubt, code it out.

Principle #4 They build it, they test it.

Principle #5 The bigger the system, the longer the runway.

Principle #6 System architecture is a role collaboration.

Principle #7 There is no monopoly on innovation.”

If you are building big systems, I think you’ll find it interesting reading.

Agile Executive – Final, Six Part BMC Executive Interview Now On Line

Israel Gat (VP, BMC Software) has just completed a six part interview series on the BMC agile transformation experience at the Agile Thinkers blog that I’ve been highlighting in some recent posts. This series is an open, honest and direct perspective from an experienced senior executive faced with all the challenges of implementing agility at scale in a formerly waterfall organization. It’s a quick read and yet it is filled with pithy and memorable takeaways on the executive perspective and “cultural gestalt” of enterprise agility. I strongly recommend this series to any organization headed down the agile path.