Upcoming September 10, 2008 Webinar – Principles of Agile Architecture

For those interested in the Agile Architecture Series, I’ll be presenting a webinar on this topic on September 10, 2008 at 1:00PM Eastern time. The webinar Principles of Agile Architecture: Intentional Architecture in Enterprise Class Systems is described below:

“As Agile development practices cross the chasm to the enterprise, many interesting debates continue about the role of systems architecture. As the continuous refactoring of emerging-only architectures becomes less practical as the system size and complexity grows, there is a natural desire to minimize unnecessary refactoring as well as to build systems from components that work in a like manner. This defines a role for “Intentional Architecture” in agile, enterprise-class systems development. In this Web seminar, learn about seven core principles that can be applied to architecting these large scale systems without sacrificing the principles or benefits of true software agility.”

This is part of a series sponsored by Rally Software, in collaboration with Sticky Minds and Better Software Magazine, designed specifically for the Enterprise Agilist. This webinar series is designed around specific topics of interest to the larger enterprise adopting agile methods. The description from the website is below:

“Join leading Agile experts, coaches, and authors—including Jean Tabaka, Dean Leffingwell, Stacia Broderick and Ryan Martens—as they paint a picture of successful Enterprise Agile adoption from a team’s first project to enterprise-wide transformation. During this Web seminar series, you will learn proven organizational transition approaches and best practices around Agile project and product management, Agile architecture and quality management. Development, management, and product teams will gain both an understanding of what it means to transition to Agile as well as specific best practices for their role in the organization.”

The series is now available for registration at: http://www.bulldogsolutions.net/SQE/SQE09242008/calendar.aspx?bdls=15887.

And it is FREE!

Advertisements

Principles of Agile Architecture – A Whitepaper

The whitepaper, Principles of Agile Architecture: Intentional Architecture in Enterprise-Class Systems, has now been posted on the Resource page. This whitepaper grew out of a series of blog posts wherein we were trying to define some governing agile principles that teams could use  to guide their agile, system-level architectural practices. My coauthors are Ryan Martens, Rally’s CTO and founder, and Mauricio Zamora, Executive Director from CSG Systems, both of whom are experienced enterprise agilists, architects and thought leaders in their own right.

Agile Architecture – The Whitepaper Now On Line

For those of you following the agile architecture series, you may know that I’ve been working with Ryan Martens, Rally’s CTO and founder, and Mauricio Zamora, from CSG Systems, to put our thoughts on agile, intentional architecture into whitepaper form. Rally has given a big assist to the production and publishing process and I’m happy to announce that the final whitepaper, Principles of Agile Architecture: Intentional Architecture in Enterprise-Class Systems: is now on line at the Rally Knowledge Portal. There’s lots of other enterprise-level content there as well so it’s well worth the simple registration.

For context, here’s an extract:

“In this paper we’ll discuss the role of “Intentional Architecture” in the development of enterprise-class systems built using agile methods and techniques. In order to gain a better understanding of this important practice, we’ll describe:

• The context and challenges of large-scale system architecture in Agile development

• The need for Intentional Architecture to buttress emerging architecture

• How the traditional systems architect can contribute to Agile teams”

The paper goes on ……

” There are a number of governing principles teams can apply to the challenge of architecture development in large-scale systems. These governing principles are:

Principle #1 The teams that code the system design the system.

Principle #2 Build the simplest architecture that can possibly work.

Principle #3 When in doubt, code it out.

Principle #4 They build it, they test it.

Principle #5 The bigger the system, the longer the runway.

Principle #6 System architecture is a role collaboration.

Principle #7 There is no monopoly on innovation.”

If you are building big systems, I think you’ll find it interesting reading.

Recap: SEVEN Principles of Agile Architecture

Those following the agile architecture series will probably note that the principles and labels have morphed over time. I’ve been collaborating with Ryan Martens, Rally’s founder and CTO, and Mauricio Zamora, Executive Director at CSG Systems, in writing a whitepaper to be published on this topic. In the process, we’ve also had some comments from Grady Booch and Philippe Kruchten and the content has been evolving as a result. The article is nearly finished and I’ll post it here just as soon as it’s available.

In the meantime, here’s the (hopefully final) set of principles the article espouses:

Principle #1 ─ The teams that code the system design the system.

Principle #2 ─ Build the simplest architecture that can possibly work.

Principle #3 ─ When in doubt, code it out.

Principle #4 ─ They build it, they test it.

Principle #5 ─ The bigger the system, the longer the runway.

Principle #6 ─ System architecture is a role collaboration.

Principle #7 ─ There is no monopoly on innovation.

Agile Architecture Principle #7 – There is no monopoly on innovation

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six (ok, now seven) principles that we can apply to help us reason about the challenge of architecting systems of scale in an agile enterprise.

“Inertia is the residue of past innovation efforts.
Left unmanaged it consumes the resources required to fund next-generation innovation.”
—Geoffrey Moore

In an earlier principle, the bigger the system, the longer the runway, we described the need for teams to have sufficient architectural runway to be able to reliably “land” near term iterations and releases. In a sense, this is the upside of inertia enough is known from prior experience and development to allow the team to provide reliable and consistent value delivery. Indeed, with agile, we have strong technical and project management mechanisms (and some architectural runway) in place to help us stay focused to this sole purpose.

In other words, agile practices provide a disciplined, production-like ability to reliably meet commitments and to rapidly evolve a system to meet existing customer requirements. We treasure our new-found reliability as a very good thing indeed.

But there is a downside as well. If we are not careful, the “tyranny of the urgent” will cause us to keep our heads down and focused solely on near term deliverables. So where does the innovation come from in such a model? Simply, mature agilists put processes in place to assure that innovation is not solely incremental and near term.

Some of our innovation practice comes from empowering the role of the system architects as part of our advanced guard, to be constantly exploring new technologies, patterns and techniques that can help us innovate. But ours is a team-centric model, so we clearly will not rely on architects as the sole source of such innovation. In fact, the team-centric model can foster innovation at an even greater rate than that generally seen in traditional software organizations. This is because true innovators innovate at all stages of their career (entry level, junior and senior) and the team-centric model enables these folks to flourish and contribute beyond what their level of experience may otherwise imply.

One way to foster iteration at the team level is by judicious backlog management that includes spikes for refactoring, design and exploration of new ideas. This can work quite well, but even more explicit models have been put into use. For example, at Rally Software Development, where they have been building their SaaS Agile Product management solution in a highly agile fashion for four years (with rarely a missed or delayed release commitment), they have evolved to an advanced development cadence as illustrated in the figure below.

An iteration cadence that fosters innovation

Figure – An iteration and release cadence with one innovation “hackthon” per release

This figure illustrates a standard release cycle, where

i” is a standard development iteration, providing new functionality for an upcoming release

h” is a one week hardening iteration, to eliminate technical debt and assure quality requirements meet the release to manufacturing criteria

k” is a “hackathon”.

The hackathon is designed to foster innovation at the team level. The rules of the hackathon are simple: any team member can explore any technology area in any way they want, so long as there is some correlation to the company’s mission. This gives the team some mental down timeto reflect, think and experimentoutside of the everyday rigor and pressures of the iteration and release cycle.

With a model such as this, innovation is expected and programmatic, and there can be no ambiguity to the point to this (and our last!) Principle # 7 There is no monopoly on innovation.

Agile Architecture: Role of the System Architect in the Agile Enterprise

In the last post (What is Intentional Architecture?) we provided the motivation and a description of the role of Intentional Architecture in agile, enterprise-class application development. I suspect that most enterprise agilists would agree that there are substantive benefits when Intentional Architecture is effectively applied (so long as we don’t slow down development or use the need for up-front architecture as a capitulation to the waterfall design phases of the past!).

Historically, Intentional Architecture was a primary function of the system architect. But the most common agile methods don’t define (or necessarily even support) such a role, so a serious agile paradigm mismatch is likely to occur. These system architects (who, by the way, often hold director-level, VP, or CTO positions!) likely have decades of experience in the technology and the domain and have strong views about the intended architecture. However, the bulk of their implementation experience has happened outside an agile context, so they may not understand the practice of building refactorable code. Indeed, refactoring (mostly good) may all look like unnecessary rework (mostly bad) to them so they may not be initially supportive of our model, which depends on refactoring as a primary technique. They may also be concerned about the potential architectural entropy of all those newly empowered and energized agile teams. In addition, they may have equally strong opinions about the software development practices teams employ. If we fail to bring these key stakeholders on board to our agile development paradigm, they could kill the initiative as quickly as a routine sixty-hour work week. In some cases, we have witnessed the Battle for agile architecture: the agile teams vs. the system architects, and there can be no winner in that battle.

Fortunately, enterprise agility is not a zero sum game and there is room for all who can contribute to the best possible technical solutions. As we are continuously raising the bar to build systems of increasing complexity the likes of which have never been built before, it only makes sense to leverage the skills of all who have the experience to match the challenges the teams face! So include them we will and we described our support for this role in Principle #6 – System Architecture is a Role Collaboration.

Agile Architecture Principle #6 – System architecture is a role collaboration

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that we can apply to help us reason about the challenge of architecting systems of scale in an agile enterprise.

In prior posts, we’ve discussed the first five principles (see thumbnail at left). In this post, we’ll discuss #6 – System architecture is a role collaboration.

In this series of posts to date, we’ve focused on the central role the teams play in designing and architecting systems of scale. For example, in Principle #1, we proposed that the teams that code the system design the system, so there could be no ambiguity about who had the responsibility for architecting the system – the team that is accountable for implementing it. If we stopped here, it wouldn’t be obvious that there is a role at all for those senior, domain experienced, system architects that play such a key role in many larger enterprises. This topic gets particularly entertaining, when as an agile proponent, I meet these people and hear their strong views about imposing architectural governance on the natural entropy of all those newly empowered (and sometimes relatively inexperienced) agile teams.

Most typically, these folks have deep technical and domain experience as well as the seniority, competence and authority to back up their opinions. Indeed they have a lot to add to the vision and implementation and it would be downright foolish of us to leave them out of the mix of what “makes an agile team a team”. Moreover, they have a strong influence on the process models the teams will adopt and if they are not supportive of an agile transformation, one may or may not take place!

Fortunately, enterprise agility is not a zero sum game and there is room for all who can contribute to the best possible technical solutions. After all, we are building systems of enormous complexity (even when we keep them as simple as possible), why would we not leverage the skills of those external team members who have the experience to match the challenges the teams face?

So leveraging their expertise is to the benefit of the team and the enterprise, but how do we incorporate them in our team-centric agile model? I’m reminded of the parallels between organizations and the systems architecture that organizations create. Conway’s law states that: “Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations” While this law may seem to imply a negative, that needn’t be the case. For example, in object oriented systems development, we design systems of collaborating objects, structured around well defined interfaces, which work together to create an output that is greater than the sum of its parts. The same approach can be applied as to how we work together to develop systems architecture (see Figure 1).

Systems architecture is a role collaboration

In this figure, you can see that system architecture can be seen as role collaboration between the system architect and tech leads of the component teams that write the code. These system-level teammates work together with the component teams to decide what the architecture needs to be. And when in doubt, they code it out (principle #3) though a series of design spikes inside iteration or release boundaries. With the support of the product owner, these spikes are mixed in the backlog based on whatever priorities the team feels is appropriate, (this is one of the primary reasons we like to see product owners with a high degree of technical experience) and over time, a consensus emerges as to how to build the system they are about to deploy.

Ok, that’s pretty theoretical, perhaps an example will help. In the last post, I noted a Single-Sign-On (SSO) architectural consideration that came from a real world experience. In this application, the component teams needed to implement SSO from their system to an external web service. The component teams had little experience in web SSO, having had no prior need (and they weren’t following the market trends for a need they hadn’t presently have!) The system architect had broader experience, and perhaps more importantly, had been researching SSO options prior to the release planning session where the discussion was initiated. There were certainly many options available to the team – from lightweight open source protocols, to higher security, SAML standards-based implementations, to a variety of vendor proprietary solutions. The system architect had formed an opionion based on prior investigation. Even with that input, a lively discussion ensued.

So who is to pick – the team closest to the implementation or the architect with more experience but who is further from the implementation? The answer must come via collaboration and agreement between the parties, supported as necessary by evaluation matrices (footprint, recurring cost, level of security, etc.) and where necessary, a series of design spikes to test each proposed solution. After all, if a design spike or two couldn’t prove the feasibility of one choice or another, then it’s unlikely that the solution chosen is likely to be the simplest architecture that could possibly work (Principle #2).

In summary, it is only by working together and by maximizing the contribution of all team players that we can hope to beat the competition by mastering systems of the complexity that we are crafting today. To do so, let’s engage the system architects in our solution via Principle # 6 – system architecture is a role collaboration.

Agile Architecture Principle #5 – The bigger the system, the longer the runway

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

In prior posts, we’ve discussed the first four principles. In this post, we’ll discuss #5 – The bigger the airship, the longer the runway.We started this thread sometime back with a discussion of the rationale behind the build out of some amount of architectural “runway’ for systems of scale. To summarize, we posited that investment in some amount of intentional (or intentionally emergent) architecture is warranted based on the fact that continuous refactoring of large scale systems becomes problematic for three reasons.

  • Avoiding larger scale, and unnecessary rework. Even apparently minor system-level refactors can cause substantive rework for large numbers of teams, some of whom would otherwise NOT have to refactor their component or module. It is one thing for a team to refactor their code based on lessons they have learned, it’s another to require multiple teams to refactor code they would otherwise NOT have had to refactor based on other teams lessons learned, especially when the teams backlog is already overloaded with value delivery priorities.
  • Managing the risk of potential Impact on deployed systems/users. Since even the best possible BVT (Build Verification Tests) systems are imperfect, we worry about unnecessarily disrupting the install base of users (thousand and tens of thousands for enterprise architectures, even more for successful consumer systems). In other words, the threat of a regressive bug in a deployed system is always there, the cost, risk and impact of which increases with scale. In this case, minimizing unnecessary refactoring is a primary risk mitigation technique.
  • Improve usability, extensibility, performance and maintenance. Some common, imposed architectural constructs can ease usability, extensibility, performance and maintenance. For example imposing a common presentation design and implementation can substantially improve usability and presentation layer maintenance and modification, resulting in higher end user satisfaction (and revenue!) In another example, imposing a common single-sign-on mechanism across a suite of products would simplify development implementation and maintenance, even though there was NO difference to the end user.

There’s a further reason that warrants discussion as well. At the release level (internal or external), value delivery focuses on delivering the features our customers need. Being able to deliver planned functionality somewhat predictably in a near term (60+days) release is a hallmark of mature agile teams. That ability, in turn, is how the agile enterprise is able to communicate expectations to customers, who often have many business dependencies on the release of new software. Since building and refactoring infrastructure for large scale systems typically takes longer than a single short release cycle, it is necessary that most features for a particular release can be built on infrastructure that is already in the system. Otherwise, the team won’t be able to reliably “land” the release on schedule. Therefore, you need enough runway to be able to build in features for at least a release or two, while larger scale refactorings may well be underway.

Further discussion of how agile teams go about building and extending runway can be found in Chapter 16 of SSA – Intentional Architecture, and is outside the scope of this post, but for now, we’ve made our case for Agile Architecture Principle #5The bigger the airplane, the longer the runway.

Agile Architecture Principle #4 – They build it, they test it

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

Posts Principle #1 – The team that codes the system designs the system, Principle #2 – Build the simplest architecture that can possibly work and Principle #3 – When in doubt, code it out described the first three principles. In this post, we’ll discuss the Principle #4 They build it, they test it.

Agile is renowned for its emphasis on testing (“If testing is good, everybody will test all the time” – Kent Beck, 2000) and there can be no doubt that agile forces testing earlier into the lifecycle than any of our prior methods. This is a key benefit of agile in general and is one of the prime reasons why quality is materially higher in agile, and is accomplished without sacrificing overall developer productivity. Indeed, testing is so important in agile that I devoted an entire chapter to it in SSA, Chapter 12 —Concurrent Testing. “But this is a thread about agile enterprise architecture”, you might ask, “so why the lecture on testing”? Fair question.

My friend Philippe Kruchten once described architecture as ” take away everything in the system you don’t need to describe how it works, and what you have left is it’s architecture”. With this simple definition, testing architecture comes down to “testing how the system works without the details”, that means that testing architecture is testing the system’s ability to meet its functional (in the large scale), operational, performance and reliability requirements.

Clearly, this testing represents complexity at its greatest and who has the skills to test systems of the complexity that we are building today? Answer – the team that codes the system must determine how best to test the system. Moreover, with the enhanced power and complexity of today’s automation frameworks, the developers are likely to be directly involved in applying testing automation to their system. And when such automations isn’t available, it is necessary for the development teams themselves to develop, test and maintain the system tests and testing frameworks to test its ability to meet its architectural and functional requirements. Such a responsibility cannot be left to any other testing resources or outsourced functions or the team cannot then be held accountable for delivering a functional system within the iteration or release timebox.

So agile architecture Principle #4 is: They build it, they test it, because in agile, All Code is Tested Code.

Agile Architecture Principle #3 – When in doubt, code it out

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

Two prior posts: Principle #1 – The team that codes the system designs the system, and Principle #2 – Build the simplest architecture that can possibly work described the first two principles. In this post, we’ll discuss Principle #3 – When in doubt, code it out.

I suspect there is a reason why most of us techies did not pick philosophy, political science, social studies, or any of the other softer careers. Our scientific orientation and data-driven analytical tendencies allow us to move forward comfortably with important decisions and not lose a lot of sleep over varying expressed opinions. This is a skill that should serve us well in the process of picking a design alternative or a high-impact infrastructure implementation choice. Even then, we occasionally find ourselves mired in technical debate. In agile, with its highly iterative, experience and code-based emphasis, we can often simply depend on our coding skills to move us efficiently through the decision-making process.

This Principle #3 – When in doubt, code it out, reminds us that whenever we have to make a tough choice, we can always turn to a rapid evaluation in code. Our fast one or two week iterations give us a quick project cadence and the demos at the end of the iteration provide objective evidence. Inherent visibility of the agile model also allows all affected stakeholders to see the real-time, in process reasoning and experimental results. In addition, Principle #1‒Build the Simplest Architecture that Can Possibly Work, reminds us that if a design alternative can’t be coded and evaluated within a few iterations, it probably isn’t the simplest thing! In practice, rarely have I seen a case where a decision wasn’t fairly obvious after a few short design spikes.

And for agile at scale, I’m again reminded of the lessons learned from the Amazon Architecture: (highlighted in the post Building Scalable, Robust Architecture with Agile: Lessons Learned at Amazon)

” Use measurement and objective debate to separate the good from the bad. I’ve been to several presentations by ex-Amazoners and this is the aspect of Amazon that strikes me as uniquely different and interesting from other companies. Their deep seated ethic is to expose real customers to a choice and see which one works best and to make decisions based on those tests. Avinash Kaushik calls this getting rid of the influence of the HiPPO’s, the highest paid people in the room. This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do, code it up, let people use it, and see which alternative gives you the results you want.”