Agile Architecture Principle #5 – The bigger the system, the longer the runway

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

In prior posts, we’ve discussed the first four principles. In this post, we’ll discuss #5 – The bigger the airship, the longer the runway.We started this thread sometime back with a discussion of the rationale behind the build out of some amount of architectural “runway’ for systems of scale. To summarize, we posited that investment in some amount of intentional (or intentionally emergent) architecture is warranted based on the fact that continuous refactoring of large scale systems becomes problematic for three reasons.

• Avoiding larger scale, and unnecessary rework. Even apparently minor system-level refactors can cause substantive rework for large numbers of teams, some of whom would otherwise NOT have to refactor their component or module. It is one thing for a team to refactor their code based on lessons they have learned, it’s another to require multiple teams to refactor code they would otherwise NOT have had to refactor based on other teams lessons learned, especially when the teams backlog is already overloaded with value delivery priorities.
• Managing the risk of potential Impact on deployed systems/users. Since even the best possible BVT (Build Verification Tests) systems are imperfect, we worry about unnecessarily disrupting the install base of users (thousand and tens of thousands for enterprise architectures, even more for successful consumer systems). In other words, the threat of a regressive bug in a deployed system is always there, the cost, risk and impact of which increases with scale. In this case, minimizing unnecessary refactoring is a primary risk mitigation technique.

• Improve usability, extensibility, performance and maintenance. Some common, imposed architectural constructs can ease usability, extensibility, performance and maintenance. For example imposing a common presentation design and implementation can substantially improve usability and presentation layer maintenance and modification, resulting in higher end user satisfaction (and revenue!) In another example, imposing a common single-sign-on mechanism across a suite of products would simplify development implementation and maintenance, even though there was NO difference to the end user.

There’s a further reason that warrants discussion as well. At the release level (internal or external), value delivery focuses on delivering the features our customers need. Being able to deliver planned functionality somewhat predictably in a near term (60+days) release is a hallmark of mature agile teams. That ability, in turn, is how the agile enterprise is able to communicate expectations to customers, who often have many business dependencies on the release of new software. Since building and refactoring infrastructure for large scale systems typically takes longer than a single short release cycle, it is necessary that most features for a particular release can be built on infrastructure that is already in the system. Otherwise, the team won’t be able to reliably “land” the release on schedule. Therefore, you need enough runway to be able to build in features for at least a release or two, while larger scale refactorings may well be underway.

Further discussion of how agile teams go about building and extending runway can be found in Chapter 16 of SSA – Intentional Architecture, and is outside the scope of this post, but for now, we’ve made our case for Agile Architecture Principle #5 – The bigger the airplane, the longer the runway.

Agile Architecture Principle #4 – They build it, they test it

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

Posts Principle #1 – The team that codes the system designs the system, Principle #2 – Build the simplest architecture that can possibly work and Principle #3 – When in doubt, code it out described the first three principles. In this post, we’ll discuss the Principle #4 ‒ They build it, they test it.

Agile is renowned for its emphasis on testing (“If testing is good, everybody will test all the time” – Kent Beck, 2000) and there can be no doubt that agile forces testing earlier into the lifecycle than any of our prior methods. This is a key benefit of agile in general and is one of the prime reasons why quality is materially higher in agile, and is accomplished without sacrificing overall developer productivity. Indeed, testing is so important in agile that I devoted an entire chapter to it in SSA, Chapter 12 —Concurrent Testing. “But this is a thread about agile enterprise architecture”, you might ask, “so why the lecture on testing”? Fair question.

My friend Philippe Kruchten once described architecture as ” take away everything in the system you don’t need to describe how it works, and what you have left is it’s architecture”. With this simple definition, testing architecture comes down to “testing how the system works without the details”, that means that testing architecture is testing the system’s ability to meet its functional (in the large scale), operational, performance and reliability requirements.

Clearly, this testing represents complexity at its greatest and who has the skills to test systems of the complexity that we are building today? Answer – the team that codes the system must determine how best to test the system. Moreover, with the enhanced power and complexity of today’s automation frameworks, the developers are likely to be directly involved in applying testing automation to their system. And when such automations isn’t available, it is necessary for the development teams themselves to develop, test and maintain the system tests and testing frameworks to test its ability to meet its architectural and functional requirements. Such a responsibility cannot be left to any other testing resources or outsourced functions or the team cannot then be held accountable for delivering a functional system within the iteration or release timebox.

So agile architecture Principle #4 is: They build it, they test it, because in agile, All Code is Tested Code.

Agile Architecture Principle #3 – When in doubt, code it out

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

Two prior posts: Principle #1 – The team that codes the system designs the system, and Principle #2 – Build the simplest architecture that can possibly work described the first two principles. In this post, we’ll discuss Principle #3 – When in doubt, code it out.

I suspect there is a reason why most of us techies did not pick philosophy, political science, social studies, or any of the other softer careers. Our scientific orientation and data-driven analytical tendencies allow us to move forward comfortably with important decisions and not lose a lot of sleep over varying expressed opinions. This is a skill that should serve us well in the process of picking a design alternative or a high-impact infrastructure implementation choice. Even then, we occasionally find ourselves mired in technical debate. In agile, with its highly iterative, experience and code-based emphasis, we can often simply depend on our coding skills to move us efficiently through the decision-making process.

This Principle #3 – When in doubt, code it out, reminds us that whenever we have to make a tough choice, we can always turn to a rapid evaluation in code. Our fast one or two week iterations give us a quick project cadence and the demos at the end of the iteration provide objective evidence. Inherent visibility of the agile model also allows all affected stakeholders to see the real-time, in process reasoning and experimental results. In addition, Principle #1‒Build the Simplest Architecture that Can Possibly Work, reminds us that if a design alternative can’t be coded and evaluated within a few iterations, it probably isn’t the simplest thing! In practice, rarely have I seen a case where a decision wasn’t fairly obvious after a few short design spikes.

And for agile at scale, I’m again reminded of the lessons learned from the Amazon Architecture: (highlighted in the post Building Scalable, Robust Architecture with Agile: Lessons Learned at Amazon)

” Use measurement and objective debate to separate the good from the bad. I’ve been to several presentations by ex-Amazoners and this is the aspect of Amazon that strikes me as uniquely different and interesting from other companies. Their deep seated ethic is to expose real customers to a choice and see which one works best and to make decisions based on those tests. Avinash Kaushik calls this getting rid of the influence of the HiPPO’s, the highest paid people in the room. This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do, code it up, let people use it, and see which alternative gives you the results you want.”

Agile Architecture Principle #2 – Build the simplest architecture that can possibly work

Note: this is one in a series of posts under the category of “agile architecture”.

In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise. This post discusses Principle #2 – Build the simplest architecture that can possibly work.

This one will certainly come as no surprise to even an agilist newbie, after all, agile is famous for its focus on simplicity:

“What is the simplest thing that can possibly work?” – Ward Cunningham

“If simplicity is good, we’ll leave the system with the simplest design that supports its current functionality.” – Kent Beck

YAGNI – You Ain’t Gonna Need It – an XP mantra

Of course, we are talking about systems of substantial complexity here, the types of systems that are crafted by hundreds of developers working over many years, rather than the types of systems developed in smaller team environments where agile practices were developed and proven, so it is a fair question to ask as to whether simplicity remains an essential attribute as the complexity increases.

Of course, it’s obvious from the postulate that I believe this to be the case, and this is supported by my own anecdotal evidence in building large scale systems. In most every case I remember (two “simpler” decisions I remember vividly: 1) dedicate a single service to a single class of server, 2) pick an off-the-shelf data base for those elements of the system that did not require extreme scalability), the simplest decisions turned out to be the best over time. And you needn’t take my word for it. Take Amazon.com, for example, where they grew a system in an agile and organic manner to eventually handle 55 million customer accounts and where between 100-150 services are accessed to build a page. In a recent post on the website highscalability.com, (http://highscalability.com/amazon-architecture) Todd Hoff notes a primary lesson learned as described by Amazon’s CTO, Werner Vogels:

“Only way to manage as large distributed system is to keep things as simple as possible. Keep things simple by making sure there are no hidden requirements and hidden dependencies in the design. Cut technology to the minimum you need to solve the problem you have. It doesn’t help the company to create artificial and unneeded layers of complexity.”

If simplicity of architecture is good enough for Amazon.com, it’s good enough for me.

Agile Architecture Principle #1 – The team that codes the system designs the system

Note: this is one in a series of posts under the category of “agile architecture”.

In the last post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

In this post, I’ll describe the first principle of agile architecture: the team that codes the system designs the system. This principle is driven both by the manifesto itself, (ie: The best architectures, requirements, and designs emerge from self-organizing teams) as well as another of the primary philosophies of agile development, which is that the teams themselves are both empowered to achieve, and accountable for, their delivered software.

From a management perspective, in order for teams to be held accountable, it naturally follows that we must allow them to make the decisions necessary to support that accountability – otherwise they are essentially being held accountable for decisions by others and that is neither an effective or motivating model for team performance! While this seems axiomatic, the reality of our behavior in earlier practice was somewhat different as the figure below shows.

This paradigm was further compounded by our time-sequenced waterfall activities; I.e. there was typically a single, up-front planning phase intended to accommodate the inherent risk of the project, perform all the necessary design and predict sequenced task dependencies running as far out as a year into the future. This phase was decoupled in time from the lessons learned in actual implementation and, once the lessons were learned, it was too late to do anything to the schedule (other than to apologize for the fact that we couldn’t meet it.)

Further, from a technical perspective, these architectural decisions are most optimally made by the coders, tech leads, team architects, etc, as they are closest to the implementation and therefore often (but perhaps not always as we’ll discuss Principle #6 – System architecture is a collaboration) have the best data available to make such a decision.

The combination of these two factors:

1) they will likely make a more optimum decision, and

2) once made, since they are accountable for the results, they will likely work a lot harder to make their decision work in the implementation

delivers a big part of the power of agile and more than warrants our first principle: the team that codes the system designs the system.

I’ll describe the second principle: Build the simplest architecture that can possible work, in my next post.

Six Principles of Agile Architecture (Part 1)

Many interesting discussions and debates about agile architecture, whether it be emergent, intentional, intentionally emergent, (or whatever!), continue as agile marches across the chasm into the enterprise. This issue gets particularly entertaining as I encounter deeply experienced and very senior architects, systems architects, VPs, CTOs and the like who have strong views about imposing architectural plans on the natural entropy of all those newly empowered and energized agile teams. Most typically, these folks are quite competent and they have the experience and authority to back it up their knowledge. More importantly, they have deep, experience-based technical insights to add to the process and it would be downright foolish of us to leave them out of the mix of what “makes an agile team a team”.

Also, the continuous refactoring of emerging-only architectures in these enterprise-class software starships becomes less and less practical as the size of the ship grows and matures (Chapter 16 of SSA). Therefore there is an understandable and reasonable desire to minimize refactoring as well as build systems of components that work in a like manner to ease usability, extensibility, performance and maintainability. These are strong and practical motivations to apply some “architectural governance” to the software development practice.

The overly simple argument of “the architecture will emerge ─ let the component teams decide” does not always go so well in some of those offices. For the reasons I’ve indicated above, some of the concernis reasonable. Other reasons include the fear of loss of control of some of the most critical, IP in the company, i.e. that which reflects an understanding of how the system works now and how it is intended to be extended over time. Whether this fear is rational or not, (and I suspect that it often is), if not addressed it will kill the agility initiative as quickly as a routine 60 hour workweek or fixed price/fixed schedule/fixed functionality mandate.

With architecture, as with all things agile, I’m reminded to return to first principles, such as the agile manifesto and the derivative works, but these works are mostly silent on the topic of enterprise-scale architecture. That’s understandable, as the methods weren’t developed for large enterprises. However, as we extend them and achieve the benefits thereto, we have to take the challenge to the next level of scale, without losing or compromising the principles that have brought us to agile success so far. To this end, I suspect that there are a set of guiding principles that we could posit to help us down this path. I’ll take a first cut at such a set in the upcoming posts in this new series: Six Principles of Agile Architecture.

Here’s the set I’m going to start with.

Principle #1 ─ The teams that code the system design the system.

Principle #2 ─ Build the simplest architecture that can possibly work.

Principle #3 ─ When in doubt, code it out.

Principle #4 ─ They build it, they test it.

Principle #5 ─ The bigger the airplane, the longer the runway.

Principle #6 ─ System architecture is a role collaboration.

I’ll elaborate further in another post later this week. As always, thoughts and comments are welcome.