Agile Architecture: Role of the System Architect in the Agile Enterprise

In the last post (What is Intentional Architecture?) we provided the motivation and a description of the role of Intentional Architecture in agile, enterprise-class application development. I suspect that most enterprise agilists would agree that there are substantive benefits when Intentional Architecture is effectively applied (so long as we don’t slow down development or use the need for up-front architecture as a capitulation to the waterfall design phases of the past!).

Historically, Intentional Architecture was a primary function of the system architect. But the most common agile methods don’t define (or necessarily even support) such a role, so a serious agile paradigm mismatch is likely to occur. These system architects (who, by the way, often hold director-level, VP, or CTO positions!) likely have decades of experience in the technology and the domain and have strong views about the intended architecture. However, the bulk of their implementation experience has happened outside an agile context, so they may not understand the practice of building refactorable code. Indeed, refactoring (mostly good) may all look like unnecessary rework (mostly bad) to them so they may not be initially supportive of our model, which depends on refactoring as a primary technique. They may also be concerned about the potential architectural entropy of all those newly empowered and energized agile teams. In addition, they may have equally strong opinions about the software development practices teams employ. If we fail to bring these key stakeholders on board to our agile development paradigm, they could kill the initiative as quickly as a routine sixty-hour work week. In some cases, we have witnessed the Battle for agile architecture: the agile teams vs. the system architects, and there can be no winner in that battle.

Fortunately, enterprise agility is not a zero sum game and there is room for all who can contribute to the best possible technical solutions. As we are continuously raising the bar to build systems of increasing complexity the likes of which have never been built before, it only makes sense to leverage the skills of all who have the experience to match the challenges the teams face! So include them we will and we described our support for this role in Principle #6 – System Architecture is a Role Collaboration.

Agile Architecture: What is Intentional Architecture?

We started this series of posts with a discussion of the fact that continuous refactoring of emerging-only architectures in enterprise-class software systems becomes problematic as the size of the system grows. In addition, to improving usability, extensibility, performance and maintenance, it seems evident that we might wish to build systems out of components that work in a like manner. Thus there are some practical motivations to apply some “architectural guidance” to the software development practice, the benefits of which include:

Avoiding large-scale and unnecessary rework ‒ Even apparently minor system-level refactors can cause substantive rework for large numbers of teams, some of whom would otherwise NOT have to refactor their module. (It is one thing for a team to refactor their code based on lessons they have learned, it’s another to require multiple teams to refactor code based on other teams lessons learned).

Managing the risk of Impact on deployed systems/users ‒ Even the best possible BVT (Build Verification Tests) systems are imperfect, so the threat of a regressive bug in a deployed system is always there; the cost, risk and impact of which increases with scale. Minimizing unnecessary refactoring is a primary risk mitigation technique.

Improve usability, extensibility, performance and maintenance ‒ Some common, imposed architectural constructs can ease usability, extensibility, performance and maintenance. For example, something as simple as imposing a common presentation design can result in higher end user satisfaction, and ultimately, revenue!

The need for this guidance brings us to the role of intentional architecture, (described further in Chapter 16 of SSA) an enterprise practice designed to produce and evolve robust system architectures in an agile fashion. Intentional Architecture has three primary objectives:

1. Leveraging common architectural patterns, design constraints and implementation technologies – These decisions that can simplify development, extensibility and usability. Some can be defined up-front (examples: use the existing user activity logging component; do all web GUIs for the new subsystem in PHP). Others emerge during the course of development and can be leveraged by other teams if we only take the care to do so.
2. Building and maintaining architectural runway – architectural runway exists when there is sufficient system infrastructure to allow incorporation of nearer-term product backlog without potentially destabilizing refactoring. In most agile practices, it is sufficient to have six months or so of runway, so that there is a high probability that the highest priority backlog items can be reliability committed to the nearest, upcoming releases.
3. Sponsoring Innovation – In many ways, agile natively fosters innovation by more quickly driving solutions to better meet real user needs. Alternatively, however, agile can also be hostage to the “tyranny of the urgent”, as our rapid iteration cycles and continuous commitment to value delivery may drive us to avoid risky or innovative experiments. When everyone is committed and accountable to a near term deliverable, who is scouting the next curve in the road?

In my experience, as an agile enterprise matures, the role of architecture resumes a place of prime importance in assuring overall efficiency of the development process and helping to assure the systems that are built provide world-class scalability, performance and extensibility. By judiciously applying some Intentional Architecture, teams can build better systems without any need to abandon the paradigms that have delivered the outstanding benefits in productivity, quality and morale that agility provides.

Agile Architecture Principle #6 – System architecture is a role collaboration

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that we can apply to help us reason about the challenge of architecting systems of scale in an agile enterprise.

In prior posts, we’ve discussed the first five principles (see thumbnail at left). In this post, we’ll discuss #6 – System architecture is a role collaboration.

In this series of posts to date, we’ve focused on the central role the teams play in designing and architecting systems of scale. For example, in Principle #1, we proposed that the teams that code the system design the system, so there could be no ambiguity about who had the responsibility for architecting the system – the team that is accountable for implementing it. If we stopped here, it wouldn’t be obvious that there is a role at all for those senior, domain experienced, system architects that play such a key role in many larger enterprises. This topic gets particularly entertaining, when as an agile proponent, I meet these people and hear their strong views about imposing architectural governance on the natural entropy of all those newly empowered (and sometimes relatively inexperienced) agile teams.

Most typically, these folks have deep technical and domain experience as well as the seniority, competence and authority to back up their opinions. Indeed they have a lot to add to the vision and implementation and it would be downright foolish of us to leave them out of the mix of what “makes an agile team a team”. Moreover, they have a strong influence on the process models the teams will adopt and if they are not supportive of an agile transformation, one may or may not take place!

Fortunately, enterprise agility is not a zero sum game and there is room for all who can contribute to the best possible technical solutions. After all, we are building systems of enormous complexity (even when we keep them as simple as possible), why would we not leverage the skills of those external team members who have the experience to match the challenges the teams face?

So leveraging their expertise is to the benefit of the team and the enterprise, but how do we incorporate them in our team-centric agile model? I’m reminded of the parallels between organizations and the systems architecture that organizations create. Conway’s law states that: “Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations” While this law may seem to imply a negative, that needn’t be the case. For example, in object oriented systems development, we design systems of collaborating objects, structured around well defined interfaces, which work together to create an output that is greater than the sum of its parts. The same approach can be applied as to how we work together to develop systems architecture (see Figure 1).

In this figure, you can see that system architecture can be seen as role collaboration between the system architect and tech leads of the component teams that write the code. These system-level teammates work together with the component teams to decide what the architecture needs to be. And when in doubt, they code it out (principle #3) though a series of design spikes inside iteration or release boundaries. With the support of the product owner, these spikes are mixed in the backlog based on whatever priorities the team feels is appropriate, (this is one of the primary reasons we like to see product owners with a high degree of technical experience) and over time, a consensus emerges as to how to build the system they are about to deploy.

Ok, that’s pretty theoretical, perhaps an example will help. In the last post, I noted a Single-Sign-On (SSO) architectural consideration that came from a real world experience. In this application, the component teams needed to implement SSO from their system to an external web service. The component teams had little experience in web SSO, having had no prior need (and they weren’t following the market trends for a need they hadn’t presently have!) The system architect had broader experience, and perhaps more importantly, had been researching SSO options prior to the release planning session where the discussion was initiated. There were certainly many options available to the team – from lightweight open source protocols, to higher security, SAML standards-based implementations, to a variety of vendor proprietary solutions. The system architect had formed an opionion based on prior investigation. Even with that input, a lively discussion ensued.

So who is to pick – the team closest to the implementation or the architect with more experience but who is further from the implementation? The answer must come via collaboration and agreement between the parties, supported as necessary by evaluation matrices (footprint, recurring cost, level of security, etc.) and where necessary, a series of design spikes to test each proposed solution. After all, if a design spike or two couldn’t prove the feasibility of one choice or another, then it’s unlikely that the solution chosen is likely to be the simplest architecture that could possibly work (Principle #2).

In summary, it is only by working together and by maximizing the contribution of all team players that we can hope to beat the competition by mastering systems of the complexity that we are crafting today. To do so, let’s engage the system architects in our solution via Principle # 6 - system architecture is a role collaboration.

Agile Architecture Principle #5 – The bigger the airplane, the longer the runway

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

In prior posts, we’ve discussed the first four principles. In this post, we’ll discuss #5 – The bigger the airship, the longer the runway.We started this thread sometime back with a discussion of the rationale behind the build out of some amount of architectural “runway’ for systems of scale. To summarize, we posited that investment in some amount of intentional (or intentionally emergent) architecture is warranted based on the fact that continuous refactoring of large scale systems becomes problematic for three reasons.

• Avoiding larger scale, and unnecessary rework. Even apparently minor system-level refactors can cause substantive rework for large numbers of teams, some of whom would otherwise NOT have to refactor their component or module. It is one thing for a team to refactor their code based on lessons they have learned, it’s another to require multiple teams to refactor code they would otherwise NOT have had to refactor based on other teams lessons learned, especially when the teams backlog is already overloaded with value delivery priorities.
• Managing the risk of potential Impact on deployed systems/users. Since even the best possible BVT (Build Verification Tests) systems are imperfect, we worry about unnecessarily disrupting the install base of users (thousand and tens of thousands for enterprise architectures, even more for successful consumer systems). In other words, the threat of a regressive bug in a deployed system is always there, the cost, risk and impact of which increases with scale. In this case, minimizing unnecessary refactoring is a primary risk mitigation technique.

• Improve usability, extensibility, performance and maintenance. Some common, imposed architectural constructs can ease usability, extensibility, performance and maintenance. For example imposing a common presentation design and implementation can substantially improve usability and presentation layer maintenance and modification, resulting in higher end user satisfaction (and revenue!) In another example, imposing a common single-sign-on mechanism across a suite of products would simplify development implementation and maintenance, even though there was NO difference to the end user.

There’s a further reason that warrants discussion as well. At the release level (internal or external), value delivery focuses on delivering the features our customers need. Being able to deliver planned functionality somewhat predictably in a near term (60+days) release is a hallmark of mature agile teams. That ability, in turn, is how the agile enterprise is able to communicate expectations to customers, who often have many business dependencies on the release of new software. Since building and refactoring infrastructure for large scale systems typically takes longer than a single short release cycle, it is necessary that most features for a particular release can be built on infrastructure that is already in the system. Otherwise, the team won’t be able to reliably “land” the release on schedule. Therefore, you need enough runway to be able to build in features for at least a release or two, while larger scale refactorings may well be underway.

Further discussion of how agile teams go about building and extending runway can be found in Chapter 16 of SSA – Intentional Architecture, and is outside the scope of this post, but for now, we’ve made our case for Agile Architecture Principle #5 - The bigger the airplane, the longer the runway.

Agile Architecture Principle #4 – They build it, they test it

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

Posts Principle #1 – The team that codes the system designs the system, Principle #2 – Build the simplest architecture that can possibly work and Principle #3 - When in doubt, code it out described the first three principles. In this post, we’ll discuss the Principle #4 ‒ They build it, they test it.

Agile is renowned for its emphasis on testing (”If testing is good, everybody will test all the time” – Kent Beck, 2000) and there can be no doubt that agile forces testing earlier into the lifecycle than any of our prior methods. This is a key benefit of agile in general and is one of the prime reasons why quality is materially higher in agile, and is accomplished without sacrificing overall developer productivity. Indeed, testing is so important in agile that I devoted an entire chapter to it in SSA, Chapter 12 —Concurrent Testing. “But this is a thread about agile enterprise architecture”, you might ask, “so why the lecture on testing”? Fair question.

My friend Philippe Kruchten once described architecture as ” take away everything in the system you don’t need to describe how it works, and what you have left is it’s architecture”. With this simple definition, testing architecture comes down to “testing how the system works without the details”, that means that testing architecture is testing the system’s ability to meet its functional (in the large scale), operational, performance and reliability requirements.

Clearly, this testing represents complexity at its greatest and who has the skills to test systems of the complexity that we are building today? Answer – the team that codes the system must determine how best to test the system. Moreover, with the enhanced power and complexity of today’s automation frameworks, the developers are likely to be directly involved in applying testing automation to their system. And when such automations isn’t available, it is necessary for the development teams themselves to develop, test and maintain the system tests and testing frameworks to test its ability to meet its architectural and functional requirements. Such a responsibility cannot be left to any other testing resources or outsourced functions or the team cannot then be held accountable for delivering a functional system within the iteration or release timebox.

So agile architecture Principle #4 is: They build it, they test it, because in agile, All Code is Tested Code.

Agile Architecture Principle #3 - When in doubt, code it out

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

Two prior posts: Principle #1 – The team that codes the system designs the system, and Principle #2 – Build the simplest architecture that can possibly work described the first two principles. In this post, we’ll discuss Principle #3 - When in doubt, code it out.

I suspect there is a reason why most of us techies did not pick philosophy, political science, social studies, or any of the other softer careers. Our scientific orientation and data-driven analytical tendencies allow us to move forward comfortably with important decisions and not lose a lot of sleep over varying expressed opinions. This is a skill that should serve us well in the process of picking a design alternative or a high-impact infrastructure implementation choice. Even then, we occasionally find ourselves mired in technical debate. In agile, with its highly iterative, experience and code-based emphasis, we can often simply depend on our coding skills to move us efficiently through the decision-making process.

This Principle #3 - When in doubt, code it out, reminds us that whenever we have to make a tough choice, we can always turn to a rapid evaluation in code. Our fast one or two week iterations give us a quick project cadence and the demos at the end of the iteration provide objective evidence. Inherent visibility of the agile model also allows all affected stakeholders to see the real-time, in process reasoning and experimental results. In addition, Principle #1‒Build the Simplest Architecture that Can Possibly Work, reminds us that if a design alternative can’t be coded and evaluated within a few iterations, it probably isn’t the simplest thing! In practice, rarely have I seen a case where a decision wasn’t fairly obvious after a few short design spikes.

And for agile at scale, I’m again reminded of the lessons learned from the Amazon Architecture: (highlighted in the post Building Scalable, Robust Architecture with Agile: Lessons Learned at Amazon)

” Use measurement and objective debate to separate the good from the bad. I’ve been to several presentations by ex-Amazoners and this is the aspect of Amazon that strikes me as uniquely different and interesting from other companies. Their deep seated ethic is to expose real customers to a choice and see which one works best and to make decisions based on those tests. Avinash Kaushik calls this getting rid of the influence of the HiPPO’s, the highest paid people in the room. This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do, code it up, let people use it, and see which alternative gives you the results you want.”

Agile Architecture Principle #2 – Build the simplest architecture that can possibly work

Note: this is one in a series of posts under the category of “agile architecture”.

In an earlier post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise. This post discusses Principle #2 – Build the simplest architecture that can possibly work.

This one will certainly come as no surprise to even an agilist newbie, after all, agile is famous for its focus on simplicity:

“What is the simplest thing that can possibly work?” – Ward Cunningham

“If simplicity is good, we’ll leave the system with the simplest design that supports its current functionality.” – Kent Beck

YAGNI – You Ain’t Gonna Need It – an XP mantra

Of course, we are talking about systems of substantial complexity here, the types of systems that are crafted by hundreds of developers working over many years, rather than the types of systems developed in smaller team environments where agile practices were developed and proven, so it is a fair question to ask as to whether simplicity remains an essential attribute as the complexity increases.

Of course, it’s obvious from the postulate that I believe this to be the case, and this is supported by my own anecdotal evidence in building large scale systems. In most every case I remember (two “simpler” decisions I remember vividly: 1) dedicate a single service to a single class of server, 2) pick an off-the-shelf data base for those elements of the system that did not require extreme scalability), the simplest decisions turned out to be the best over time. And you needn’t take my word for it. Take Amazon.com, for example, where they grew a system in an agile and organic manner to eventually handle 55 million customer accounts and where between 100-150 services are accessed to build a page. In a recent post on the website highscalability.com, (http://highscalability.com/amazon-architecture) Todd Hoff notes a primary lesson learned as described by Amazon’s CTO, Werner Vogels:

“Only way to manage as large distributed system is to keep things as simple as possible. Keep things simple by making sure there are no hidden requirements and hidden dependencies in the design. Cut technology to the minimum you need to solve the problem you have. It doesn’t help the company to create artificial and unneeded layers of complexity.”

If simplicity of architecture is good enough for Amazon.com, it’s good enough for me.

Agile Architecture Principle #1 – The team that codes the system designs the system

Note: this is one in a series of posts under the category of “agile architecture”.

In the last post, (Six Principles of Agile Architecture) we identified six principles that can help us reason about the challenge of architecting systems of scale in an agile enterprise.

In this post, I’ll describe the first principle of agile architecture: the team that codes the system designs the system. This principle is driven both by the manifesto itself, (ie: The best architectures, requirements, and designs emerge from self-organizing teams) as well as another of the primary philosophies of agile development, which is that the teams themselves are both empowered to achieve, and accountable for, their delivered software.

From a management perspective, in order for teams to be held accountable, it naturally follows that we must allow them to make the decisions necessary to support that accountability – otherwise they are essentially being held accountable for decisions by others and that is neither an effective or motivating model for team performance! While this seems axiomatic, the reality of our behavior in earlier practice was somewhat different as the figure below shows.

This paradigm was further compounded by our time-sequenced waterfall activities; I.e. there was typically a single, up-front planning phase intended to accommodate the inherent risk of the project, perform all the necessary design and predict sequenced task dependencies running as far out as a year into the future. This phase was decoupled in time from the lessons learned in actual implementation and, once the lessons were learned, it was too late to do anything to the schedule (other than to apologize for the fact that we couldn’t meet it.)

Further, from a technical perspective, these architectural decisions are most optimally made by the coders, tech leads, team architects, etc, as they are closest to the implementation and therefore often (but perhaps not always as we’ll discuss Principle #6 – System architecture is a collaboration) have the best data available to make such a decision.

The combination of these two factors:

1) they will likely make a more optimum decision, and

2) once made, since they are accountable for the results, they will likely work a lot harder to make their decision work in the implementation

delivers a big part of the power of agile and more than warrants our first principle: the team that codes the system designs the system.

I’ll describe the second principle: Build the simplest architecture that can possible work, in my next post.

Six Principles of Agile Architecture (Part 1)

Many interesting discussions and debates about agile architecture, whether it be emergent, intentional, intentionally emergent, (or whatever!), continue as agile marches across the chasm into the enterprise. This issue gets particularly entertaining as I encounter deeply experienced and very senior architects, systems architects, VPs, CTOs and the like who have strong views about imposing architectural plans on the natural entropy of all those newly empowered and energized agile teams. Most typically, these folks are quite competent and they have the experience and authority to back it up their knowledge. More importantly, they have deep, experience-based technical insights to add to the process and it would be downright foolish of us to leave them out of the mix of what “makes an agile team a team”.

Also, the continuous refactoring of emerging-only architectures in these enterprise-class software starships becomes less and less practical as the size of the ship grows and matures (Chapter 16 of SSA). Therefore there is an understandable and reasonable desire to minimize refactoring as well as build systems of components that work in a like manner to ease usability, extensibility, performance and maintainability. These are strong and practical motivations to apply some “architectural governance” to the software development practice.

The overly simple argument of “the architecture will emerge ─ let the component teams decide” does not always go so well in some of those offices. For the reasons I’ve indicated above, some of the concernis reasonable. Other reasons include the fear of loss of control of some of the most critical, IP in the company, i.e. that which reflects an understanding of how the system works now and how it is intended to be extended over time. Whether this fear is rational or not, (and I suspect that it often is), if not addressed it will kill the agility initiative as quickly as a routine 60 hour workweek or fixed price/fixed schedule/fixed functionality mandate.

With architecture, as with all things agile, I’m reminded to return to first principles, such as the agile manifesto and the derivative works, but these works are mostly silent on the topic of enterprise-scale architecture. That’s understandable, as the methods weren’t developed for large enterprises. However, as we extend them and achieve the benefits thereto, we have to take the challenge to the next level of scale, without losing or compromising the principles that have brought us to agile success so far. To this end, I suspect that there are a set of guiding principles that we could posit to help us down this path. I’ll take a first cut at such a set in the upcoming posts in this new series: Six Principles of Agile Architecture.

Here’s the set I’m going to start with.

Principle #1 ─ The teams that code the system design the system.

Principle #2 ─ Build the simplest architecture that can possibly work.

Principle #3 ─ When in doubt, code it out.

Principle #4 ─ They build it, they test it.

Principle #5 ─ The bigger the airplane, the longer the runway.

Principle #6 ─ System architecture is a role collaboration.

I’ll elaborate further in another post later this week. As always, thoughts and comments are welcome.

Agile Axioms from Paul Beavers

Paul Beavers, Senior Director of R&D at BMC Software, has been blogging about the successful BMC agile enterprise transformation experience. Paul was a collaborator of mine on that fun project, and as a senior director, he was personally responsible for driving much of the change that facilitated success. He has been synthesizing this experience into a set of “agile axioms” and has been posting on his blog at working software every two weeks. Paul is a pretty direct guy and he doesn’t mince or waste words, so it’s some concise, pithy, and practical content. For example his first three axioms are:

Agile Axiom #1 - Agile development actually requires more discipline that traditional waterfall methods.

Agile Axiom #2 - Without automation it is mathematically impossible to stay releasable at the end of every iteration.

Agile Axiom #3 - An automated build system is critical to the success of most Agile software projects.

If you are transforming a large enterprise, you might want to subscribe to his blog.