Agile Architecture Principle #7 – There is no monopoly on innovation

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six (ok, now seven) principles that we can apply to help us reason about the challenge of architecting systems of scale in an agile enterprise.

“Inertia is the residue of past innovation efforts.
Left unmanaged it consumes the resources required to fund next-generation innovation.”
—Geoffrey Moore

In an earlier principle, the bigger the system, the longer the runway, we described the need for teams to have sufficient architectural runway to be able to reliably “land” near term iterations and releases. In a sense, this is the upside of inertia ‒enough is known from prior experience and development to allow the team to provide reliable and consistent value delivery. Indeed, with agile, we have strong technical and project management mechanisms (and some architectural runway) in place to help us stay focused to this sole purpose.

In other words, agile practices provide a disciplined, production-like ability to reliably meet commitments and to rapidly evolve a system to meet existing customer requirements. We treasure our new-found reliability as a very good thing indeed.

But there is a downside as well. If we are not careful, the “tyranny of the urgent” will cause us to keep our heads down and focused solely on near term deliverables. So where does the innovation come from in such a model? Simply, mature agilists put processes in place to assure that innovation is not solely incremental and near term.

Some of our innovation practice comes from empowering the role of the system architects as part of our advanced guard, to be constantly exploring new technologies, patterns and techniques that can help us innovate. But ours is a team-centric model, so we clearly will not rely on architects as the sole source of such innovation. In fact, the team-centric model can foster innovation at an even greater rate than that generally seen in traditional software organizations. This is because true innovators innovate at all stages of their career (entry level, junior and senior) and the team-centric model enables these folks to flourish and contribute beyond what their level of experience may otherwise imply.

One way to foster iteration at the team level is by judicious backlog management that includes spikes for refactoring, design and exploration of new ideas. This can work quite well, but even more explicit models have been put into use. For example, at Rally Software Development, where they have been building their SaaS Agile Product management solution in a highly agile fashion for four years (with rarely a missed or delayed release commitment), they have evolved to an advanced development cadence as illustrated in the figure below.

Figure – An iteration and release cadence with one innovation “hackthon” per release

This figure illustrates a standard release cycle, where

“i” is a standard development iteration, providing new functionality for an upcoming release

“h” is a one week hardening iteration, to eliminate technical debt and assure quality requirements meet the release to manufacturing criteria

“k” is a “hackathon”.

The hackathon is designed to foster innovation at the team level. The rules of the hackathon are simple: any team member can explore any technology area in any way they want, so long as there is some correlation to the company’s mission. This gives the team some mental down time‒to reflect, think and experiment‒outside of the everyday rigor and pressures of the iteration and release cycle.

With a model such as this, innovation is expected and programmatic, and there can be no ambiguity to the point to this (and our last!) Principle # 7 — There is no monopoly on innovation.

New enterprise agile blog at agilejuice.wordpress.com

As if it wasn’t obvious from its name, my blog is dedicated to the challenges of agility at scale, and I try to stay focused on that space because that is where so many of the challenges and so many of the potential benefits of agile adoption lie. Of course, it’s often a mix of team and enterprise practices, because if something doesn’t work for the team, it doesn’t work for the enterprise. And if there’s an impediment in the enterprise, then agile probably doesn’t work well for the team, which doesn’t work for the enterprise, and well, you get the idea, it’s always a mix.

In any case, I’ve been aware of a new blog devoted to enterprise challenges that some people I know have been working on. It’s at http://agilejuice.wordpress.com. Here’s an intro:

“Agile Juice is a blog dedicated to the advancement of software agility. We’re experienced agile advocates in the Telecom / Cable software industry. With the help of others throughout the software industry, we have learned a ton and in the process, have been successful at leveraging agile best practices to deliver enterprise scale BSS and OOS software of high value in within reduced timeframes. It’s now our turn to give back to the community. For the bulk of our careers, our agile experiences have been centered on development shops of 30 folks or less. However, that has recently changed and over the past year, we have transformed a 100+ resource development shop into a synchronized agile release train that operates in concurrent and parallel 2 week iterations.”

I know these guys pretty well and their recent experiences have a lot to contribute to our growing pile of intellectual property and best practices of agile at scale. Their latest post (If you see the spotlight, don’t ignore it) points out one of the challenges with agile – if an individual can’t keep pace with the intensity of the quality software delivery process – then that can be a big and immediate cultural challenge for the enterprise.Here’s a quote that should tweak your interest:

“What if it’s customary to give average performers second, third and even fourth chances? In the agile world, you are dead.”

I suspect you’ll find it interesting reading, so I’ve added them to my blogroll.

 

 

 

Agile Architecture: Role of the System Architect in the Agile Enterprise

In the last post (What is Intentional Architecture?) we provided the motivation and a description of the role of Intentional Architecture in agile, enterprise-class application development. I suspect that most enterprise agilists would agree that there are substantive benefits when Intentional Architecture is effectively applied (so long as we don’t slow down development or use the need for up-front architecture as a capitulation to the waterfall design phases of the past!).

Historically, Intentional Architecture was a primary function of the system architect. But the most common agile methods don’t define (or necessarily even support) such a role, so a serious agile paradigm mismatch is likely to occur. These system architects (who, by the way, often hold director-level, VP, or CTO positions!) likely have decades of experience in the technology and the domain and have strong views about the intended architecture. However, the bulk of their implementation experience has happened outside an agile context, so they may not understand the practice of building refactorable code. Indeed, refactoring (mostly good) may all look like unnecessary rework (mostly bad) to them so they may not be initially supportive of our model, which depends on refactoring as a primary technique. They may also be concerned about the potential architectural entropy of all those newly empowered and energized agile teams. In addition, they may have equally strong opinions about the software development practices teams employ. If we fail to bring these key stakeholders on board to our agile development paradigm, they could kill the initiative as quickly as a routine sixty-hour work week. In some cases, we have witnessed the Battle for agile architecture: the agile teams vs. the system architects, and there can be no winner in that battle.

Fortunately, enterprise agility is not a zero sum game and there is room for all who can contribute to the best possible technical solutions. As we are continuously raising the bar to build systems of increasing complexity the likes of which have never been built before, it only makes sense to leverage the skills of all who have the experience to match the challenges the teams face! So include them we will and we described our support for this role in Principle #6 – System Architecture is a Role Collaboration.

Agile Architecture: What is Intentional Architecture?

We started this series of posts with a discussion of the fact that continuous refactoring of emerging-only architectures in enterprise-class software systems becomes problematic as the size of the system grows. In addition, to improving usability, extensibility, performance and maintenance, it seems evident that we might wish to build systems out of components that work in a like manner. Thus there are some practical motivations to apply some “architectural guidance” to the software development practice, the benefits of which include:

Avoiding large-scale and unnecessary rework ‒ Even apparently minor system-level refactors can cause substantive rework for large numbers of teams, some of whom would otherwise NOT have to refactor their module. (It is one thing for a team to refactor their code based on lessons they have learned, it’s another to require multiple teams to refactor code based on other teams lessons learned).

Managing the risk of Impact on deployed systems/users ‒ Even the best possible BVT (Build Verification Tests) systems are imperfect, so the threat of a regressive bug in a deployed system is always there; the cost, risk and impact of which increases with scale. Minimizing unnecessary refactoring is a primary risk mitigation technique.

Improve usability, extensibility, performance and maintenance ‒ Some common, imposed architectural constructs can ease usability, extensibility, performance and maintenance. For example, something as simple as imposing a common presentation design can result in higher end user satisfaction, and ultimately, revenue!

The need for this guidance brings us to the role of intentional architecture, (described further in Chapter 16 of SSA) an enterprise practice designed to produce and evolve robust system architectures in an agile fashion. Intentional Architecture has three primary objectives:

1. Leveraging common architectural patterns, design constraints and implementation technologies – These decisions that can simplify development, extensibility and usability. Some can be defined up-front (examples: use the existing user activity logging component; do all web GUIs for the new subsystem in PHP). Others emerge during the course of development and can be leveraged by other teams if we only take the care to do so.
2. Building and maintaining architectural runway – architectural runway exists when there is sufficient system infrastructure to allow incorporation of nearer-term product backlog without potentially destabilizing refactoring. In most agile practices, it is sufficient to have six months or so of runway, so that there is a high probability that the highest priority backlog items can be reliability committed to the nearest, upcoming releases.
3. Sponsoring Innovation – In many ways, agile natively fosters innovation by more quickly driving solutions to better meet real user needs. Alternatively, however, agile can also be hostage to the “tyranny of the urgent”, as our rapid iteration cycles and continuous commitment to value delivery may drive us to avoid risky or innovative experiments. When everyone is committed and accountable to a near term deliverable, who is scouting the next curve in the road?

In my experience, as an agile enterprise matures, the role of architecture resumes a place of prime importance in assuring overall efficiency of the development process and helping to assure the systems that are built provide world-class scalability, performance and extensibility. By judiciously applying some Intentional Architecture, teams can build better systems without any need to abandon the paradigms that have delivered the outstanding benefits in productivity, quality and morale that agility provides.

Agile Architecture Principle #6 – System architecture is a role collaboration

Note: this is one in a series of posts under the category of “agile architecture”. In an earlier post, (Six Principles of Agile Architecture) we identified six principles that we can apply to help us reason about the challenge of architecting systems of scale in an agile enterprise.

In prior posts, we’ve discussed the first five principles (see thumbnail at left). In this post, we’ll discuss #6 – System architecture is a role collaboration.

In this series of posts to date, we’ve focused on the central role the teams play in designing and architecting systems of scale. For example, in Principle #1, we proposed that the teams that code the system design the system, so there could be no ambiguity about who had the responsibility for architecting the system – the team that is accountable for implementing it. If we stopped here, it wouldn’t be obvious that there is a role at all for those senior, domain experienced, system architects that play such a key role in many larger enterprises. This topic gets particularly entertaining, when as an agile proponent, I meet these people and hear their strong views about imposing architectural governance on the natural entropy of all those newly empowered (and sometimes relatively inexperienced) agile teams.

Most typically, these folks have deep technical and domain experience as well as the seniority, competence and authority to back up their opinions. Indeed they have a lot to add to the vision and implementation and it would be downright foolish of us to leave them out of the mix of what “makes an agile team a team”. Moreover, they have a strong influence on the process models the teams will adopt and if they are not supportive of an agile transformation, one may or may not take place!

Fortunately, enterprise agility is not a zero sum game and there is room for all who can contribute to the best possible technical solutions. After all, we are building systems of enormous complexity (even when we keep them as simple as possible), why would we not leverage the skills of those external team members who have the experience to match the challenges the teams face?

So leveraging their expertise is to the benefit of the team and the enterprise, but how do we incorporate them in our team-centric agile model? I’m reminded of the parallels between organizations and the systems architecture that organizations create. Conway’s law states that: “Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations” While this law may seem to imply a negative, that needn’t be the case. For example, in object oriented systems development, we design systems of collaborating objects, structured around well defined interfaces, which work together to create an output that is greater than the sum of its parts. The same approach can be applied as to how we work together to develop systems architecture (see Figure 1).

In this figure, you can see that system architecture can be seen as role collaboration between the system architect and tech leads of the component teams that write the code. These system-level teammates work together with the component teams to decide what the architecture needs to be. And when in doubt, they code it out (principle #3) though a series of design spikes inside iteration or release boundaries. With the support of the product owner, these spikes are mixed in the backlog based on whatever priorities the team feels is appropriate, (this is one of the primary reasons we like to see product owners with a high degree of technical experience) and over time, a consensus emerges as to how to build the system they are about to deploy.

Ok, that’s pretty theoretical, perhaps an example will help. In the last post, I noted a Single-Sign-On (SSO) architectural consideration that came from a real world experience. In this application, the component teams needed to implement SSO from their system to an external web service. The component teams had little experience in web SSO, having had no prior need (and they weren’t following the market trends for a need they hadn’t presently have!) The system architect had broader experience, and perhaps more importantly, had been researching SSO options prior to the release planning session where the discussion was initiated. There were certainly many options available to the team – from lightweight open source protocols, to higher security, SAML standards-based implementations, to a variety of vendor proprietary solutions. The system architect had formed an opionion based on prior investigation. Even with that input, a lively discussion ensued.

So who is to pick – the team closest to the implementation or the architect with more experience but who is further from the implementation? The answer must come via collaboration and agreement between the parties, supported as necessary by evaluation matrices (footprint, recurring cost, level of security, etc.) and where necessary, a series of design spikes to test each proposed solution. After all, if a design spike or two couldn’t prove the feasibility of one choice or another, then it’s unlikely that the solution chosen is likely to be the simplest architecture that could possibly work (Principle #2).

In summary, it is only by working together and by maximizing the contribution of all team players that we can hope to beat the competition by mastering systems of the complexity that we are crafting today. To do so, let’s engage the system architects in our solution via Principle # 6 – system architecture is a role collaboration.