Building Scalable, Robust Architecture with Agile: Lessons Learned at Amazon

For any out there worried about whether or not large scale systems can be build with emergent (or intentionally emergent) architecture, I found the following post on the “High Scalability” blog by Todd Huff.

In this post, Todd reflects on lessons learned at Amazon on architecture and scalability. The information was derived from various blog posts by Amazoners and interviews and talks with Amazon’s CTO, Werner Vogels.

I don’t know that Amazon applied agility in any defined methodology sense, but you’ll see agility spread throughout the philosophy of build-out and deployment of the architecture. I’ve included only a few pithy agilist-philosophy extracts with respect to enterprise-scale agility below. For a deeper treatment of their large scale, decentralized, SOA-based web services architecture, refer to the post itself and its various sources.

From the post:

  • Only way to manage a large distributed system is to keep things as simple as possible – making sure there are no hidden requirements and hidden dependencies in the design. Cut technology to the minimum you need to solve the problem you have.
  • You must change your mentality to build really scalable systems. In traditional systems we present a perfect world where nothing goes down and then we build complex algorithms (agreement technologies) on this perfect world. Instead, take it for granted stuff fails, that’s reality, embrace it.
  • Create a shared nothing infrastructure. Infrastructure can become a shared resource for development and deployment with the same downsides as shared resources in your logic and data tiers. A service oriented architecture allows the creation of a parallel and isolated development process that scales feature development to match your growth.
  • Organizing around services gives agility. You can do things in parallel is because the output is a service. This allows fast time to market. Create an infrastructure that allows services to be built very fast.
  • Use measurement and objective debate to separate the good from the bad. …. expose real customers to a choice and see which one works best and to make decisions based on those tests. ….get rid of the influence of the HiPPO’s, the highest paid people in the room. This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do code it up, let people use it, and see which alternative gives you the results you want.
  • People’s side projects, the one’s they follow because they are interested, are often ones where you get the most value and innovation. Never underestimate the power of wandering where you are most interested.
  • Create a staging site where you can run thorough tests before releasing into the wild.
  • Have a way to rollback if an update doesn’t work. Write the tools if necessary.
  • Innovation can only come from the bottom. Those closest to the problem are in the best position to solve it. Any organization that depends on innovation must embrace chaos. Loyalty and obedience are not your tools.
  • Creativity must flow from everywhere. Everyone must be able to experiment, learn, and iterate. Position, obedience, and tradition should hold no power. For innovation to flourish, measurement must rule.
  • Developers themselves know best which tools make them most productive and which tools are right for the job.
  • Don’t impose too many constraints on engineers. Provide incentives for some things, such as integration with the monitoring system and other infrastructure tools. But for the rest, allow teams to function as independently as possible.
  • You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer – essential for improving the quality of the service. ”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s