John Deere ISG Case Study Post #3: All Aboard the GreenStar™ Release Train

Background: In prior posts, we introduced the almost-real-time case study describing a large-scale agile transformation at John Deere’s ISG. In the last post, we described some of the work that was required to get Deere to the tipping point—the point at which we got the go-agile approval from the leadership team. This post describes the all-in “quickstart” implementation of the first ISG Agile Release Train.

Constitution of the Train

In preparation for training and release planning, the first challenge was to figure out the “domain” of the train; In other words, what teams needed to work together to deliver the end user value? While it seems obvious in hindsight, at the time it wasn’t at all clear what products and teams would constitute the train. In the end, we decide that it would be a big train, with as many as 100-130 practitioners, and it would encompass all the products generally carried under the GreenStar™ brand. Agile Coach, Chad Holdorf recalls,

“While there was a strong pull from parts of the GreenStar™ products to go agile, some were still unsure how this would work. Finally, we all realized that we couldn’t have part of the product on the train and the other parts still operating as is, so we just decided to roll it all together. We decided to include all currently supported products, some web portals and utilities, as well as some future platforms currently in advanced development. Cleverly, we then just decided to call it the ‘GreenStar™ Release Train’, even though not every system element was part of the GreenStar™ solution per se. But, we had to call it something!”

Organizing Scrum Teams and Assigning Responsibilities

Once we knew the domain, the next order of business was to establish the approximately 12-14 teams (Scrum team =  7+/2 people, including devs, testers, product owner and Scrum Master). Again, some were obvious as small teams were already working on some components, but there were new features and new platforms that also needed dedicated resources.  Plus all the testers reported into a separate organization, called Product Verification and Validation (PV&V). Chad worked with the line managers to create a spreadsheet “straw dog proposal” that we could at least use as a starting point.

Chad comments:

“This was actually quite difficult. It sounds easy, but when people are already on 2-5 projects at one time, it was a challenge to not put a person on more than one team.” Once we formed teams, we then had to figure out roughly what each teach would be responsible for. This part was eye opening as some teams had many things that they were responsible for, but clearly they could not work on all of it at the same time.”

Susan Morrison, then PV&V manager, commented

“It was also a bit cathartic to think about refactoring the entire organization so that testers would be collocated with developers and likely became part of those teams, but my team leads were all in favor, and it was clearly the right thing to do for the business. So, we just rolled up our sleeves, and started making initial assignments of 1-2 testers per dev team”. Susan continues, “none of us knew exactly how our management roles would evolve, but sometimes you just have to take the plunge and trust the business to do the right thing.”

Picking ScrumMasters

Being that Agile and Scrum were relatively new to the majority of the ISG organization, we needed some people that were familiar with Scrum and planning activities to help ensure the teams were on track to meet their objectives. Many of our Program Managers were assigned the role of Scrum Master to help the teams follow the Scrum process. Because there were so many teams, some Program Managers were the Scrum Master for multiple teams. This created problems during Release Planning because we were all planning at the same time; Program Managers struggled to support all teams. Since then, we have dedicated Scrum Masters for each team.

Picking Product Owners

Given the critical role that the product owner plays (see posts), picking the right people for that role was no trivial task either. Chad comments

“We recognized immediately that finding and empowering Product Owners that could guide the team toward our business goals was a major challenge and a key success factor. Not only was this a new role, but it also created a bit of an organizational and political challenge. A great Product Owner needs to understand the organization’s business goals and the product’s capabilities and technologies. As importantly, the Product Owner needs to love to work daily with developers and testers, and most importantly, can make crisp, on-the-fly decisions that affects the product direction. They have to have the full respect of the developer, too. Picking current business analysts or system requirements folks and just inserting them into this role is’t always a good fit. Instead, we decided to find the skills we needed first and then didn’t hesitate to pull them from any functional area.”

The Greenstar™ Training and Release Planning event

With that behind us, we were ready for the fun part—launching the train. We considered a number of rollout strategies, including incremental, but decided it would be easiest to get everyone on board asap. Further, to avoid dragging it out and potentially suffering either a loss of momentum, or subjecting the teams to different methods or trainers, we decided to do it one time only, all at once, and all in one week. We thought that would minimize disruption and make the move to agile as fast and efficient as possible. Also, there could be no looking back, as the “boats that got us here” were left far, far behind. To this end, we built a training plan as illustrated below:

An All-In Agile Program Quick Start

Of course, it was no trivial feat to get 100+ people trained and transitioned to agile in a week, but it was really, really fun and the teams themselves provided the energy (we provided trainer, facility and food!) Chad made this cool video which gives you a sense for the magnitude and dynamic nature of event:http://www.youtube.com/watch?v=UPMBh-E7ahY.

My partner/agilemaster/Alex Yakyma wasn’t actually there, but it looked a lot like this other one that he pictured for me.

A Special Note on the Role of “Subtle Control” of Self-Organizing Teams

Well, that’s it for this post, describing how we kick-started the first ISG release train.

This was an exercise in semi-self organization of a significant portion of an enterprise, and self-organization is an interesting topic in agile.  A few years back, while researching the roots of Scrum/agile in preparation for the writing of Scaling Software Agility, I found this presentation by Jeff Sutherland. In turn, that triggered a vector into a 1986 Harvard Business review article entitled the New, New Product Development Game. This is one of the most interesting things I’ve ever read on concurrent engineering, and product development in general.  For me, it’s the Rosetta stone of most things agile, and I highly recommend it to all. In the article, authors Takeuchi and Nonaka note:

“Although project teams are largely on their own, management establishes enough checkpoints to prevent instability, ambiguity and tension from turning into chaos…..

That will be the topic of the next post in this series: “Subtle Control”

Background on John Deere ISG

John Deere Intelligent Solutions Group plays a key role for Deere & Company in designing and delivering the technology needed by their customers to help them meet the challenge of growing more food and building much-needed infrastructure – challenges that must be met as the world’s population is expected to grow to 9 billion people by 2050.  They’re a great place to work, too. With technology that utilizes satellite-based global-positioning, John Deere Intelligent Solutions Group designs displays and receivers, guidance systems, field and crop management, and information and logistics systems their customers rely on. They leverage Agile project methods to deliver the most advanced innovations to their products. They foster a creative environment where employees feel empowered to put their best ideas forward and forge their own career path.” (And yes, they have ping pong and foosball tables. Maybe a future post about office environment changes).

Jobs at John Deere ISG

Like every successful software business, Deere is always looking for talented people. If you are interested in a new career, building software with agility, and more importantly, helping address the challenges of feeding the world, contact Arneda Shelton and check out: www.johndeere.com/careers/

John Deere ISG Case Study: Reaching the Tipping Point

In an earlier post, we introduced the Agile transformation occurring at John Deere’s Intelligent Systems Group. In this post, we’ll describe some of the challenges and activities we encountered to get the development and management teams to the “agile tipping point” – the point at which we got the final green light for an “all in” Agile transformation.

Background: Earlier Pilots at ISG

In 2006, a few teams at ISG started experimenting with Agile development. Chad Holdorf and Steve Harty were two of the early adopters. Chad described it this way: “I remember being asked one day by Mano Mannoochahr, ISG Manager of Global Solutions Engineering, if I knew anything about Agile or Scrum. I replied ‘nope’.” Mano said, “Well then, it’s time you learned. I’d like you to lead the desktop teams (North America and India) and use Scrum.”

When asked about the change vector, Mano noted “the primary reason was to introduce the discipline of Agile –I just didn’t think our current process was producing the highest possible code quality. Plus with all the issues and product opportunities we were facing, we needed to increase the productivity of teams and the quality at the same time. Scrum gave us a way to get the whole team engaged more effectively towards that goal.”

Steve Harty, ISG Program Manager & (and now) Release Train Manager, commented:  “We knew we needed to increase our speed to market while keeping our budget and resources static. Moving our team to Scrum was scary, challenging, and liberating, all at the same time.  Scrum was ‘our little secret’ that helped move our delivery time timeframe from 12-18 months to 2-4 weeks. Plus, our engineering teams were happier and customer satisfaction went up.”

Based on this positive experience, an ISG Agile Transformation Team was formed to develop a plan to move forward with a much larger, and far more impactful, rollout.  Of course, as they did so many, many questions from important stakeholders started to appear.  For example:

1)   Is this an undisciplined process wherein we will lose control of requirements, system behavior and worse, the entire project?

2)   If we collocate testers with devs, what happens to the independent quality assurance function? Without requirements, how will we even know what we are supposed to test?

3)   What will engineering and test managers do if the teams are empowered and self-organizing?

4)   How will the factories react to our Agile plans? What about our continued need for long-term commitments to support new vehicle rollouts?

5)   What do architects do in agile? What happens to our architecture?

6)   Will software capitalization change, and if so, how?

7)   Will this even work with our Enterprise Product Delivery Process (stage-gated governance model)?

8)   What is all this write-the-test-first stuff anyway? That seems kinda stupid. And, are we going to have to do pair programming? What if we don’t want to?

9)   How do 10-20-30 or more Agile teams coordinate their activities? How can great solutions possibly emerge from all this potential entropy?

10)  How and who gets trained and what will the costs be to accomplish that? Who has the budget?

11)  How will long range portfolio planning work when the teams can change anything at any time?

12) Who will serve as ScrumMasters and what is a ScrumMaster anyway?

13) This Product Owner thing, who could we possibly empower to make such key decisions that affect our future?

14)  Will we suffer a loss of productivity during the transition? (“we can’t afford to go backwards now, not even one day….”)

15)  Does management really support this?

16)                  ……(editors note: we could do a whole post on this, but I tend to write too much anyway)………..

Clearly, there were more possible objections and objectors than we might have imagined. This was starting to look like a losing round of “agile objection whack-a-mole”. So working collaboratively, we organized our ongoing work in four threads.

#1 – Get management onboard and up to speed fast

As this was not “our first country fair”, we knew that we would never succeed without active support (and drive!) from middle and top management. So, in addition to a series of ad hoc informational meetings, we decided to educate them more formally as quickly as possible. After all, these are our leaders, so we resolved that they should lead, not follow, the transformation. (Note to a Scrum team: managers may be “chickens” to you, but to an enterprise, they are “pigs”, and big pigs indeed. Ignore them at your peril.) To this end, most managers participated in a two-day Lean|Agile Leadership Workshop we hosted. Workshop modules included Lean Thinking, Agile Orientation, Experiencing Scrum, Scaling Agile, Agile Technical Practices, and a final half day on Lean|Agile Leadership. This was a very successful series of trainings. We reached virtually all the managers in the R&D organization, as well as key stakeholders in marketing, product validation, factories, etc., in just a few short months. With that background, and an understanding of the potential business benefits, most of them came on board fairly quickly.

Lean|Agile Leadership Workshop. Reprinted with Permission of John Deere.

#2 – Establish groundswell and pull from the Dev Teams

Agile was developed by and for software developers and testers, but that doesn’t mean that all those who have never experienced Agile know what it is or necessarily think they want to do it. To that end, we did a significant amount of socializing with the team leads and first line development managers, looking, not only for support, but for a program that was ready to take the plunge. This was not a particularly difficult challenge, and it culminated at Agile 2010 over a dinner meeting, where we had the opportunity to discuss the challenges and opportunities face to face with a number of the Agile Transformation team members and various team leads.

Andy Beck, ISG Development Lead, Machine Controls, led the charge: “I always thought that Agile could be a great process, but until the conference, I had difficulty seeing how we could transform the John Deere waterfall methodology. After just a few days at the conference and that evening meal, the lights came on and the way was clear.  I knew that we needed to make a change in the culture at ISG and Agile had the potential to make a huge impact…if we could just get it started. Well then, might as well start with my teams.”

#3 – All In

Because ISG’s displays contain many millions of lines of code that had 200+ developers and tester working on it, we couldn’t only have half of the people on board. Joel Hergenreter, ISG Manager of Product Validation and Verification, noted, “Because our system was not well componentized, it would be difficult to implement Agile with a portion of our teams.  First, the merge strategy: how do we plan and continuously integrate if some teams are Agile and others are not in a tightly coupled system?  Second, once we implemented Agile on a few teams, others would want to jump on board quickly due to the strong cultural desire to deliver high quality products faster.  It would be impossible to say who wanted to make these types of improvements. We also knew that we would need more than Scrum to scale agile across the enterprise, so we adopted Dean Leffingwell’s Scaled Agile Delivery Model, which consists of a series of release “trains” operating on a common cadence.”

GreenstarTM 2630 Display, developed and produced by ISG. Reprinted with permission of John Deere.

Holdorf notes: “Once we agreed to add everyone to the train, we began drafting initial teams of 7 +/-2 that would work together until the 2630 launched. You wouldn’t believe how hard it was to just put people on ONE team and have then focus on ONE thing. People were so multiplexed it was painful and nearly impossible to understand the current situation or even write it all down, but we knew we needed teams to focus if we wanted to be successful”.

# 4- Establish a training plan and budget

Chad continues, “Because we were training so many people so quickly, we decided to train them all at once. I remember Dean asking me, ‘well, how big a room do you have, Chad? We’ll just stick ‘em all in it and just do it’.  We have a really big all employee room at ISG, so the game was on. Of course, we had to put together a budget for trainers and coaches, and that was not trivial, but at least we now had a plan to budget against (though I admit to being quite nervous about the scope and impact of even the initial training)”.

The Final Go Ahead

Eventually, we had enough support from all the key stakeholders, and a budget, so we were ready to forward. However, one small problem remained – we needed a program. One program – a display, controllers, guidance and documentation systems, and related software projects largely branded under the Greenstar^TM label –was an obvious candidate. Plus, that’s where we had established the most pull from Andy and other development managers and teams. However, therein, another problem appeared – that program was in the throes of a waterfall-end-game-prerelease-firefight, with an immovable goal of a January 15 product launch.  It was then September of 2010. Could we launch such a major transformation at the tail end of a waterfall program? Will we make it worse? NO? Are we sure? Do the potential benefits outweigh the real risks?  Or should we simply wait until next year?

With the courage of its newfound Agile convictions, ISG eventually concluded to simply go, and GO now, on the Greenstar program.  At that time, with the support of his peers, Larry Clemens, (Manager, Program Management), stepped up and signaled the final go ahead. Clemens noted the angst present during the deliberation and how he came to the “GO” decision: “There comes a time in all major change initiatives when you have to stop piloting and just make the change within the organization.  We don’t have any programs that are not critical to experiment with, so these are never easy decisions. But the longer you wait, the later you receive the benefits”.

Summary

Now we had our program, a budget, a method, consultants and trainers, dev teams, a really big room, and a lot of (perhaps not unanimous?) management support. Now, it was our time to execute. How hard can that be? Hmmmm.

Next post: The All Aboard the Greenstar^TM Release Train

Background on John Deere ISG

From the company’s job posting website:

John Deere Intelligent Solutions Group plays a key role for Deere & Company in designing and delivering the technology needed by their customers to help them meet the challenge of growing more food and building much-needed infrastructure – challenges that must be met as the world’s population is expected to grow to 9 billion people by 2050.  They’re a great place to work, too. With technology that utilizes satellite-based global-positioning, John Deere Intelligent Solutions Group designs displays and receivers, guidance systems, field and crop management, and information and logistics systems their customers rely on. They leverage Agile project methods to deliver the most advanced innovations to their products. They foster a creative environment where employees feel empowered to put their best ideas forward and forge their own career path.” (And yes, they have ping pong and foosball tables. Maybe a future post about office environment changes).

Jobs at John Deere ISG

Like every successful software business, Deere is always looking for talented people. If you are interested in a new career, building software with agility, and more importantly, helping address the challenges of feeding the world, contact Arneda Shelton and check out: www.johndeere.com/careers/

Agile Meets Big Iron: Agile Transformation at John Deere ISG

John Deere’s Intelligent Solutions Group (ISG) has recently announced that they are in the midst of a large-scale agile transformation that affects hundreds of software practitioners worldwide. This is a particularly interesting agile transformation for a number of reasons (many of which have been used as excuses by other enterprises as to why NOT to go agile), including:

1. It’s BIG and they took an “all in” approach. Many hundreds of developers were immediately impacted.
2. It’s distributed. Affecting teams in the US, Europe and India.
3. Systems are embedded, real time and complex.  ISG builds solutions that consist of displays, receivers, guidance systems and more for BIG hardware, including tractors, sprayers and combines.  They also build telecommunication systems, websites and portals.
4.  Delivery dates are fixed. New vehicles leave the factory every year, and they leave on time. The software has to be extremely high quality, and it must be ready on time, too. (No easy pushes to a web site for corrections, here)

Here’s a few pictures, just to give you some context on building software for things that really can “go bump in the night” (daytime too).

John Deere Tractor with GPS guidance system. Reprinted by permission of John Deere.

GreenstarTM 2630 Display, developed and produced by ISG. Reprinted by permission of John Deere.

Chad Holdorf, Scaled Agile Coach at ISG, has started blogging about his experiences at http://www.scaledagiledelivery.com . He has already put a number of interesting posts, all relevant to large-scale development, and many showing advanced agile practices at scale.

A Case Study

I’ll be blogging about this transformation too, in a lightweight “case study” form. The next post in the series, tentatively entitled “Reaching the Tipping Point”, should be available in about a week. Follow on posts will focus mostly on the major rollout activities and how they successfully addressed the larger challenges, challenges that (most) all large enterprises face when they head down this path.

A Special Thanks to John Deere

Rarely in our industry do we get chance to describe a real transformation, in real time, in a real company, so we are very appreciative of John Deere for allowing us to describe their experiences in this format. And we take pride in Deere’s courage to drive technological innovation in a business that was founded in 1837! Our hope is that these experiences might encourage others to follow down this path, and receive the many benefits – including the improvements, in morale, job satisfaction, and yes, just plain fun – that every agile enterprise can experience.

Background on John Deere ISG

From the company’s job posting website:

John Deere Intelligent Solutions Group plays a key role for Deere & Company in designing and delivering the technology needed by their customers to help them meet the challenge of growing more food and building much-needed infrastructure – challenges that must be met as the world’s population is expected to grow to 9 billion people by 2050.  They’re a great place to work, too. With technology that utilizes satellite-based global-positioning, John Deere Intelligent Solutions Group designs displays and receivers, guidance systems, field and crop management, and information and logistics systems their customers rely on. They leverage Agile project methods to deliver the most advanced innovations to their products. They foster a creative environment where employees feel empowered to put their best ideas forward and forge their own career path.” (And yes, they have ping pong and foosball tables. Maybe a future post about office environment changes).

Jobs at John Deere ISG

Like every successful software business, Deere is always looking for talented people. If you are interested in a new career, building software with agility, and more importantly, helping address the challenges of feeding the world, check out: www.johndeere.com/careers/

Agile in Regulated Environment: Abbott Labs Experience Report

In earlier posts in this series on Agile Development as Applied in High Assurance and Regulated Environments, I mentioned that applying agile development in the context of highly regulated environments is not a new thought, and at least some others have gone down this road before. And while the analysis of CFR 820.30 we did in the last post might indicate a waterfall mentality, the fact is that it doesn’t have to be that way at all.

One real-world example is from Abbott Labs in the development of a nucleic acid purification platform and companion real-time analyzer (whatever that is!) .  Some of the team members gave a presentation at Agile 2009 in Chicago. (I’ve contacted the authors to see if the PPT is still posted on line somewhere. I can’t seem to find it now. If I hear back, I’ll update this post with a link at some point.)

Here’s a few “grabbers” for context:

– “software was developed in iterations (short time-boxed iterations or sprints) and released in increments (collection of iterations, validated etc). –

– it is a myth that you can predict the requirements up-front

– Results: estimated team size and schedule reduction of 20%-30%.  Estimated cost savings of 35%-50%.

– at time of launch, a number of features, once thought to be essential, were not included. Some were deferred for up to three years …nonetheless the product was highly successful and trading off features for three years of sales is an easy choice.”

This surely speaks to the fact that medical device doesn’t have to be developed in a waterfall model, and indeed when it isn’t, typical agile business benefits may well result. The whitepaper on the same topic, Adopting Agile in an FDA Regulated Environment can be purchased here.

In future posts, I’ll be introducing an agile process model for use in high assurance software development that reflects this key fact.

[Nov 10 Update. A reader, Rob Matheson found the deck on line at http://c-spin.net/2009/cspin200909Agile_In_Regulated_Experience_Report.ppt. It actually has deeper  content than the written report.

Thanks Rob!]

Agile Executive – Final, Six Part BMC Executive Interview Now On Line

Israel Gat (VP, BMC Software) has just completed a six part interview series on the BMC agile transformation experience at the Agile Thinkers blog that I’ve been highlighting in some recent posts. This series is an open, honest and direct perspective from an experienced senior executive faced with all the challenges of implementing agility at scale in a formerly waterfall organization. It’s a quick read and yet it is filled with pithy and memorable takeaways on the executive perspective and “cultural gestalt” of enterprise agility. I strongly recommend this series to any organization headed down the agile path.

More on the Virtues of the Agile Executive

Israel Gat continues his “view from the executive suite” discussion of BMC’s successful transformation at the Agile Thinkers blog, noting that agile has now reached approximately 1,000 practitioners at BMC. That puts it in the class of one of the three or four largest transformations that I am personally aware of, truly enterprise scale. Moreover, these were not pockets of 10-50 people working on individual projects, these were many hundreds of people working together on highly complex, interdependent systems.

It’s really refreshing to get an engaged executives view on the challenge of the transformation at this level of scale and with this level of frankness and openness.

In this post, he highlights three core “ingredients” that characterized the view from the executive suite at BMC. They are:

1. Intentionality – a one way trip to agility without looking back
2. Trust – they trusted the teams and the teams “trusted them back”
3. Risk taking as an acceptable art. Without risk, can there really be any reward?

But this is a woefully inadequate summary so please read the post at http://www.agilethinkers.com/2008/07/q2—israel-gat.html.

Update on British Telecom Agile Adoption

I met Roger Leaton (British Telecom employee and Agile Advocate) and Geoff Watts (consultant, InspectandAdapt.com) at an in-house agile community gathering last week. They gave a peer presentation of the status of the agile rollout that they have been involved with at British Telecom. (A similar presentation was provided last year at Agile 2007). Roger noted that there were about 8,000 IT employees at BT and he estimated that as many as 25-30% of these had received Scrum training and were now operating in a substantially agile manner. That correlates to approximately 2,000-2,500 new agilists at BT worldwide. This is the largest rollout that I am personally aware of. (If anyone knows of even larger rollout, please shoot me an email.)

While, our paths had never crossed before, I was struck by the similarities of their experiences and my own. Clearly, there are some common patterns in Scrum and agile adoption in the really large scale, and I’ll comment on those on future posts. For now, however, with BTs permission I’ve posted their slides here (Scrum at BT) as a work-in-progress form of case study. Heads up though, it’s a 7MB download.

Roger notes that they have been collecting some hard metrics on the rollout, (hopefully to include productivity and quality measures) and we could hope to hear more from them soon.

Two New Enterprise Case Studies

I’m always on the lookout for documented case studies of agility at enterprise scale which highlight impediments, lessons learned, anecdotal benefits, and wherever possible, hard metrics of improvement in productivity, quality or morale. I’ve (b)logged two such case studies, the transformations at BMC Software and Salesforce.com in this blog under the category “Case Studies”.

Since then, British Telecom, and Siemens each presented an experience report in presentation form at the Scrum Alliance Gathering in November of 2007. While not exactly written case studies with lessons learned, benefits, metrics, etc., and while it’s unclear exactly how many practitioners they transformed, these presentations do provide insights of agile “crossing the chasm” in two extremely large software enterprises.

They can be found on the Scrum Alliance Website at:

Agile at Siemens

Scrum at British Telecom

New Case Study: Enterprise Agility at Salesforce.com

Salesforce.com recently completed an agile transformation involving a two-hundred person team in a three month window. Wow, that’s the big bang approach to be certain. My friend Pete Behrens was involved in this work, so you know it’s good work and you can can find the case study at:

http://trailridgeconsulting.com/files/salesforce_agile_adoption_2007.pdf

As is often the case, the “lessons learned” are the pithiest part. In short, Salesforce recommends:

1) have executive commitment to the change
2) create a dedicated rollout team to facilitate the change
3) focus on principles over mechanics
4) focus early on automation and continuous integration
5) provide radical transparency
6) leverage external agile training and coaching.

New Blog from Paul Beavers at BMC

Paul Beavers is a Senior Director of Engineering at BMC Software, and since he was one of the lead executives in their agile transformation, we’ve been encouraging to him to share his experiences. He just started his new blog at http://every2weeks.wordpress.com/. Check it out and get a glimpse of the passion that drives so many of us to continue to improve our software outcomes.