Ah Ah, The Daily Build Revisited ….. Revisited

“We place the highest value on actual implementation and taking action.

There are many things one doesn’t understand; therefore, we ask them, why don’t you just go ahead and take action?

You realize how little you know, and you face your own failures and redo it again, and at the second trial you realize another mistake . . . So you can redo it once again.

So by constant improvement one can rise to the higher level of practice and knowledge.

This guidance reminds us that there is no problem too large to be solved if we are only willing to take the first step.”

— Fuijo Sho, President, Toyota

 

See the post “Ah, the Daily Build revisited”, dated March 23, which describes one agile teams initial reticence to invest in a true, automated, daily build process.
Six weeks later, I am back in the Ukraine and I am happy to report that the team has successfully converted to daily (nightly) builds. It was not a trivial project, nor was it as demanding as the team thought going in. The daily build process they use is as follows:

1) Check source code out of SCM system along with current versions of unit tests and functional tests

2) Run Make to build the application. Check logs. Abort if error.

3) Execute units tests on build server. Check logs. Abort if error.

4) Run configuration scripts to initialize/prepare Deployment Server

5) Deploy built application and unit tests to Deployment Server via SSH

6) Initiate Functional Test Framework (custom in this example) to execute functional test suites on application, builds error logs and reports.

During this process, however, they also discovered that the current BVT (Build Verification Tests) took a full 7 hours to run. Obviously this would kill the Daily build process, so the team was forced to refactor the functional tests to a lighter weight version. However, in doing so, they did not compromise the BVT quality as they simply refactored the test data sets to match only those data configurations that were actually used by the functional test framework and test suites. (in other words, as part of the BVT, they were spending time downloading and initializing data that was superfluous to the actual test cases.)

After about six weeks, the team had mastered the process and it is now a critical ability that helps the team build in quality and avoid the big bang crunch at the end of the iteration.

Kudos to yet another agile team that has now mastered this previously un-masterable process!

Project results:

Nightly build: Fully automated, daily build successful!
Project time: 6 elapsed weeks
Total person-hours invested: <80 hours

Lessons Learned:

1) There is some invention in developing scripts to automate all this stuff

2) If you discover that the BVT is just too big to run in less than an hour, stop and refactor it immediately until you have a sufficient test that can be run in less than an hour

3) Don’t be intimidated. Initially the team believed that the data dependencies and continuous refactoring of the baseline would prevent nightly build success.

Benefits:

§ Team has immediate feedback on any changes that affect the baseline and automated tests.

§ Small debugging of daily builds increases confidence. Confidence in the iteration grows daily, along with the code.

§ Daily builds foster communication. ( new developer: “I checked this in but broke the build, what did I do wrong?”)

Moral: Just Try It!

The Standard Two-Week (well maybe one is just as good or better…) Iteration

Philippe Kruchten, while reviewing the book in preparing for writing the foreword ( and thereby causing me no end of grief as well as the obvious need to spend a weekend or two rewriting some sections that he fairly, and roundly criticized) noted that in Chapter 11, Mastering the Iteration, that I was “ again being dogmatic about iteration length” with my steady drumbeat of a two week iteration.

I defended the recommendation as I have substantial and quite successful experience with this pattern in over a half dozen projects, large (think BMC and others) and small (no names provided because no one thinks their project is small) , and therefore left it as such in the book.

And now you might ask, (link) how do I then find myself directly managing an agile project with one week iteration lengths? The answer, I never even saw it coming. When we first started this project in summer of 2006, the team was brand new and new to each other, all raw recruits, the project was not well defined, the technology choices were all over the map. So naturally we started with shorter iterations, giving us a more frequent cadence and giving us the chance to review the exploratory results as well as individual talent on a frequent weekly basis. As the team learned the domain, we moved from exploration to hard core development and never changed the pattern.

Simply, the team learned how to divide their work into one week size chunks even as the system grew. When faced with a major refactoring or particularly large and seemingly atomic chunk of work, we would simply lay out the objectives in two or maybe three separate iterations, (Not necessarily forcing fully integrated functional outcomes at each boundary, but absolutely forcing fully testable outcomes, protocols, interfaces and the like) but all of one week length. The weekly pattern is so simple and obvious (“if it’s Tuesday I better have all my stories in acceptance testing…”) and so easy to manage, I doubt we’ll ever leave the pattern. We could, and we would likely be successful, but do we need to? Do we think we would benefit? – nope.

Will I update the two week to one week recommendation in the book? I have no idea, but I suspect you will never hear or see the words “three week iteration.”