Software Verification and Validation in High Assurance Agile Development: Validation Sprint

Series Background: In this series of posts, I’ve been using medical device development (as Regulated by U.S. FDA via CFR 820.30 and international standard IEC62304) as an exemplar for suggesting ways to develop high quality software in regulated (and other high assurance, high economic cost of failure) environments in an agile manner. This series is sponsored, in part, by Rally Software Development. Thanks to Craig Langenfeld of Rally for his ongoing collaboration on this series.

In prior posts, I’ve introduced a suggested agile lifecycle model for verification and validation of software and a set of ground rules that agile teams will have to follow. I’ve also referenced an agile, requirements metamodel which defines the requirements artifacts and relationships we’ll use to implement and document required system behavior. For convenience, these two graphics are reproduced as thumbnails here:

We can see from the model, and as we describe in an earlier verification post, that verification occurs continuously during the course of each iteration. Validation, however, is a different matter entirely. This is the first in a short series of validation posts.

=============================================

In Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise (which was released just ten days ago, YES!), I described a “big picture” of enterprise agility which illustrated a series of development iterations punctuated periodically by a “hardening iteration” (sprint). This is pictured below, (with the iteration cadence highlighted in red):

This picture illustrates four development iterations (indicated by full backlogs) followed by a “hardening” iteration. This pattern is arbitrary but fairly typical as it produces a Potentially Shippable Increment (PSI) (valuable and evaluate-able in our medical device setting)  every 10 weeks or so. (Shorter than that, and the overhead of the hardening iteration may be too high, longer than that and the teams will have deferred final system integration and validation too long, resulting in delayed risk discovery—and delayed delivery…..).

The hardening iteration has an empty backlog, implying no new user stories, and yes, somewhat of a (note: agilists, prepare thyself for the following heretical words …) “code freeze” for new functionality. As , ideally, we could ship our product every day, the dedication of time to a “hardening” (or stabilization, or pick your word <here>) is not ideal from an agile purist standpoint,  the fact is that it is gives credence to a practical reality, development of the code itself is only part of the job, especially for systems of complexity and scale, and some of the work to prepare to release the code is likely to be most efficiently done outside a normal development iteration.

Therefore, in our experience, the hardening sprint is a high value, dedicated time for focusing on some of these remaining activities. Teams use the hardening sprint time to do whatever they have to do to prepare the system increment for potential use. This can include:

  • Technical debt such as defects, minor code refactoring,
  • Final, full, system-wide integration and regression testing
  • Testing of system qualities (nonfunctional requirements) such as performance, reliability, or standards compliance
  • Exploratory testing
  • Finalization of user documentation, release notes, internationalization etc
  • Preparation of assets for distribution and deployment

The Validation Sprint

In our medical device exemplar, this particular iteration takes on additional significance as it often used to perform the full “system validation” necessary to assure efficacy and safety for human use. Performing full validation for a high assurance system is no trivial effort, and will likely include at least the activities identified below:

  • Resolve/close/note outstanding defects
  • Final integration with other components and systems
  • Finalize trace of all Product Requirements Specifications to System Requirements SpecificationsValidate (and document) the software works as intended
    • Final regression test all Unit Tests and Story Acceptance Tests (note: this should be accomplished normally in the course of each dev iteration)
    • Regression test all Feature Acceptance tests (same comment as above)
    • Regression test all System Qualities tests
    • Perform any exploratory testing
    • Perform all internal acceptance (development, clinical, product marketing) testing
    • Perform any/all user acceptance testing
  • Update quality systems documentation
  • Update, version control and “sign” PRD
  • Update, version control and “sign” SRS
  • Update and finalize all traceability matrices
    • PRD to SRS
    • PRD to Feature Acceptance Tests
    • SRS to Story Acceptance Test
    • SRS to code
    • SRS to unit tests
  • Finalize user documentation
    • Manuals and on line help
    • Release notes and any other user guidance
    • Installation and maintenance

Validation iteration length

Depending on the levels of test automation and other tooling, accomplishing this in a single, two-week iteration may not be practical. In that case, some teams treat this iteration as “fixed scope”, not fixed length, allowing them to take the time necessary to do a proper job. However, there is a negative impact to this as cadence (cadence and synchronization are our primary tools to manage product development variability, [see Reinertsen 2009]) is lost and the schedule becomes unpredictable. Instead, it is generally better to have an additional iteration dedicated to this purpose, or perhaps a longer one than normal. Then, if the teams fail to get the job done, they can always add another iteration for this purpose, and the “failure” will indicate where they need to invest in additional automation and documentation tooling, or address whatever other impediments remain. (After all, completing critical work in a quality fashion in a short timebox, is what good agile teams do.)

All of the activities we have described above are necessary for effective validation, but three are of particular interest to our agile approach:

  • PRD to SRS validation and traceability
  • Validation that all features are tested
  • Validation that the system conforms to its non-functional requirements

These will be the subjects of the next posts in this series.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s