Handling Development in a Medical Device World (Part 8)

To answer the first question, before it is even asked: Yes, a software project should have continuous integration (CI) builds. This goes for projects with a large team as well as projects with a small team. While the CI build is very useful for a large group to collaborate there is tremendous value even for the smallest team. Even a software engineer working alone can benefit from a continuous integration build.

What’s so great about CI builds (using Hudson)?

1. A build can be traced to a Subversion changeset, and therefore to our ticketing system (and
entire process).

Okay, so I’m starting to sound repetitive, but tracing is a good thing, and with this setup I can trace all over the place! With all of the process in place, from our standard operating procedures to work instructions to use cases and requirements to tickets to changests, how do we know that the team members working on a project are actually following procedure? The CI environment, at least with regard to the ongoing development of code, gives us a single point of overview for all other activities. From here we can see changesets, tickets, build status and test coverage. Using the proper add-ons, we can even gain insight into the quality of the code that is being developed.

To pull this of, of course, we need to use our ticketing system wisely. With Redmine, and just about any good ticketing system, we can capture elements of software requirements and software design as parent tickets. These parent tickets have one to many sub-tickets, which themselves can have sub-tickets. A parent ticket may not be closed or marked complete until a child ticket is completed.

2. Immediate feedback

At its most basic level, Hudson only does one thing: It runs whatever scripts we tell it to. The power of Hudson is that we can tell it to run whatever we want, log the outcome, keep build artifacts, run third party evaluation tools and report on results. With Subversion integration, Hudson will display the changeset relevant to a particular build. It can be configured to generate a build at whatever interval we want (nightly, hourly, every time there is a code commit, etc.).

Personally, every time I do any code commit of significance, one of the first things I do is check the CI build for success. If I’ve broken the build I get to work on correcting the problem (and if I cannot correct the problem quickly, I roll my changeset out so that the CI build continues to work until I’ve fixed the issue).

Hudson can be configured to email team members on build results, and it may be useful to set it up so that developers are emailed should a build break.

3.  A central location for builds

The build artifacts of your project, in general, should not be a part of the repository (there are exceptions to this rule). Build artifacts belong in your continue integration build tool, where they can be viewed and used by anyone on the team who needs them. These artifacts include test results, compiled libraries and executables. Too often a released build is created locally on some developer’s machine. This is a serious problem, because we have no good way of knowing what files were actually used to create that build. Was there a configuration change? A bug introduced? An incorrect file version?

While developers have good reason to generate and test build locally, a formal build used for testing (or, more importantly, release) much never be created locally. Never.

Build artifacts are not checked in to the source control repository for a number of reasons, but the biggest reason is because we never want to make assumptions about the environment in which those items were build. The build artifacts should instead remain in our CI environment, where we know the conditions within which the build was generated.

Also, because these builds remain easily accessible and labeled in the CI build environment, any team member can easily access any given build. In particular, it may become necessary to use a specific build to recreate an issue in a build that has been released for internal or external use. Because we know the label of the build (the version number given to it) as well as the repository changeset number of the build (because our build and install scripts include it), we know precisely which build to pull from the CI build server to recreate the necessary conditions.

4. Unit tests are run over and over (and over)

Developers should do whatever they can to keep the CI build from breaking. Of course, this doesn’t always work well. I’ve broken the CI build countless times for a number of reasons:
– I forgot to add a necessary library or new file
– I forgot to commit a configuration change
– I accidentally included a change in my changeset
– It worked locally but not when built on the CI build server (this is why the CI build server should, as much as possible, mimic the production environment)
– Unit tests worked locally but no on the CI build server because of some environmental difference

If not for a CI build environment with good unit tests, such problems would only be discovered at a later time or by some other frustrated team member. In this regard, the CI build saves us from many headaches.
The most difficult software defects to fix (much less, find) are the ones that do not happen consistently. Database locking issues, memory issues and race conditions can result in such defects. These defects are serious, but if we never detect them how can we fix them?

It’s a good idea to have unit tests that go above and beyond what we traditionally think of as “unit tests,” and go several steps further, automating functional testing). This is another one of those areas where team members often (incorrectly) feel that there is not sufficient time to do all the work.

With Hudson running all of these tests ever time a build is performed we sometimes encounter situations where a test fails suddenly for no apparent reason. It worked before, an hour ago, and there has not been any change to the code, so what did it fail this time? Pay attention, because this WILL happen.

5. Integration with other nice-to-haves, such as Findbugs, PMD and Cobertura

Without going into too much detail, there are great tools out there that can be used with Hudson to evaluate the code for potential bugs, bad coding practices and testing coverage. These tools definitely come in handy. Use them.