Valuable Unit Tests in a Software Medical Device, Part 2

While I personally have never been a fan of test-driven development (I feel that the assumptions required by test-driven development do not allow for a true iterative approach), I do believe that creation of unit tests in parallel with development leads to much more quality software. In the world of Agile, this means that no functional requirement (or user story) is considered fully implemented without a corresponding unit test. This strict view of unit tests may be a bit extreme, but it is not without merit.

The first unit test a developer may ever write is likely so simple that it was nearly useless. It may go something like this.

Given a method:

public int doSomething (int a, int b) {

return c;
}

A simple unit test may look something like this:

public class MyUnitTests {

@Test
public void testDoSomething() {
assertEquals(doSomething(1, 2), expectedResult);
}
}

Given a very simple method the developer is able to assert that, essentially, a + b = c. This is easy to write, and there is little overheard involved, but it really isn’t a very useful unit test.

Early Attempts to Automated Functional Testing

Long ago I was involved with a project in which management invested a significant amount of time and training in an attempt to implement automated testing. The chosen tool was Rational Robot ™ (now an IBM product). The idea behind tools such as Robot was that a test creator could record test macros, note points of verification and replay the macros later with test results noted. Tools such as Rational Robot and WinRunner attempted to replace the human tester with record scripts. These automated scripts could be written using a scripting language or, more commonly, by recording mouse movements, clicks and keyboard actions. In this regard, these tools of test automation allowed black-box testing through a user interface.

In this over-simplified view of automated testing, there were simply too many logistical problems with test implementation to make them practical. Any minor changes to the user interface could result in a broken test script. Those responsible for maintaining these automated scripts often found themselves spending more time maintaining the tests than using them for actual application testing.

Rational Robot and tools like it are alive and well, but I refer to them in the past tense because such tools, in my experience, have proven themselves to be a failure. I say this because I have personally spent significant amounts of time creating automated scripts in such tools, and I have been frustrated to learn later that they would not be used because of the substantial amount of interface code that changes as a project progresses. Such changes are absolutely expected, and yet, a recorded automated test does not lend itself well to an iterative development environment or an ongoing project.

Valuable Unit Tests in a Software Medical Device, Part 1

In computer programming, unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual function or procedure. In object-oriented programming a unit is usually a method. Unit tests are created by programmers or occasionally by white box testers during the development process.

-Wikipedia

Note: In the world of Java, we have a number of popular options for the implementation of unit tests, with JUnit and TestNG being, arguably, the most popular. Examples provided in this article will use TestNG syntax and annotations.

Traditionally (and by traditionally, I mean in their relatively brief history), unit tests have been thought of as very simple tests to validate basic inputs and outputs of a software method. While this can be true, and such simple tests can serve of some amount of value, it is possible to achieve much more with unit tests. In fact, it is not only possible, but recommended that we implement much of our user acceptance, functional and possibly even some non-functional tests within a unit test framework.

To further enhance quality, we can augment the acceptance with unit tests. [6]
Dean Leffingwell, Agile Software Requirements

While I personally have never been a fan of test-driven development (I feel that the assumptions required by test-driven development do not allow for a true iterative approach), I do believe that creation of unit tests in parallel with development leads to much more quality software. In the world of Agile, this means that no functional requirement (or user story) is considered fully implemented without a corresponding unit test. This strict view of unit tests may be a bit extreme, but it is not without merit.

  • [1] Device Advice: Regulation and Guidance, Software Validation Guidelines, http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance
  • [2] Safe and Sound Software – Creating an Efficient and Effective Quality System for Software Medical Device Organizations, Thomas H. Farris. ASQ Quality Press, Milwaukee, Wisconsin, 2006, Figure 4.9, “Types of software testing” pg. 120
  • [3] Safe and Sound Software – Creating an Efficient and Effective Quality System for Software Medical Device Organizations, Thomas H. Farris. ASQ Quality Press, Milwaukee, Wisconsin, 2006, pg. 118
  • [4] Safe and Sound Software – Creating an Efficient and Effective Quality System for Software Medical Device Organizations, Thomas H. Farris. ASQ Quality Press, Milwakee, Wisconsin, 2006, Figure 4.9, “Types of software testing” pg. 118
  • [5] CFR – Code of Federal Regulations Title 21. Subpart C – Design Controls, Section 820.30 Design Controls
  • [6] Agile Software Requirements, Dean Leffingwell. Addison-Wesley. Copyright © 2011, Pearson Education, Inc. Boston, MA, page 61
  • [7] Agile Software Requirements, Dean Leffingwell. Addison-Wesley. Copyright © 2011, Pearson Education, Inc. Boston, MA, page 196
  • [8] Continuous Delivery, Jez Humble, David Farley. Addison-Wesley, Copyright © 2011, Pearson Education, Inc. Boston, MA, page 124
  • [9] Safe and Sound Software – Creating an Efficient and Effective Quality System for Software Medical Device Organizations, Thomas H. Farris. ASQ Quality Press, Milwakee, Wisconsin, 2006, Figure 4.9, “Types of software testing” pg. 123

Ticket System as a Trigger for Peer Reviews

Somewhat recently I was thinking about what ticket status might be appropriate when using issue tracking for all tasks from functional requirements to documentation to defect tracking. It got me thinking about the need for peer reviews of code and how tedious these reviews can be. It turns out there is at least one plugin for Trac that includes hooks for annotation of code for the sake of peer review. It does not, however, appear to include any kind of formal sign-off capability.

I started thinking that it would be nice to have a plugin for peer reviews (for Trac or Redmine or whatever). Used wisely, however, defining our workflow in a manner that makes the peer review process an integral part of it, we can probably simplify things. Do we really need a plugin, or can we simply use a “In Review” status to achieve the same thing? I suppose the answer to this depends on how strict you want to be.

Here’s what I’m thinking with regard to the history of a ticket (or issue or task or work item or whatever we choose to call it):

  1. New
  2. In-Progress
  3. Resolved (or, if we determine that a ticket should not be completed, we have alternatives, such as deferred, rejected, duplicate, etc.)
  4. In-Review
  5. Closed

With a setup such as this, we can use the Resolved status as an indicator that an Issue has been completed, but it is not yet ready to be closed. Tickets are only closed when appropriate peer review actions have been taken. Who determines what these actions are? That is up to the project manager (or the team lead), and it is enforced by proper routing of the ticket. Easy individual responsible for peer review is assigned the ticket. Seeing the “In-Review” status, this colleague reviews the code changes (observing the changset that is attached to the ticket) and makes comments (in the ticket notes).

I know this sounds like a bit of legwork, but I see a few major benefits of an approach like this:

  1. Tracing - We now have an audit log of all peer review comments. Using our ticket system with configuration management integration, tickets, changesets and review comments are linked together and not lost in some email thread or document somewhere.
  2. Time Savings – Anyone who has ever sat through a peer review (and I’m guessing most project managers and developers have) knows how insanely time consuming they can be. Because nobody ever seems to have time, we attempt to save time by doing a large review of code; We wait for a long time, and then we are faced with a peer review involving an overwhelming amount of code. This leads to the next benefit…
  3. Better Focus of Reviews - I don’t know about you, but I find that I am much better at reviewing a smaller amount of code or a single functional area than attempting to review thousands of lines of code all at once. We’re all busy, and this isn’t going to change. What happens when you find out that you have a peer review at the end of the week and you have to read through and mark up 5 class files? Do you set aside everything you are working on and do it? You try, but time is short, and so you hurry.
  4. Communication - When I take the time to review a changeset, it benefits both team and the individual performing the review. Now I am better informed about what others are working on, where it is implemented, how it is implemented, etc. I don’t have to go bug Joe the Developer to ask him if he finished such-and-such. I already know that he did because I reviewed his code.

This all assumes that our team follows good project management when it comes to the handling of issue tracking and version control. It means that we have to have well organized tickets and we have to commit changesets in some meaningful fashion. This should be a no-brainer.

Use a Single Version Control System for Everything

Every single artifact related to the creation of your software should be under version control. Developers should use it for source code, of course, but also for tests, database scripts, build and deployment scripts, documentation, libraries and configuration files for your application, your compiler and collection of tools, and so on—so that a new member of your team can start working from scratch.

–Continuous Delivery, Jez Humble, David Farley

I agree, but I’d add to this statement. We don’t keep everything under version control simply to ease the ramp up time for new team members (this is just one of many reasons). An even more important reason to keep everything under version control is because it is imperative that we be able to see all the items related to our project at a given point in time. We need traceability and re-creatability! (I think I just invented a new word.)

In nearly every position I’ve ever been in this question has come up: Should we keep project documentation in the same system as source code? Even in my citation above there seems to be some desire to separate development from documentation (developers do much more than simply write source code). I say to even consider placing project documentation in a separate system is a mistake.

Our project is composed of many items, from code to test to documentation, and all of these items need to reference each other. If our documentation resides in some other system (or in the same system but in a separate repository), it becomes impossible to use a ticketing system to link issues to either documentation or code (or both). Instead, our ticketing system may still be used to write up document needs, but now we have lost the ability to link from a ticket to a changeset that includes non-source code files (think of Redmine or Trac and linking to changesets) .

So what’s the point of keeping documentation in a separate version control or ECM system? I have yet to think of any good reason. I suppose there is a natural tendency to want to separate source code from documentation, but as to why, I have no idea.

I’m not suggesting we should mix documentation throughout source code paths. Naturally, we want to have some reasonable repository/project layout. Here’s what I would do (or something similar):

(Note: In a software medical device environment, there would be DHF and DMR paths in the trunk.) There are many other fine ways to lay out a repository. The point here is simply that it is important to keep everything together in the same version control system. This way we can trace, tag, branch and link issues for a project together.

As you can see in my above example, I’ve used a single repository with all sub-projects below. Others prefer a per-project repository setup. That’s fine, and probably appropriate in many situations, but it doesn’t allow relating issues across projects. I like using a single repository because it becomes much easier to reference related items regardless of project (and, in most issue tracking systems, we will have no problem referencing our project repository using its sub-path).

Running Redmine with an Oracle Backend

Well, it took some figuring out, but I have succeeded in getting Redmine to run with an Oracle database. I did have a little help from this post, but there have been a few changes since then to deal with. Once I got all figured out, it really isn’t all that difficult to repeat (it just took a while to get there). (I’m not going to discuss setting up the Oracle adapter here, only Redmine.)

Here goes.
1. Oracle sees ” and “null” as the same thing.
The biggest issue is the fact that Oracle see ” and ‘null’ as the same thing. That said, there are a few setup scripts that will need modified. I went through db/migrate/001_setup.rb and looked for everything that sets a default value of ” and told it to allow nulls for any such column.

For example, the users table is created like this:

create_table “users”, :force => true do |t|
t.column “login”, :string, :limit => 30, :default => “”, :null => true
t.column “hashed_password”, :string, :limit => 40, :default => “”, :null => true
t.column “firstname”, :string, :limit => 30, :default => “”, :null => true
t.column “lastname”, :string, :limit => 30, :default => “”, :null => true
t.column “mail”, :string, :limit => 60, :default => “”, :null => true
t.column “mail_notification”, :boolean, :default => true, :null => false
t.column “admin”, :boolean, :default => false, :null => false
t.column “status”, :integer, :default => 1, :null => false
t.column “last_login_on”, :datetime
t.column “language”, :string, :limit => 2, :default => “”, :null => true
t.column “auth_source_id”, :integer
t.column “created_on”, :timestamp
t.column “updated_on”, :timestamp
end

…and versions:

create_table “versions”, :force => true do |t|
t.column “project_id”, :integer, :default => 0, :null => true
t.column “name”, :string, :limit => 30, :default => “”, :null => true
t.column “description”, :string, :default => “”
t.column “effective_date”, :date, :null => true
t.column “created_on”, :timestamp
t.column “updated_on”, :timestamp
end

I’m simply allowing nulls where before null was no longer accepted. This shouldn’t be a problem, its just a lacking database constraint. I’ll definitely follow up if it does become a problem.

There are a few other places where you’ll have to make similar changes:

  • 074_add_auth_sources_tls.rb
  • 091_change_changesets_revision_to_string.rb
  • 108_add_identity_url_to_users.rb
  • 20091017214336_add_missing_indexes_to_users.rb

2. Version Effective Date:
Oracle won’t like the syntax of 048_allow_null_version_effective_date.rb. I simply removed this file (it appears that this was a later change in Redmine) and made the version effective data column nullable in 001_setup.rb like this:

create_table “versions”, :force => true do |t|
t.column “project_id”, :integer, :default => 0, :null => true
t.column “name”, :string, :limit => 30, :default => “”, :null => true
t.column “description”, :string, :default => “”
t.column “effective_date”, :date, :null => true
t.column “created_on”, :timestamp
t.column “updated_on”, :timestamp
end

3. The UTF-8 Problem
Assuming your Oracle database uses AL32UTF8, you’ll want to do something like this in environment.rb:

ENV['NLS_LANG']=’american_america.AL32UTF8′

4. Oracle 30-character limitation on table names
Oracle limits table names to 30 characters in length. This is a problem in one particular Redmine db migration script: 107_add_open_id_authentication_tables.rb
I changed this file to use smaller table names:

class AddOpenIdAuthenticationTables < ActiveRecord::Migration
def self.up
create_table :open_id_auth_associations, :force => true do |t|
t.integer :issued, :lifetime
t.string :handle, :assoc_type
t.binary :server_url, :secret
end

create_table :open_id_auth_nonces, :force => true do |t|
t.integer :timestamp, :null => false
t.string :server_url, :null => true
t.string :salt, :null => false
end
end

def self.down
drop_table :open_id_authentication_associations
drop_table :open_id_authentication_nonces
end
end

5. A problem with the Activity Tab
There a problem with Oracle CLOB String comparison (as in, you can’t compare a CLOB to a String). This is documented here: http://www.redmine.org/issues/3699
Unfortunately, the Redmine response is always “Oracle is not supported.” Hey, I don’t care for being stuck with Oracle either, but some of us are. Anyway, you can take care of this problem with a simple change to the comparison. Change redmine/app/models/journal.rb:

acts_as_activity_provider :type => ‘issues’,
:permission => :view_issues,
:author_key => :user_id,
:find_options => {:include => [{:issue => :project}, :details, :user],
:conditions => “#{Journal.table_name}.journalized_type = ‘Issue’ AND” +
” (#{JournalDetail.table_name}.prop_key = ‘status_id’ OR length(#{Journal.table_name}.notes) > 0)”}

8. Change the sequences
Before you get moving, you’re probably going to want to fix the table sequences, setting them to start at 1. This is just a matter of preference, but at least for the ISSUES_SEQ, since the unique ID is used to identify the ticket, it makes sense to start with lower numbers. For some reason that I don’t understand (yet), Oracle 11 starts sequences at 10,000. I recommend doing this:

DROP SEQUENCE REDMINEUSER.ISSUES_SEQ;
CREATE SEQUENCE ISSUES_SEQ
START WITH 1
MAXVALUE 9999999999999999999999999999
MINVALUE 1
NOCYCLE
CACHE 20
NOORDER;
Now your tickets will start with lower numbers. As far as other sequences go, it doesn’t matter too much, since we generally don’t view the id columns.

9. Annoyingly Short VARCHAR2 Defaults
The default VARCHAR2 setting is going to be VARCHAR2(255). When it comes to certain fields, such as the project description, this is a little on the short side. I went ahead and modified the column width myself:

alter table PROJECTS modify description VARCHAR2(4000);
commit;

10. Database Trigger Needed to Copy Workflows
When creating a Tracker, Redmine allows a user to “Copy from existing workflow.” (I.e., You can copy the Tracker workflow from an existing Tracker, making the process of adding a new Tracker much more quick.) This creates a possible error when inserting the new workflow into the Workflows table because for some reason (and I’m not yet sure why), no trigger was created on the Oracle database. Without this trigger you will see a “Cannot insert null” error in your production log file if you attempt to copy a workflow. To get around this problem it is easy enough to create a trigger on the Workflows table:


CREATE OR REPLACE TRIGGER workflows_before_insert
BEFORE INSERT
ON WORKFLOWS
FOR EACH ROW
BEGIN
if :new.id is null then
SELECT workflows_seq.nextval INTO :new.ID FROM dual;
end if;
END;
/

11. Database connection
Finally, database.yml will end up looking something like this:

production:
adapter: oracle_enhanced
database:
host:
port: username: redmine_db_user
password: redmine_db_pass

These changes are very important to note should you ever have to upgrade Redmine.

Subversion Authentication Using Unix Accounts

Standard Subversion authentication using plain text user settings and passwords in the passwd and authz files in the Subversion conf directory. This is all fine for a very small installation on a localhost, but it doesn’t work very well in a work environment. Alternatively, SVN can be set up to use LDAP authentication. That’s okay too, but as a lead developer I’d like to have a little more control over the development setup and users. So, even better, its not too difficult to set up Subversion to authenticate using Unix accounts. I say its not too hard because there are some tricky things to contend with. Here goes.

1. Set up your Subversion repository.

I prefer a single repository for all projects, but this is a separate writeup.

2. Install Apache

3. Install Apache Modules

Okay, 1 and 2 were easy. Here’s where it starts to get tricky. We’re going to need a few things (and I’m assuming we’re using Redhat/Fedora/CentOS here). Some of these things may already be installed. Gcc (you probably already have this) and http-devel are both necessary. Http-devel (the development tools for Apache). This is required because we will be building a couple Apache modules. We’re also going to need to install an apache module for Subversion: mod_dav_svn.

  yum install gcc
  yum install http-devel
  yum install mod_dav_svn

4. mod_authnz_external

We need an Apache module for mod_authnz_external. There is no yum installer, so it will have to be downloaded and built (no biggie). Grab the tarball from here Building an Apache module is a little different than building other things… To build an install we use the apxs command (this was installed previously using yum install http-devel). So once the mod-auth-external tarball is downloaded we’ll do something like this:

  tar xvf mod_authnz_eternal-3.2.5.tar.gz
  cd mod_authnz_external-3.2.5
  apxs -c mod_authnz_external.c (build the module)
  apxs -i -a mod_authnz_external.la (install the module)

5. User Groups

Next we’re going to install a tool to handle the authentication part.

This is important: Before we install this tool (pwauth) we need to think about how we want to handle Subversion authentication. Do we want specific users to have access? Do we want to create a specific group with access? As a lead developer in charge of administering Subversion access, I recommend creating a subversion group, and call it something like “svnusers.” This is important because when pwauth is built we will have to provide it with a list of groups and/or users who will be allowed to run it (for security purposes).

  groupadd svnusers

Go ahead and add a user or two to this group.

  useradd -G <user> svnusers

Now take a look at what the group ID of svnusers is:

  id <user>

This command will show all the groups that a user belongs to, and something like 501(svnusers) should be one of them.

6. Download, build and install pwauth

Pwauth is a handy program for authenticating a user against the Linux user db (/etc/shadow) that is designed to be used with mod_authnz_external. Download it and untar it and go to the directory where you untarred it.

In the pwauth directory you will see a config.h file. Open this up and make the following changes:

  #define SERVER_UIDS 501 (or whatever the UID of your svnusers group is)

The SERVER_UIDS setting pwauth that anyone in the svnusers group can be authenticated. The MIN_UNIX_UID setting tells pwauth what the minimum user UID that pwauth can authenticate is. Leave this value out entirely or set it to 0.With this changes save it is safe to compile. From the command line:

  make

After pwauth is built it is time to install it. There is no simple install script, so it is important to read the INSTALL text file that appears in this directory. Everything that you’ll ever need to know about installing pwauth is in this text file. A few important notes are: * pwauth is owned by root so that is has the necessary access to read the shadow file. So wherever you placed the pwauth executible, you’ll have to do something like this:

  chmod o+s pwauth

7. Finally, we need to do some stuff to our Apache configuration (httpd.conf).

These are the modules that make the SVN and pwauth magic happen.

Later in the httpd.conf file we’ll create a VirtualHost specifically for serving up Subversion. Any client will use http:// rather than svn:// to access the Subversion repository. That said, I recommend using Apache Virtual Hosts to use different ports for different purposes (for example, plain old port 80 remains your normal http index and port 3600 can be used for Subversion access).

So in httpd.conf do something like this:

Listen 80
Listen 3600
NameVirtualHost *:80
NameVirtualHost *:3600

<VirtualHost *:80>
    DocumentRoot "/var/www/html"
    ServerName myserver
</VirtualHost>

<VirtualHost *:3600
    DocumentRoot "/"
    ServerName myserver
    <Location />
        DAV svn
        SVNPath /var/svn/repo
        SVNListParentPath on
        AuthType Basic
        AuthNAme "Restricted"
        AuthBasicProvider external
        AuthExternal pwauth
        require valid-user
        require group svnusers
    </Location>
    AddExternalAuth pwauth /usr/local/bin/pwauth
    SetExternalAuthMethod pwauth pipe
</VirtualHost>

Restart Apache (service httpd restart). Now when you want to view, checkout or commit to the Subversion repo you’ll use an ip address something like:

  http://my.host.name:3600

And, of course, you’ll need to enter the username and password of a valid user who belongs to the svnusers group on the Subversion server. To check out a particular subfolder you would use http://my.host.name/project. Notices in the example above I made the SVNPath include the repo (/var/svn/repo). I could have made it simply /var/svn, requiring clients to include the /repo at the end of the URL. This is just a matter of personal preference.

8. One last thing: Back to the repository

That repository that you already created is owned by the user it was created as but it is now accessed via Apache, and we are no longer using the default Subversion authz settings. This means that if a client attempts to modify it (check something in), it will be done as the Apache user, and change are the Apache user doesn’t have read/write access to the repository. So you’ll want to do something like this:

  chown -R apache.apache /var/svn

Notes: There are certainly other ways to handle this configuration, and it may be wise to make a few different groups (for example, svnusers, svnreaders, svndevelopers, svntechwriters and so on).

Subversion Authentication Using Unix Creds

Here‘s a simple way to configure Subversion to authenticate  a user using Unix credentials (through http) with Apache, pwauth and mod_authnz_external. Its not really rocket science, and its not anything that others aren’t already doing (the post is over 3 years old), but it certainly is useful.

I like authenticating Subversion users in this way because it offers a very convenient non-LDAP approach for a development group. It allows me to maintain the ability to add and remove users and access levels from any repository and do so using the same user accounts that are already present within our development environment. This assumes, of course, that users of Subversion have a user account on the Subversion server. There are a few hoops to jump through, such as adding the mod and pwauth, but nothing too severe.

There are some gotchas to be aware of when setting up pwauth, such as compile with the correct user access SERVER_UIDS and MIN_UNIX_UID.

[Blogfish - Apache 2.2 Authentication]

Redmine Forked (ChiliProject)

Oh boy. Redmine has been forked now? There are many, many comments here, which I’ll be reading through. I’m a fan of Redmine, and I fully intend to continue using it.

There are certainly a number of alternatives out there, but the appeal of Redmine is that it is widely used, and as such, all the plugins you could ever ask for have already been created. I also like its high level of configuration and the fact that it works well for multiple projects (unlike Trac, which I loved until the day I saw Redmine).

[Redmine - Statement about the ChiliProject fork? - Redmine]

Handling Development in a Medical Device World (Part 8)

CONTINUOUS INTEGRATION BUILDS
To answer the first question, before it is even asked: Yes, a software project should have continuous integration (CI) builds. This goes for projects with a large team as well as projects with a small team. While the CI build is very useful for a large group to collaborate there is tremendous value even for the smallest team. Even a software engineer working alone can benefit from a continuous integration build.

What’s so great about CI builds (using Hudson)?

1. A build can be traced to a Subversion changeset, and therefore to our ticketing system (and
entire process).

Okay, so I’m starting to sound repetitive, but tracing is a good thing, and with this setup I can trace all over the place! With all of the process in place, from our standard operating procedures to work instructions to use cases and requirements to tickets to changests, how do we know that the team members working on a project are actually following procedure? The CI environment, at least with regard to the ongoing development of code, gives us a single point of overview for all other activities. From here we can see changesets, tickets, build status and test coverage. Using the proper add-ons, we can even gain insight into the quality of the code that is being developed.

To pull this of, of course, we need to use our ticketing system wisely. With Redmine, and just about any good ticketing system, we can capture elements of software requirements and software design as parent tickets. These parent tickets have one to many sub-tickets, which themselves can have sub-tickets. A parent ticket may not be closed or marked complete until a child ticket is completed.

2. Immediate feedback

At its most basic level, Hudson only does one thing: It runs whatever scripts we tell it to. The power of Hudson is that we can tell it to run whatever we want, log the outcome, keep build artifacts, run third party evaluation tools and report on results. With Subversion integration, Hudson will display the changeset relevant to a particular build. It can be configured to generate a build at whatever interval we want (nightly, hourly, every time there is a code commit, etc.).

Personally, every time I do any code commit of significance, one of the first things I do is check the CI build for success. If I’ve broken the build I get to work on correcting the problem (and if I cannot correct the problem quickly, I roll my changeset out so that the CI build continues to work until I’ve fixed the issue).

Hudson can be configured to email team members on build results, and it may be useful to set it up so that developers are emailed should a build break.

3.  A central location for builds

The build artifacts of your project, in general, should not be a part of the repository (there are exceptions to this rule). Build artifacts belong in your continue integration build tool, where they can be viewed and used by anyone on the team who needs them. These artifacts include test results, compiled libraries and executables. Too often a released build is created locally on some developer’s machine. This is a serious problem, because we have no good way of knowing what files were actually used to create that build. Was there a configuration change? A bug introduced? An incorrect file version?

While developers have good reason to generate and test build locally, a formal build used for testing (or, more importantly, release) much never be created locally. Never.

Build artifacts are not checked in to the source control repository for a number of reasons, but the biggest reason is because we never want to make assumptions about the environment in which those items were build. The build artifacts should instead remain in our CI environment, where we know the conditions within which the build was generated.

Also, because these builds remain easily accessible and labeled in the CI build environment, any team member can easily access any given build. In particular, it may become necessary to use a specific build to recreate an issue in a build that has been released for internal or external use. Because we know the label of the build (the version number given to it) as well as the repository changeset number of the build (because our build and install scripts include it), we know precisely which build to pull from the CI build server to recreate the necessary conditions.

4. Unit tests are run over and over (and over)

Developers should do whatever they can to keep the CI build from breaking. Of course, this doesn’t always work well. I’ve broken the CI build countless times for a number of reasons:
- I forgot to add a necessary library or new file
- I forgot to commit a configuration change
- I accidentally included a change in my changeset
- It worked locally but not when built on the CI build server (this is why the CI build server should, as much as possible, mimic the production environment)
- Unit tests worked locally but no on the CI build server because of some environmental difference

If not for a CI build environment with good unit tests, such problems would only be discovered at a later time or by some other frustrated team member. In this regard, the CI build saves us from many headaches.
The most difficult software defects to fix (much less, find) are the ones that do not happen consistently. Database locking issues, memory issues and race conditions can result in such defects. These defects are serious, but if we never detect them how can we fix them?

It’s a good idea to have unit tests that go above and beyond what we traditionally think of as “unit tests,” and go several steps further, automating functional testing). This is another one of those areas where team members often (incorrectly) feel that there is not sufficient time to do all the work.

With Hudson running all of these tests ever time a build is performed we sometimes encounter situations where a test fails suddenly for no apparent reason. It worked before, an hour ago, and there has not been any change to the code, so what did it fail this time? Pay attention, because this WILL happen.

5. Integration with other nice-to-haves, such as Findbugs, PMD and Cobertura

Without going into too much detail, there are great tools out there that can be used with Hudson to evaluate the code for potential bugs, bad coding practices and testing coverage. These tools definitely come in handy. Use them.

[Hudson]

[TestNG]

Handling Development in a Medical Device World (Part 7)

USE THE ISSUE TRACKING SYSTEM FOR ALL PROJECT DESIGN AND DEVELOPMENT
I propose that it is not enough to simply leave functional requirements in the software requirements specification document. This does not provide sufficient tracing, nor does it provide a clear path from idea to functional code. Here are the steps that I suggest:

  1. All requirements and software design items are entered as tickets. For now they are simply high level tickets with no “child tickets.”
  2. The development team, organized by a lead developer, breaks down each high level ticket into as many child tickets as necessary. Using the ticketing system, we set up relationships so that the parent ticket (the requirement itself) cannot be closed until all child tickets are completed. (Note: It may be a good idea to require corresponding unit tests with each ticket.)
  3. Hazards (and I’m not doing to bother an explanation of hazard and risk analysis in this article) are mitigated by a combination of documentation, requirements and tests. We can leverage our ticketing system to capture our hazards and provide tracing in much the same way as with requirements. This does not remove the need for a traceability matrix, but it does enhance our ability to create and maintain it. (As a side note, I think it would be great to use the Redmine wiki for use cases, requirements, hazard analysis, software design documents and traceability matrices, thereby allowing for linking within, but this may be a hard sell for now).
  4. Not all requirements are functional code requirements. Many are documentation and/or quality requirements. These should be capture in the same ticketing system. Use the ability of the system to label the type of a ticket to handle the fact that there are different categories. By doing this, even documentation requirements are traceable in our system.
  5. I’m not suggesting that the tickets will be locked down this early. Not for a moment! Tickets are created, closed and modified throughout project design and development process. Our project plan (created before we started writing code) explains to us which tickets need to be done when, focusing more on the highest level tickets. That said, I find it best to use some sort of iterative approach (and allow the development team to use sub-iterations, or “sprints”).

Handling Development in a Medical Device World (Part 5)

RELEASING SOFTWARE
When software is released it is typically given some kind of version number (e.g., 1.0). This is good, but it doesn’t tell us the specifics of what went into that build. It’s a good idea to include the Subversion changeset number somewhere in the release so that we always know EXACTLY what went into the build. I would include a build.xml (or build.prop) file somewhere that includes the version number of the release, the Subversion changeset number and the date of the build. As far as the last two values, these can (and should) be generated automatically by your build scripts.

As far as actually using Subversion, within Linux/Unix all commands are available from the command line. When working in Windows, I really like using TortoiseSVN. It integrates with Windows File Explorer, showing icons that indicate the status of any versioned file. It also provides a nice interface for viewing file differences (even differences of your Word documents!) and repository history.

TICKETING AND ISSUE TRACKING
For too long we’ve thought of our ticketing systems as “bug trackers.” One of the previously most popular open source tools, Bugzilla, even used the word “bug” in its name. But issue tracking does not need to imply that it is only useful for tracking software defects. On the contrary! It can be used for everything in software design and development, from addressing documentation needs to capturing software requirements to handling software defect reporting.

(On this note, I would like to add something that may require an entire post. I think it might be best to get away from using standard documents for the capture of software use cases, requirements, hazards and so on. By capturing everything related to a software project in our issue tracking tool we can leverage the power of a tool like Trac or Redmine to enhance team collaboration and project tracing. But I won’t bite off more than I can chew right now.)

When I first began writing all of these thoughts of mine down I was going to use Trac as my example (http://trac.edgewall.org/). Trac is a great tool, but there’s something better now, and that something is Redmine (http://www.redmine.org/).

The principal shortfall of Trac is the fact that it doesn’t lend itself well (at all) to handling multiple projects. One installation of Trac can be integrated with only single Subversion repository, and the ticketing system can only handle a single project. I still like the way Trac can be used to group tickets into sprints, but using subprojects in Redmine, a similar grouping can be achieved. In the past, if a tool was used at all, we used Bugzilla or Clearquest to handle our issue tracking. These tools were very good at the time, but they did not integrate well with other tools, nor did they include features like wikis, calendars or Gantt charts. (Admittedly, I have not used Clearquest in many years, so I really have no idea whether or not it has since addressed some of these needs.)

Handling Development in a Medical Device World (Part 4)

VERSION CONTROL
The earliest phase of any software project is the planning phase. At this stage, people involved with the project have meetings and discuss some very high level needs. There are probably some presentations and documents that are created. Project management plans have not been developed, but they should be thought about. And as I stated previously, we begin creation of our work instructions (the application of our SOPs) in this stage.

The design history file (DHF) of a project must contain all of the historical “stuff” that goes into the project, so even at this early stage it is necessary to decided on a version control system and create a repository. Sure, there may be no tracing involved yet, but this early “stuff” should still be kept around In the DHF. Because the earliest phases of the software project result in outputs that are to be included in the DHF, it is necessary determine the version control tool and establish the version control repository early on (for the sake of this article, I assume a basic understanding of version control/revision control systems).

PROJECT TRACING
Once again: Tracing. Tracing is everything, and Subversion, with its changesets, lends itself extremely well to integration with other tools used throughout the project. When used with your issue tracking software every issue can be linked directly with a set of items in the repository that are related to addressing and resolving that issue. With a click of the mouse we can see a list of all the project file modifications related to a single issue.

I recommend using a single version control system and repository for all of the “stuff” that goes into a project (there are many good reasons to use a repository-per-project approach as well). This means that project management plans, documents, presentations, code, test data and results should all go into the same repository, and the repository itself is laid out so that each project has its own trunk, tags and branches. If documents are stored in a separate repository (or in a different version control system altogether) and software code is stored in a different repository, we lose much of the project traceability that we could have.

(Note: When placing binaries in a version control system there is no merge path as there is with text file source code. This means it is good practice for team members, when editing documents, to place a strict lock on the file while editing. This can be done in Subversion. Strict file locking allows others to be notified that another user is currently working on a file.)

While a clear benefit of this approach is the fact that all of the “stuff” of a project is associated with the same repository, some may view this as a problem with the setup. I suggest this setup only because I am thinking specifically in terms of an FDA regulated software product in which it is beneficial to relate all elements of the project in a single traceable repository. In this setup documentation will be versioned (and tagged) along with project source code, and this may or may not be desirable depending on project needs.

Subversion is superior to many of its predecessors because of (among other things) its introduction to “changesets.” A changeset provides a snapshot in time of the entire project repository. When documents, presentations or source code are changed and committed to the repository, a new changeset number is created. Now at any time in the future we can checkout all items in the repository tree as of that changeset. When asked what version of a product something was changed in we can pull everything relevant to the project at the point of that change. No longer do we have to tag or label our repository in order to revisit a particular instance in time (although Subversion still allows tagging). Every single commit to the repository effectively results in a “tag.”

This is not to say that tagging is no longer useful. On the contrary, it remains very useful. All software releases, included internal releases, should be tagged (and our work instructions should tell us when and how to perform tagging).

Another advantage of Subversion is that, unlike some of its predecessors, it allows for the control and history of directories as well as files (including file and directory name changes). The most commonly used predecessor to Subversion, CVS, did not maintain a version history of a file or directory was renamed. Subversion can handle the renaming of any version controlled object.

Handling Development in a Medical Device World (Part 3)

ONE APPROACH
After that long introduction, my desire is to write about one approach to handling design and development activities of a software project. This is an approach that works well for any software project, not just one in which a 510K is being pursued. Given the fact that the biggest needs are for good communication and tracing, perhaps we shouldn’t be surprised that the open source community has taught us a thing or two about team collaboration and communication on software project. It is this community that has learned a great deal about pulling together work from a community across the global to create high quality software.

When it comes to a regulated project we are very concerned with traceability. We want to know that use cases, business rules, requirements, documents, code revisions, hazards and tests can be traced backward and forward. Despite having some excellent technology available (for free!), many companies insist on manual documentation of such tracing. This creates a mess of documentation that is difficult (if not impossible) to maintain and likely to contain errors. You’re just begging an auditor to find a problem! If it is your full time job to edit and verify these documents, you may want to stop reading now.

Here goes: You don’t need to hire someone to work full time to shuffle through this mess of documentation! Sure, you may need a technical writer, but there is plenty of technology available to make this aspect of design and development easy. There are tools out there already that, when used properly, can make software project management from the highest level to the most detailed tracing downright fun.

WHY ISN’T EVERYONE DOING IT THIS WAY THEN?

(Well, in fact, many are…)

  1. There is a perception that using 3rd party tools is difficult within the confines of the CFR.
  2. “But how do we validate all this software?” you ask, “The CFR requires us to validate all the tools we use.” Show me where. What the CFR states is that we must show intended usage of our software and show that the tools we use work the way we think they should work for our needs. Simple.
  3. The CFR doesn’t exist so that we may engineer software poorly. Nor does it exist to tie our hands from using the best new tools available. It exists to tell us how to do it right.
  4. Open source software cannot be trusted.
  5. This objection can only be addressed in a separate article! All I can say to this is that we have learned repeatedly that widely-used open source software is often better and just as well supported as its closed-source counterpart. One need only look as far as GNU for proof (www.gnu.org).
  6. Technologies change. How do I know Subversion will be used in 10 years?
  7. You don’t, and it may not be. But for everyone out there using other legacy systems (Visual Sourcesafe anyone?), we can’t let this be a concern. The technology may change, but our servers can be maintained and archived for as long as necessary.
  8. We don’t have the time for such overhead or IT support.

At the risk of sounding cliché, you don’t have time to not do this. A little setup and thought up front streamlines the process and mitigates the risk of serious problems down the road. If you ever find yourself wondering how to update tracing long after the completion of a software requirement, document change or test, your procedures have failed you. And if you don’t have a practical approach to applying your procedures, this WILL happen at some point in the project. Sure, there is some upfront cost to be considered, but it comes with greater efficiency throughout the project lifecycle.
THE TOOLS

When it comes to a setup such as the one I am about to describe, there is no “one size fits all” approach. The tools used must be evaluated with consideration to the environment, project and corporate needs. So while the tools I am about to list will no doubt work for a wide range of needs, there are many other tools that are worthy alternatives (Trac, Git, Mercurial, etc.).
The main tools in my example are:

  • Subversion – http://subversion.tigris.org/ – Version control (of everything!)
  • Redmine – http://www.redmine.org/ – Ticketing and issue tracking system
  • Hudson – http://hudson-ci.org/ – Continuous Integration builds
  • Other tools that I will mention include:
  • TortoiseSVN – http://tortoisesvn.tigris.org/
  • PMD – http://pmd.sourceforge.net/
  • Cobertura – http://cobertura.sourceforge.net/
  • Findbugs – http://findbugs.sourceforge.net/

The tools I’ve listed above are ones that I am very familiar with and like. They serve the needs of a software project (regulated or non-regulated) very well when integrated, they are widely used and supported and they integrate very nicely. Best of all, being open-source, they are all free (and written by some of the best software developers in the world). For use in the typical corporate environment, each tool can use LDAP authentication. As far as environment restrictions go, each of the tools can be run on Windows, Linux, Solaris and MAC OS X.