Running Redmine with an Oracle Backend

Well, it took some figuring out, but I have succeeded in getting Redmine to run with an Oracle database. I did have a little help from this post, but there have been a few changes since then to deal with. Once I got all figured out, it really isn’t all that difficult to repeat (it just took a while to get there). (I’m not going to discuss setting up the Oracle adapter here, only Redmine.)

Here goes.
1. Oracle sees ” and “null” as the same thing.
The biggest issue is the fact that Oracle see ” and ‘null’ as the same thing. That said, there are a few setup scripts that will need modified. I went through db/migrate/001_setup.rb and looked for everything that sets a default value of ” and told it to allow nulls for any such column.

For example, the users table is created like this:

create_table “users”, :force => true do |t|
t.column “login”, :string, :limit => 30, :default => “”, :null => true
t.column “hashed_password”, :string, :limit => 40, :default => “”, :null => true
t.column “firstname”, :string, :limit => 30, :default => “”, :null => true
t.column “lastname”, :string, :limit => 30, :default => “”, :null => true
t.column “mail”, :string, :limit => 60, :default => “”, :null => true
t.column “mail_notification”, :boolean, :default => true, :null => false
t.column “admin”, :boolean, :default => false, :null => false
t.column “status”, :integer, :default => 1, :null => false
t.column “last_login_on”, :datetime
t.column “language”, :string, :limit => 2, :default => “”, :null => true
t.column “auth_source_id”, :integer
t.column “created_on”, :timestamp
t.column “updated_on”, :timestamp
end

…and versions:

create_table “versions”, :force => true do |t|
t.column “project_id”, :integer, :default => 0, :null => true
t.column “name”, :string, :limit => 30, :default => “”, :null => true
t.column “description”, :string, :default => “”
t.column “effective_date”, :date, :null => true
t.column “created_on”, :timestamp
t.column “updated_on”, :timestamp
end

I’m simply allowing nulls where before null was no longer accepted. This shouldn’t be a problem, its just a lacking database constraint. I’ll definitely follow up if it does become a problem.

There are a few other places where you’ll have to make similar changes:

  • 074_add_auth_sources_tls.rb
  • 091_change_changesets_revision_to_string.rb
  • 108_add_identity_url_to_users.rb
  • 20091017214336_add_missing_indexes_to_users.rb

2. Version Effective Date:
Oracle won’t like the syntax of 048_allow_null_version_effective_date.rb. I simply removed this file (it appears that this was a later change in Redmine) and made the version effective data column nullable in 001_setup.rb like this:

create_table “versions”, :force => true do |t|
t.column “project_id”, :integer, :default => 0, :null => true
t.column “name”, :string, :limit => 30, :default => “”, :null => true
t.column “description”, :string, :default => “”
t.column “effective_date”, :date, :null => true
t.column “created_on”, :timestamp
t.column “updated_on”, :timestamp
end

3. The UTF-8 Problem
Assuming your Oracle database uses AL32UTF8, you’ll want to do something like this in environment.rb:

ENV['NLS_LANG']=’american_america.AL32UTF8′

4. Oracle 30-character limitation on table names
Oracle limits table names to 30 characters in length. This is a problem in one particular Redmine db migration script: 107_add_open_id_authentication_tables.rb
I changed this file to use smaller table names:

class AddOpenIdAuthenticationTables < ActiveRecord::Migration
def self.up
create_table :open_id_auth_associations, :force => true do |t|
t.integer :issued, :lifetime
t.string :handle, :assoc_type
t.binary :server_url, :secret
end

create_table :open_id_auth_nonces, :force => true do |t|
t.integer :timestamp, :null => false
t.string :server_url, :null => true
t.string :salt, :null => false
end
end

def self.down
drop_table :open_id_authentication_associations
drop_table :open_id_authentication_nonces
end
end

5. A problem with the Activity Tab
There a problem with Oracle CLOB String comparison (as in, you can’t compare a CLOB to a String). This is documented here: http://www.redmine.org/issues/3699
Unfortunately, the Redmine response is always “Oracle is not supported.” Hey, I don’t care for being stuck with Oracle either, but some of us are. Anyway, you can take care of this problem with a simple change to the comparison. Change redmine/app/models/journal.rb:

acts_as_activity_provider :type => ‘issues’,
:permission => :view_issues,
:author_key => :user_id,
:find_options => {:include => [{:issue => :project}, :details, :user],
:conditions => “#{Journal.table_name}.journalized_type = ‘Issue’ AND” +
” (#{JournalDetail.table_name}.prop_key = ‘status_id’ OR length(#{Journal.table_name}.notes) > 0)”}

8. Change the sequences
Before you get moving, you’re probably going to want to fix the table sequences, setting them to start at 1. This is just a matter of preference, but at least for the ISSUES_SEQ, since the unique ID is used to identify the ticket, it makes sense to start with lower numbers. For some reason that I don’t understand (yet), Oracle 11 starts sequences at 10,000. I recommend doing this:

DROP SEQUENCE REDMINEUSER.ISSUES_SEQ;
CREATE SEQUENCE ISSUES_SEQ
START WITH 1
MAXVALUE 9999999999999999999999999999
MINVALUE 1
NOCYCLE
CACHE 20
NOORDER;
Now your tickets will start with lower numbers. As far as other sequences go, it doesn’t matter too much, since we generally don’t view the id columns.

9. Annoyingly Short VARCHAR2 Defaults
The default VARCHAR2 setting is going to be VARCHAR2(255). When it comes to certain fields, such as the project description, this is a little on the short side. I went ahead and modified the column width myself:

alter table PROJECTS modify description VARCHAR2(4000);
commit;

10. Database Trigger Needed to Copy Workflows
When creating a Tracker, Redmine allows a user to “Copy from existing workflow.” (I.e., You can copy the Tracker workflow from an existing Tracker, making the process of adding a new Tracker much more quick.) This creates a possible error when inserting the new workflow into the Workflows table because for some reason (and I’m not yet sure why), no trigger was created on the Oracle database. Without this trigger you will see a “Cannot insert null” error in your production log file if you attempt to copy a workflow. To get around this problem it is easy enough to create a trigger on the Workflows table:


CREATE OR REPLACE TRIGGER workflows_before_insert
BEFORE INSERT
ON WORKFLOWS
FOR EACH ROW
BEGIN
if :new.id is null then
SELECT workflows_seq.nextval INTO :new.ID FROM dual;
end if;
END;
/

11. Database connection
Finally, database.yml will end up looking something like this:

production:
adapter: oracle_enhanced
database:
host:
port: username: redmine_db_user
password: redmine_db_pass

These changes are very important to note should you ever have to upgrade Redmine.

Subversion Authentication Using Unix Accounts

Standard Subversion authentication using plain text user settings and passwords in the passwd and authz files in the Subversion conf directory. This is all fine for a very small installation on a localhost, but it doesn’t work very well in a work environment. Alternatively, SVN can be set up to use LDAP authentication. That’s okay too, but as a lead developer I’d like to have a little more control over the development setup and users. So, even better, its not too difficult to set up Subversion to authenticate using Unix accounts. I say its not too hard because there are some tricky things to contend with. Here goes.

1. Set up your Subversion repository.

I prefer a single repository for all projects, but this is a separate writeup.

2. Install Apache

3. Install Apache Modules

Okay, 1 and 2 were easy. Here’s where it starts to get tricky. We’re going to need a few things (and I’m assuming we’re using Redhat/Fedora/CentOS here). Some of these things may already be installed. Gcc (you probably already have this) and http-devel are both necessary. Http-devel (the development tools for Apache). This is required because we will be building a couple Apache modules. We’re also going to need to install an apache module for Subversion: mod_dav_svn.

  yum install gcc
  yum install http-devel
  yum install mod_dav_svn

4. mod_authnz_external

We need an Apache module for mod_authnz_external. There is no yum installer, so it will have to be downloaded and built (no biggie). Grab the tarball from here Building an Apache module is a little different than building other things… To build an install we use the apxs command (this was installed previously using yum install http-devel). So once the mod-auth-external tarball is downloaded we’ll do something like this:

  tar xvf mod_authnz_eternal-3.2.5.tar.gz
  cd mod_authnz_external-3.2.5
  apxs -c mod_authnz_external.c (build the module)
  apxs -i -a mod_authnz_external.la (install the module)

5. User Groups

Next we’re going to install a tool to handle the authentication part.

This is important: Before we install this tool (pwauth) we need to think about how we want to handle Subversion authentication. Do we want specific users to have access? Do we want to create a specific group with access? As a lead developer in charge of administering Subversion access, I recommend creating a subversion group, and call it something like “svnusers.” This is important because when pwauth is built we will have to provide it with a list of groups and/or users who will be allowed to run it (for security purposes).

  groupadd svnusers

Go ahead and add a user or two to this group.

  useradd -G <user> svnusers

Now take a look at what the group ID of svnusers is:

  id <user>

This command will show all the groups that a user belongs to, and something like 501(svnusers) should be one of them.

6. Download, build and install pwauth

Pwauth is a handy program for authenticating a user against the Linux user db (/etc/shadow) that is designed to be used with mod_authnz_external. Download it and untar it and go to the directory where you untarred it.

In the pwauth directory you will see a config.h file. Open this up and make the following changes:

  #define SERVER_UIDS 501 (or whatever the UID of your svnusers group is)

The SERVER_UIDS setting pwauth that anyone in the svnusers group can be authenticated. The MIN_UNIX_UID setting tells pwauth what the minimum user UID that pwauth can authenticate is. Leave this value out entirely or set it to 0.With this changes save it is safe to compile. From the command line:

  make

After pwauth is built it is time to install it. There is no simple install script, so it is important to read the INSTALL text file that appears in this directory. Everything that you’ll ever need to know about installing pwauth is in this text file. A few important notes are: * pwauth is owned by root so that is has the necessary access to read the shadow file. So wherever you placed the pwauth executible, you’ll have to do something like this:

  chmod o+s pwauth

7. Finally, we need to do some stuff to our Apache configuration (httpd.conf).

These are the modules that make the SVN and pwauth magic happen.

Later in the httpd.conf file we’ll create a VirtualHost specifically for serving up Subversion. Any client will use http:// rather than svn:// to access the Subversion repository. That said, I recommend using Apache Virtual Hosts to use different ports for different purposes (for example, plain old port 80 remains your normal http index and port 3600 can be used for Subversion access).

So in httpd.conf do something like this:

Listen 80
Listen 3600
NameVirtualHost *:80
NameVirtualHost *:3600

<VirtualHost *:80>
    DocumentRoot "/var/www/html"
    ServerName myserver
</VirtualHost>

<VirtualHost *:3600
    DocumentRoot "/"
    ServerName myserver
    <Location />
        DAV svn
        SVNPath /var/svn/repo
        SVNListParentPath on
        AuthType Basic
        AuthNAme "Restricted"
        AuthBasicProvider external
        AuthExternal pwauth
        require valid-user
        require group svnusers
    </Location>
    AddExternalAuth pwauth /usr/local/bin/pwauth
    SetExternalAuthMethod pwauth pipe
</VirtualHost>

Restart Apache (service httpd restart). Now when you want to view, checkout or commit to the Subversion repo you’ll use an ip address something like:

  http://my.host.name:3600

And, of course, you’ll need to enter the username and password of a valid user who belongs to the svnusers group on the Subversion server. To check out a particular subfolder you would use http://my.host.name/project. Notices in the example above I made the SVNPath include the repo (/var/svn/repo). I could have made it simply /var/svn, requiring clients to include the /repo at the end of the URL. This is just a matter of personal preference.

8. One last thing: Back to the repository

That repository that you already created is owned by the user it was created as but it is now accessed via Apache, and we are no longer using the default Subversion authz settings. This means that if a client attempts to modify it (check something in), it will be done as the Apache user, and change are the Apache user doesn’t have read/write access to the repository. So you’ll want to do something like this:

  chown -R apache.apache /var/svn

Notes: There are certainly other ways to handle this configuration, and it may be wise to make a few different groups (for example, svnusers, svnreaders, svndevelopers, svntechwriters and so on).

Subversion Authentication Using Unix Creds

Here‘s a simple way to configure Subversion to authenticate  a user using Unix credentials (through http) with Apache, pwauth and mod_authnz_external. Its not really rocket science, and its not anything that others aren’t already doing (the post is over 3 years old), but it certainly is useful.

I like authenticating Subversion users in this way because it offers a very convenient non-LDAP approach for a development group. It allows me to maintain the ability to add and remove users and access levels from any repository and do so using the same user accounts that are already present within our development environment. This assumes, of course, that users of Subversion have a user account on the Subversion server. There are a few hoops to jump through, such as adding the mod and pwauth, but nothing too severe.

There are some gotchas to be aware of when setting up pwauth, such as compile with the correct user access SERVER_UIDS and MIN_UNIX_UID.

[Blogfish - Apache 2.2 Authentication]