/ Linux

Updating Gitlab from a non-Omnibus to an Omnibus Installation

Recently Gitlab announced that there are some serious security issues in their code. As I am responsible for the Gtilab instance in our group, I decided to devote some time of my weekend to update Gitlab.

Context

We were running a VM with Debian 7 stable with manually installed Gitlab version 7.1.0. Fortunately, my colleague has installed Gitlab with Postgresql so the upgrade should be easy–at least in theory. We have a Gitlab installation with 74 projects, 45 users, and several groups. We are connected to an LDAP server that contains information about all users at the faculty. Most of the hosted git projects are active and receive regular commits.

Update plan

I have prepared a plan for updating the Gitlab installation.

First, I backup everything, including a snapshot of the virtual machine and the backup offered by Gitlab using the commands:

sudo gitlab-rake gitlab:backup:create

or

bundle exec rake gitlab:backup:create RAILS_ENV=production

for older versions. Second, I clone the production environment to a new VM and conduct the update there. Third, I download the oldest ever Omnibus package (version 7.10) and follow the instructions from the manual Upgrading Gitlab from a non-Omnibus installation to an Omnibus installation. Finally, I use sudo aptitude upgrade and so the work should be ready. At least so was the plan.

Let's start!

The test-update in the VM went very well, but it would be too easy if it runs without problems in reality. Here is the procedure that I followed in the production environment.

First let's install some dependencies. Some of them are mentioned on the official Gitlab page, and here are mine:

sudo apt-get install curl openssh-server ca-certificates postfix postgresql-contrib

Stop Gitlab and its relevant services:

sudo service gitlab stop
sudo update-rc.d gitlab disable
sudo service nginx stop
sudo update-rc.d nginx disable
sudo service redis-server stop
sudo update-rc.d redis-server disable

Create and edit the config file for new Gitlab:

mkdir /etc/gitlab
vim /etc/gitlab/gitlab.rb

And insert the relevant blocks updating the values that relate to your installation:

# Use your own GitLab URL here
external_url 'http://git.example.com'
# We assume your repositories are in /home/git/repositories (default for source installs)
git_data_dir '/home/git'
# Re-use the Postgres that is already running on your system
postgresql['enable'] = false
# This db_host setting is for Debian Postgres packages
gitlab_rails['db_host'] = '/var/run/postgresql/'
gitlab_rails['db_port'] = 5432
# We assume you called the GitLab DB user 'git'
gitlab_rails['db_username'] = 'git'

This config is important and you need to focus on it a little bit more. There might be some options that are relevant for your installation but were not for mine. Here is template of gitlab.rb config file with all options that you may need.

Now, we need to change one parameter in the postgresql database:

sudo -u postgres psql -d gitlabhq_production -c "CREATE EXTENSION pg_trgm;"

In case you use a big LDAP instance, prepare the user and the group separately as it may take longer (~2 minutes in my case):

groupadd gitlab-www
useradd -g gitlab-www -s /bin/false -d /var/opt/gitlab/nginx -r gitlab-www

The system is prepared for Omnibus packet installation. Add the repositories, install the oldest Omnibus-based Gitlab, reconfigure and run the package:

# ADD repository
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
# Install the oldest version of the omnibus package
apt-get install gitlab-ce=7.10.0~omnibus-1
sudo gitlab-ctl reconfigure
sudo gitlab-rake gitlab:shell:setup

I followed SSL configuration of nginx from here: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md It was enough to rename/copy the key and certificate to get it working.

Make sure that the current installation runs. If yes, you can upgrade Gitlab from 7.10.0 to the current version (in my case 8.7.5).

aptitude upgrade
sudo gitlab-ctl restart

In case you cannot login via LDAP, check the configuration using this manual: http://docs.gitlab.com/ee/administration/auth/ldap.html .

Problems

In theory everything should be fine now. But you may run into the following problems.

After updating Gitlab, my repositories are empty and git push returns error broken hooks

The organization of hooks has changed. You need to adapt the files in the repository. Note that I store git repositories in non-standard directory /data/gitlab/.

# do backup
find /data/gitlab/repositories -name \*.git -type d -exec sh -c 'cp -r $0/hooks $0/hooks.bak' {} \;
# Delete old hooks
find /data/gitlab/repositories -name \*.git -type d -exec sh -c 'rm -r $0/hooks' {} \;
# Link hooks to the common hook directory
find /data/gitlab/repositories -name \*.git -type d -exec sh -c 'ln -s /opt/gitlab/embedded/service/gitlab-shell/hooks $0/hooks' {} \;
# Fix permissions
find /data/gitlab/repositories -name \*.git -type d -exec sh -c 'chown -R git:git $0/hooks $0/hooks.bak' {} \;
find /data/gitlab/repositories -name \*.git -type d -exec sh -c 'chmod a+rwx $0/hooks' {} \;
# Check: 
sudo -u git -H gitlab-rake gitlab:check RAILS_ENV=production

After updating Gitlab some repos are empty

In case you see empty repositories in the web interface you don't need to worry, you data is safe! You may click "Create empty bare repository" and the old content will appear.

After updating Gitlab, I see the files but no commits

Your commits are also safe, just the counter shows zero. It will get back to normal after you push your next commit to the repo.

After updating Gitlab, I cannot push commits to branches

First solution is to reinstall gitlab:

sudo aptitude reinstall gitlab-ce

Your data will be safe.

The second solution might be:

chown -R git:git /opt/gitlab/embedded/service/gitlab-shell/hooks/
chown git /opt/gitlab/embedded/service/gitlab-rails/db/schema.rb
chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public
but I have not tested the second solution.

Do not forget to document your solution for the future and share you ideas in the comments!

Resources

Piotr

Piotr

IT specialist with 7 years of experience in research at four European universities and in private sector. I believe that good quality is always worth to invest few € and some free time.

Read More