What is this?

DCOR uses virtualbox images created with Vagrant for testing. This repository contains the recipe for creating the dcor-test vagrant boxes. The default test user is "peterpan:peterpan".

Why virtualbox/vagrant?

There is a docker recipe for CKAN and I could probably also have adapted that to DCOR, but that looked like a lot of code to maintain. The upside of virtualbox images is that you can take your time to set it up (which is very convenient if you use vagrant) and then just distribute the final box image for testing.

By default, the Vagrantfile gives access to the DCOR web interface (http://127.0.0.1:8888) as well as the SOLR search interface (http://127.0.0.1:8983)

How does testing work?

Let's use the ckanext-dcor_schemas repository as an example. If you clone into it, you will have the files Vagrantfile (the recipe for setting up the box) and vagrant-run-tests.sh (that installs the current working tree into the CKAN environment and executes the tests). When you run vagrant up in this directory, vagrant uses Vagrantfile to setup the virtualbox image (download and rsync the current working directory of the host to /testing on the guest). To run the tests, simply run:

vagrant ssh -- sudo bash /testing/vagrant-run-tests.sh

Basic box creation workflow

This is only necessary if you want to create an image from scratch (from an existing ubuntu box). If you only have to do something incrementally, you can just base your box on the existing dcor-test images.

Install prerequisites:

apt install virtualbox vagrant

Download and setup the base image:

cd from_scratch
vagrant up

Login to the virual machine:

vagrant ssh

At this point, install CKAN/DCOR according to the description at https://dc.readthedocs.io/en/latest/https://dc.readthedocs.io/en/latest/sec_self_hosting/index.html. The default admin user and password for MinIO is minio-vagrant.

Also, make sure that the following packages are installed in the ckan environment:

pip install --upgrade codecov coverage pytest-ckan
# required by ckan test suite:
pip install --upgrade pytest_factoryboy
pip install --upgrade pytest-rerunfailures

When job is finished then create the box:

# cleanup apt
apt autoremove
apt clean
# cleanup pip
source /usr/lib/ckan/default/bin/activate
pip cache purge
# At this point it makes sense to restart.
# zero-out swap
swapoff /dev/sda2
dd if=/dev/zero of=/dev/sda2 bs=1M
mkswap /dev/sda2  # and edit `/etc/fstab` as well as `/etc/initramfs-tools/conf.d/resume` with new UUID
# zero-out the empty space on the boot and root partitions
mount -o remount,ro /dev/sda1
zerofree -v /dev/sda1
# This requires you to stop all services via systemctl
systemctl stop minio nginx postgresql rsyslog solr supervisor
systemctl stop syslog.socket rsyslog.service systemd-journald
# Try `lsof / | awk '$4 ~ /[0-9].*w/'` if this does not work:
systemctl stop systemd-journald systemd-journald.socket && mount -o remount,ro /dev/sda3
zerofree -v /dev/sda3
# logout and create the package
vagrant halt
vagrant package --output dcor-test_0.6.0.box

Box upgrade workflow

If apt/CKAN/DCOR packages only require updates, then it makes sense to base a new version of an image on an existing image:

cd from_previous
vagrant destroy -f
vagrant up
vagrant ssh

Once finished with the DCOR update, follow the box creation steps above.