Hinkskalle

On-Premises Container Registry for OCI/docker and singularity

What Am I

(buckethead) - I can store, retrieve and manage OCI and singularity containers in a central library.

Compatible with/re-implementation of the singularity library protocol and the OCI distribution spec.

Documentation

Installation + Usage instructions can be found here:

https://csf-ngs.github.io/hinkskalle/

Continue to read here if you are interested in mucking around in the source code (and maybe helping out with the odd PR or two).

Development

Clone Current HEAD

git clone https://github.com/csf-ngs/hinkskalle.git

Docker development environment

Achieve the best development experience with continuous reloads and frontend builds! No need to set up/mess up your computer!

Hinkskalle comes with a development environment based on docker-compose.

The ghcr.io/csf-ngs/hinkskalle-dev image contains a complete development environment.

First Setup

Initial setup (or maybe you want to reset your dev environment):

# (of course you can use your own favorite dummy secrets)
cat <<_EOF > conf/db_secrets.env
POSTGRES_PASSWORD=supersecret
_EOF

cat <<_EOF > conf/secrets.env
HINKSKALLE_SECRET_KEY=superdupersecret
DB_PASSWORD=supersecret
HINKSKALLE_LDAP_BIND_PASSWORD=superldapsecret
_EOF

cat <<_EOF > conf/slapd_secrets.env
LDAP_ROOT_PASSWORD=superrootsecret
LDAP_LOGIN_PASSWORD=superldapsecret
_EOF

# start hinkdb first to set up base database
docker-compose up -d hinkdb
# give it a second
# install current database schema
docker-compose run --rm api flask db upgrade

# set up first admin user
docker-compose run --rm api flask localdb add-user \
    -u admin.hase \
    -p oink \
    -e 'admin.hase@testha.se' \
    -f Admin \
    -l Hase \
    --admin
# set up a normal user
docker-compose run --rm api flask localdb add-user \
    -u test.hase \
    -p oink \
    -e 'test.hase@testha.se' \
    -f Test \
    -l Hase 

# ONLY when you need to reset the dev env: clean everything
docker-compose down
docker-compose config --volumes | xargs docker volume rm 

Running Development Instances

Dev server: http://localhost:7660

# WARNING: On first startup we need to install node modules and build the frontend
# this might take a few minutes and the dev server will show
# The requested URL was not found on the server. until that's done.
#
# Whole stack (rarely needed)
docker-compose up -d
# bare minimum
docker-compose up -d api build_frontend
# log output
docker-compose logs -f

The current working directory (base) is mounted into the relevant containers. You can edit the source files with your favorite editor/IDE directly. Services will automatically rebuild and/or restart on changes.

This starts the following services:

api: Local Backend Instance at port 7660

Using script/start-dev.sh. Restarts on changes in backend/

build_frontend: Continuous Frontend Build

Using script/start-dev-frontend.sh, basically a yarn build --watch

hinkdb: Postgres database

rq_scheduler, rq_worker, redis: Backend async job queue (optional)

ldap: for testing LDAP authentication (optional)

hockeypuck, hockeypuck_db: PGP keyserver (optional)

Side Notes

• uploaded images are stored in ./tmp

Development Install

Needs postgresql dev libraries! Install according to your OS instructions, e.g.:

# mac os x
brew install postgresql
# ubuntu/debian
apt install postgresql-dev
# etc.

cd backend/
python3 -m venv venv
source venv/bin/activate
pip install -e '.[dev]'

This will also install nose2, Jinja2, fakeredis and psycopg2 for running tests and generating typescript classes.

You also need to set up Node, Vue and vue-cli for testing and compiling the frontend:

# install node according to your OS
cd frontend/
yarn install

Patch Singularity

Not necessary for singularity v3.9.0 or newer, use --insecure and --no-https params instead

Singularity absolutely required that the library server is reachable via https. While you can set this up for your development server, it's much easier to patch the source code and recompile your own.

The necessary patch is provided in share/singularity-plain-http.patch and should work an all versions.

ORAS requires a similar patch. If you want to play around with that, apply share/oras-plain-http.patch.

Follow the instructions on https://sylabs.io/guides/3.7/admin-guide/installation.html (adjust for the version you would like) and apply the patch between the steps "Checkout Code from Git" and "Compile Singularity":

cd ${GOPATH}/src/github.com/sylabs/singularity
patch -p1 < /path/to/singularity-plain-http.patch
patch -p1 < /path/to/oras-plain-http.patch

Start Development Server

script/start-dev.sh
# continuous build of frontend
script/start-dev-frontend.sh
# (optional: start rq worker)
# script/start-dev-worker.sh

Backend Tests

cd backend
nose2

Frontend Tests

cd frontend
yarn test:unit

OCI Conformance Tests

Requires a docker image built from https://github.com/opencontainers/distribution-spec/tree/main/conformance.

cd share/oci
./conformance-test.sh

Your backend should be available at localhost:7660

Generate Typescript Classes for models

pip3 install git+https://github.com/csf-ngs/swagspotta
# make sure your local dev server is running at localhost:7660
share/generate-models.sh

Built With

• Python Flask
• Vue.js

Contributing

Please do!

Authors

• Heinz Axelsson-Ekker - initial work - VBCF Next Generation Sequencing

License

This project is licensed under the MIT License - see the LICENSE.md file for details