Provide a supported way to build docker images in tasks

[petemoore]

NSS currently can't upgrade taskcluster/image_builder from version 0.1.5 to version 1.3.0 since the 1.3.0 version is gecko-specific.

For this reason, it might make sense to move this into its own (generic) tool.

A disadvantage of this is it might be harder to hack on, in-tree, if you are working on gecko.

Without knowing too many details, I'm creating this issue so that people who know better than me can chime in.

Issue came to light today in #taskcluster.

See also bug 1361413.

CC @djmitche @ttaubert

[gregarndt]

+1 for having a generic image building tool, but I think that it would need to wait for the QEMU engine to be done so we have a better environment to offer to people for building images.

CC @jonasfj

[jonasfj]

We already have docker running under qemu in tc-worker... just started using it for CI :)

[djmitche]

@petemoore can you give more detail on what you envision?

[jonasfj]

I could imagine a base ubuntu image for qemu that already contains docker and a script to clone a git repo and run a script...
So that you just write:

task.payload = {
  image: 'https://static-url-to-ubuntu-image-for-qemu.tar.zst',
  env: {
    REPOSITORY: 'https://github.com/<org>/<repo>.git'
    REVISION: '<revision>',
  },
  command: ['clone-and-exec.sh', 'make', 'build-docker-image'],
  artifacts: [
    {type: 'file', path: '/myimage.zst', name: 'public/images/myimage.zst'},
  ]
}

Assuming your repository contains a Makefile that defines the target build-docker-image which builds a docker image and exports it as a file to /myimage.zst...

Spoiler alert, we have this now :)

[djmitche]

I'm not sure tc-worker, and specifically QEMU, is ready for "prime time" yet. And @petemoore seems to be indicating using docker-in-docker rather than QEMU in the title of this issue. I'm not sure what functionality that would include, and what would be left to the user. @jonasfj's suggestion omits (well, leaves to the user) a lot:

• mercurial support
• "recipes" to copy data from elsewhere in the tree into the Dockerfile directory
• hash generation
• zstd compression
• interfacing with docker (is this via the docker cli, or with curl talking to the docker API?)

Outside of QEMU, the equivalent would be a basic ubuntu:16.04 image with docker and mercurial installed. I don't see a lot of value to us writing and owning a 2-line Dockerfile.

So, I think my question still stands:

@petemoore can you give more detail on what you envision?

[jonasfj]

Another option IMO is an image build that reads an env var CONTEXT_URL, downloads and extracts a tarball from given url, and then builds Dockerfile from said tarball...

Then you be a task that extracts things from you repo... Created said tarball and the.create a dependent task..

Note: IMO using docker CLI and zstd is the easy part...

[djmitche]

Tc-worker is certainly more "prime time" now. Maybe it's time to think about this again? I think this will be easier to solve than #51 using something like what @jonasfj suggested in the previous comment.

[jonasfj]

yeah, tc-worker with qemu engine is the future for building docker images. Once it lands in-tree we'll have stable workerType people can use too.

[djmitche]

@jonasfj, @petemoore I think @glandium has done a lot of the work here to make the existing docker-in-docker solution workable in Gecko. But I'm still missing what the proposed change is here.

[glandium]

Note that I've actually originally written the code that is now in Gecko for git-cinnabar CI, meaning that I am also doing docker-in-docker image building on taskcluster-github. I'm using the python:2.7 image to do that, pip install requests-unixsocket zstandard==0.8.1 and then I run a script within my repo that has the python foo now in Gecko to drive docker-in-docker to build the image for me.

[djmitche]

Related to #51, and something we'll probably want with redeployability.

[petemoore]

@SimonSapin, you mentioned today you are building docker image artifacts from a Dockerfile inside a non-gecko task - are your tasks using taskcluster/image_builder for this, or do you have a different solution? I'm wondering if this issue can be closed. I'm not sure we have docs on how to set this up, in which case we should probably only close it once we have docs, so I'm curious how you set this up, and who you got help from (or if you stumbled on some docs)! Many thanks.

[SimonSapin]

I did not use taskcluster/image_builder. I made a minimal image whose Dockerfile is in https://github.com/servo/taskcluster-bootstrap-docker-images/. It is hosted on and built by Docker Hub.

As far as I can tell, the necessary pieces are:

• A worker type where docker-in-docker is enabled

• An image builder image where Docker 1.6 (at least the client parts) is installed. A more recent version of the Docker client might cause "protocol version mismatch" errors, at least with the AMIs currently running on servo-docker-worker. 1.6 happens to be the version packaged in Ubuntu 14.04 as docker.io.

• A docker-worker task with payload.features.dind = true

Then that task should be able to run docker build and docker save. (And save the resulting tarball as an artifact.)

To cache built images, I took inspiration (though not code) from https://firefox-source-docs.mozilla.org/taskcluster/taskcluster/docker-images.html to use the Index service with a hash of the source Dockerfile. The code for that is at:

https://github.com/servo/servo/blob/c33f5cc/etc/ci/taskcluster/decisionlib.py#L60-L131

See also https://github.com/servo/servo/tree/master/etc/ci/taskcluster#in-tree-docker-images

[glandium]

FWIW, for git-cinnabar, I'm using an ad-hoc docker client with docker-in-docker. Similar code is actually now used for gecko too. https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/util/docker.py

[petemoore]

@petemoore can you give more detail on what you envision?

Answering a little late, but essentially I was thinking of the following:

In mozilla-central, users simply need to update a Dockerfile, and docker image build tasks are automatically created. I'm not sure how the internals of this work, but I suspect there is some integration in the taskgraph library which the decision task uses. As I understand it, the automatic creation of docker image build tasks that occurs in mozilla-central is not something that can be used outside of mozilla-central, because for example it executes ./mach taskcluster-build-image to build the image, and ./mach is something specific to the mozilla-firefox build system (according to this). So I think we need something that doesn't assume the mach build system is used.

It is probably better to build docker images in a generic-worker multiuser engine task than a docker-worker task, since docker-worker isn't so easy to maintain and we hope will be replaced, and also building a docker image inside a docker container is probably more complicated and/or risky than building one outside of docker.

I would need to study the internals of the docker image building code to provide a more concrete proposal, but this is at least the higher level objective: the taskgraph library should support creating docker image building tasks that publish new docker images as task artifacts, if they determine that a docker image definition has been updated (e.g. a Dockerfile or referenced objects) and when the source code that builds the docker image is explicitly configured to be the source image of a downstream task. It should also make sure to use the correct docker image build (like is done in mozilla-central) for the tasks that run inside a container created from that docker image. The docker images themselves should be binary task artifacts, like in mozilla-central, that either docker-worker directly uses, or a generic-worker task can mount and start a docker container with.

[escapewindow]

Tom moved us to kaniko; we can build docker images within docker containers without dind with that. See bug 1626058