Accessing build-options after shrinkwrap for separate builders

[ArcticXWolf]

When using the shrinkwrap option to generate a build context for separate builders like buildkit, kaniko or else, we copy all information about the build (Files + Dockerfile) to the ./build directory. However, for a successful build we also need the build-args and build-options which are fed into the docker build process (or in our case the kaniko process).

In our case, we use kaniko to build the function images inside our Kubernetes cluster with one-off-jobs. Our images need some additional packages installed, so we use the build options feature of our templates. The kaniko-job is very similar to the one @alexellis created for his blogpost. The problem is that we wont populate the ADDITIONAL_PACKAGES arg in the Dockerfile and thus the build fails.

So a local workaround would be to parse the template.yml and stack.yml ourselves, identify which function is being built currently and which options apply and then construct the build args. However this would be a lot of duplicated effort, as faas-cli already has this information.

A good idea would be to settle on a way to exchange that information with the builder, maybe adding it to a specific file in the build context?

Expected Behaviour

It should be possible to save the relevant build args for the current build during a shrinkwrap. This might take the form (to be discussed) of a file containing all build args as key-values:

ADDITIONAL_PACKAGES="gcc"

Current Behaviour

During a shrinkwrap build only the files of a build context are copied. The build args are missing and thus cannot be restored for an external builder.

Possible Solution

See Expected Behaviour.

Context

We build function images via kaniko similar to the way @alexellis did it for his blogpost. There build options are sadly not usable with the builds.

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ):

$ faas version
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|

CLI:
 commit:  f7c29ea19b5df9d7aa87e9c70aacf4d9315da2cd
 version: 0.12.4

Gateway
 uri:     http://localhost:9645
 version: 0.18.17
 sha:     18f6c720b50db7da5f9c410f9fd3369ed7aff379
 commit:  Extract a caching function_query type


Provider
 name:          faas-netes
 orchestration: kubernetes
 version:       0.10.5
 sha:           9be50543b372381a505e9e54a1356bb076c8f01f

• Docker version: 19.03.8

• Are you using Docker Swarm (FaaS-swarm ) or Kubernetes (FaaS-netes)?
  FaaS-netes

• Operating System and version (e.g. Linux, Windows, MacOS):
  Ubuntu and Windows 10

[alexellis]

Hi @ArcticXWolf

This is something we can fix relatively quickly I would think, we just need to think about the options because this needs to be parsed by your container builder of choice. It will probably be that a custom bash script or process will need to load it and invoke kaniko.

For openfaas-cloud we do something similar for build-args and have the target image name stored too, it's a JSON file. openfaas/openfaas-cloud@63a2f66

Alex

[ArcticXWolf]

Hi, thanks for the quick reply!

Yes, a wrapper script is needed to parse that file.

A JSON file would be great, it could contain even more metadata concerning the build.

[ArcticXWolf]

Hey,

if it is okay, I would like to try and implement this. Go isn't my main language, so I might need some help/advice on style and tests later, but I would love to give it a shot (and I guess that's what Code reviews are for ^^).

However we should specify some things:

• How shall we name and structure the configFile? I think it makes sense to stick to the format and name of of-cloud (thus 'com.openfaas.docker.config' (or .json as extension?) as name and including the same buildConfig type). Or shall we add even more information about the build while we have all that infomation there?
• Shall we add this config file per default or just as a command line option? I propose the latter, how shall we name it (--build-config)? And how shall it behave if --shrinkwrap is not set?
• How shall we deal with the case that there is already a file with the config file name in the template? It would get overwritten which could be bad.

EDIT:
I gave it a quick thought and think it is maybe not a good idea to use json as the format, because it might not be as easy to parse inside a shell script. For example with kaniko, the debug-image only contains a simple busybox where jq or python is not available. Maybe it is easier to expose them in a different format, like a simple key=value list of lines.

[alexellis]

I would like to see you make a proof of concept for how you would parse and read the config file and then "send it to Kaniko" and bear in mind that other container builders exist. I appreciate the suggested solutions too, but let's first prove out that this is a viable approach before opening PRs or hacking on the code.

[alexellis]

@ArcticXWolf It's been just over a week since we heard from you, is this still something you need and want to work on?

[ArcticXWolf]

Ah, I was waiting for your input, because your answers were posted just 30 min ago.

Sure, I have already a proof of concept, gonna type it out in detail later.

[ArcticXWolf]

Here is my kubernetes-job that I would use. The volume build-context would get populated with a shrinkwrap build context, that contains the build-args like an env-file. This shrinkwrap will be done via an init-container like @alexellis did in his blogpost mentioned above. [FUNCTION_NAME] has to be substituted for the name of the function we want to build (Can be done via envsubst for example). "docker-registry" is the domain name of the registry where the image gets push onto.

There is a small problem with Kaniko, as it does not support spaces in their build-arg command line option, this can be temporarily solved by providing the build-arg via env-variables.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kaniko-job-remote-build-script
  namespace: kaniko-system
data:
  buildscript.sh: |-
    #!/busybox/sh
    
    # Small disclaimer to whoever has to read this. At the time of writing,
    # Kaniko does not support spaces in --build-arg or --label.
    # (See https://github.com/GoogleContainerTools/kaniko/issues/1231)
    # However we can use environment variables to encode strings with spaces
    # so that they do not break. 
    # https://github.com/GoogleContainerTools/kaniko/blob/a47a78edae81e81c4fead1d7a1dc2d50a637133e/cmd/executor/cmd/root.go#L231
    # So what might happen is, that if we specifc build-args that are named
    # the same as a normal env variable (e.g. HOME), then we overwrite it here..
    # Should not pose a problem, because there are only few variables set and
    # this is a one-shot-container, but still.
    # If the above issue gets fixed, we can move to giving the build-args via
    # commandline.
    
    # Import build-args to env
    set -a
    source /workspace/build/[FUNCTION_NAME]/com.openfaas.docker.config
    set +a

    # Create a list of build-arg names to trigger import from env
    # Choose a name that likely wont be used as buildarg.
    BUILDARG_QSVAOIHEFFDASDPWEW=`cat /workspace/build/[FUNCTION_NAME]/com.openfaas.docker.config | sed -E 's/([^=]+)=.*/--build-arg \1/g'`

    /kaniko/executor \
        --cleanup \
        -c "/workspace/build/[FUNCTION_NAME]/" \
        -d "docker-registry:5000/[FUNCTION_NAME]:latest" \
        $BUILDARG_QSVAOIHEFFDASDPWEW

---
apiVersion: batch/v1
kind: Job
metadata:
  name: build-job-[FUNCTION_NAME]
  namespace: kaniko-system
  labels:
    app: kaniko-builder
spec:
  backoffLimit: 0
  template:
    spec:
      containers:
      - name: build
        image: gcr.io/kaniko-project/executor:debug
        command: ["/busybox/sh","-c"]
        args: ['. /scripts/buildscript.sh']
        volumeMounts:
        - name: build-context
          mountPath: /workspace
        - name: build-script
          mountPath: /scripts
      restartPolicy: Never
      volumes:
      - name: build-context
        emptyDir: {}
      - name: build-script
        configMap:
          name: kaniko-job-remote-build-script
          defaultMode: 0755

Thus a sample com.openfaas.docker.config file would look like:

ADDITIONAL_PACKAGE="jq gcc"

The buildscript adds this to the environment and then passes the variable to the kaniko executor. The final command would be:

/kaniko/executor \
        --cleanup \
        -c "/workspace/build/[FUNCTION_NAME]/" \
        -d "docker-registry:5000/[FUNCTION_NAME]:latest" \
        --build-arg ADDITIONAL_PACKAGE

Of course this example is now quite specific to Kaniko because of the spaces-issue mentioned in the comments, however this should work for other builders too.

To summarize my proposal:

1. Addition of new commandline flag "--write-build-args" that has to be used in conjunction with shrinkwrap
2. When "--write-build-args" is set, the shrinkwrap adds the build-args to the build-context as an .env-file called "com.openfaas.docker.config"

Someone who wants to use it can use the following workflow:

1. Create build-context via shrinkwrap with "--write-build-args".
2. Parse the com.openfaas.docker.config file inside a shell script (either import into environment or into a local variable)
3. Call builder with the args (either by providing them via command-line options or via env, depending on your builder)

[alexellis]

If you know the builder that you want to use, we could just generate a shell command for it like we do for docker?

[alexellis]

@LucasRoesler + @utsavanand2 we were discussing this topic on the call WRT to using buildkit in GitHub Actions. Do you have any suggestions on how to pass the image name / build args etc to an external builder after we've run "build --shrinkwrap"?

[utsavanand2]

There are so many ways that one can build a Docker image, I think it would be hard to support each and every one ourselves. But can we do something like what we have done with the inletsctl provisioner? Define a basic way of how we will get some commands for a particular builder (an interface), say starting with Docker Buildx, and other people can contribute for their builders of choice.
What I’m thinking is to store these additional build-args and related metadata in memory say a struct, and based on a flag that specifies a particular builder, we output a shell script, that can be piped to sh or to a .sh file to be executed later.
What do you think @LucasRoesler does it even make any sense?

[LucasRoesler]

what if we add an option to output the flags to stdout, for example --show-flags. We can provide several flags to allow the caller to pick a format: flags, json, yaml, key=value, etc, and they can redirect to a file or to another program.

With the flags format we could also allow the user to specify a replacement name for the flag.

For example, our build command is currently

docker
build 
<--no-cache>
<--squash>
<--build-arg http_proxy=?>
<--build-arg https_proxy=?>
<--build-arg ?=?>
<--label ?=?>
-t <name> 

so the user can say

faas-cli build \
--shrinkwrap \
--show-flags \
--format=flags \
--rename='no-cache=nocache' \
--rename='squash=flatten' \
--rename='t=name'

etc to further customize the output.

[ArcticXWolf]

How would we separate the output of the flags from the output of the faas-cli? Stdout/stderr? Or disable other output when the --show-flags is given?

[SaigyoujiYuyuko233]

How about when user run faas build with --build-option <input> --shrinkwrap, the faas will generate a env file in the build dir and when user build it, that file will be source and become the env of the container.

[alexellis]

That may be an option.

The problem is that using shrinkwrap implies that you are not using a standard builder, it's a very non-standard and bespoke path, so without a customer for this it's difficult to design something that would work.