action wtih 2 secret files - GHA is not mounting the second one

[t-readyroc]

Here is the Dockerfile:

FROM python@sha256:7c11ba98b104bed6dec12a3b10cb59b8dd1fa47ae1bae1ced6b609d5cacca8fc

COPY root/etc /etc
COPY root/usr /usr

ARG INSTALL_PKGS="\
    curl=7.74.0-1.3+deb11u9 \
    iproute2=5.10.0-4 \
    netcat=1.10-46 \
    vim-tiny=2:8.2.2434-3+deb11u1 \
    gnupg2=2.2.27-2+deb11u2 \
    procps=2:3.3.17-5"

RUN --mount=type=secret,id=artifactory_env,target=/etc/secrets/artifactory.env \
    --mount=type=secret,id=sources_list,target=/etc/apt/sources.list \
    . /etc/secrets/artifactory.env \
    && cat /etc/apt/sources.list \
    && update-ca-certificates \
    && apt-get update \
    && apt-get -y install --no-install-recommends $INSTALL_PKGS \
    && echo '. /etc/profile' >> /root/.bashrc

This Dockerfile builds successfully locally when run with the following script:

#!/bin/bash
#
# $1 local path to the artifactory environment file
# $2 local path to the sources list file
# both files contain credentials. See README

SUDO=$(type -P sudo)
OS=$(uname -s)
[[ $OS == Darwin ]] && unset SUDO

export DOCKER_BUILDKIT=1
$SUDO docker build --progress=plain --no-cache --secret id=artifactory_env,src=$1 --secret id=sources_list,src=$2 -f debian/Dockerfile_3.8-slim -t "<tag>" debian

Here are the relevant portions of the workflow file:

      - name: Build the Debian sources.list file
        if: inputs.REPO_CREDS == true && inputs.DEBIAN == true
        run: |
          echo "deb [trusted=yes] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/debian-main bullseye main" >> .sources_list.env
          echo "deb [trusted=yes] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/debian-security bullseye-security main" >> .sources_list.env
          echo "deb [trusted=yes] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/debian-main bullseye-updates main" >> .sources_list.env
          echo "deb [trusted=yes] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/debian-postgresql bullseye-pgdg main" >> .sources_list.env
          echo "deb [signed-by=/etc/apt/keyrings/libcontainers-stable.gpg] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@@${{ vars.ARTIFACTORY_URL }}/artifactory/libcontainers-stable-debian-11 /"

Followed by:

      - name: Build and push ${{ inputs.IMAGE_NAME }} Debian-based image with repo creds
        uses: docker/build-push-action@v4
        if: |
          inputs.PUSH_IMAGE == true &&
          inputs.REPO_CREDS == true &&
          inputs.DEBIAN == true
        with:
          context: ${{ inputs.CONTEXT }}
          file: ${{ inputs.DOCKERFILE }}
          push: true
          pull: true
          no-cache: true
          tags: ${{ vars.ARTIFACTORY_URL }}/${{ env.DOCKER_REPO_NAME }}/${{ inputs.IMAGE_NAME }}:${{ inputs.TAG }}
          build-args: ${{ inputs.BUILD_ARGS }}
          labels: ${{ inputs.LABELS }}
          secret-files: |
            "artifactory_env=./.artifactory.env"
            "sources_list=./.sources_list.env"

The first file, ./.artifactory.env is being created, and mounted. I've tested by echoing a couple of the environment variables that it's setting. The second file, ./.sources_list.env is also being created (tested with an extra cat step in the workflow), but is not being mounted. I've tested this by adding a cat step in the Dockerfile, which returns the default sources.list file. Resulting build attempts also fail, and you can see in the workflow output that it's attempting to fetch the packages from the default locations.

Anyone have any idea why it would mount one secret-file, but not the other? Any input would be greatly appreciated.

[crazy-max]

Can you post the full workflow file? You might need double escapes for quote signs in the secret value as shown in https://docs.docker.com/build/ci/github-actions/secrets/.

[t-readyroc]

Sure. Here's the whole file, however, I'm not doing a secret but rather secret-files, so I'm not sure that the double escapes apply here:

---

name: Docker build and publish image
on:
  workflow_call:
    inputs:
      IMAGE_NAME:
        required: true
        type: string
      TAG:
        required: true
        type: string
      DOCKERFILE:
        required: false
        default: Dockerfile
        type: string
      CONTEXT:
        required: false
        default: .
        type: string
      PUSH_IMAGE:
        required: false
        default: false
        type: boolean
      TEST_IMAGE:
        required: false
        default: false
        type: boolean
      BUILD_ARGS:
        required: false
        type: string
      REPO_CREDS:
        required: false
        default: false
        type: boolean
      DEBIAN:
        required: false
        default: false
        type: boolean
      LABELS:
        required: false
        type: string
      node_version:
        required: false
        type: string
env:
  DOCKER_BUILDKIT: 1
  DOCKER_REPO_NAME: docker
  ARTIFACTORY_TOKEN: ${{ secrets.ARTIFACTORY_TOKEN }}
jobs:
  build_and_push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Login to artifactory
        uses: docker/login-action@v2
        with:
          registry: ${{ vars.ARTIFACTORY_URL }}
          username: ${{ vars.ARTIFACTORY_USERNAME }}
          password: ${{ env.ARTIFACTORY_TOKEN }}
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          driver: docker
      - name: Create artifactory env file
        if: inputs.REPO_CREDS == true
        run: >
          for var in DNF_VAR NPM_CONFIG ; do
          echo "export ${var}_ARTIFACTORY_USERNAME=${{ vars.ARTIFACTORY_USERNAME }}" >> .artifactory.env ;
          echo "export ${var}_ARTIFACTORY_TOKEN=${{ env.ARTIFACTORY_TOKEN }}" >> .artifactory.env ;
          echo "export ${var}_ARTIFACTORY_URL=${{ vars.ARTIFACTORY_URL }}" >> .artifactory.env
          ; done
      - name: Add yum/dnf vars to the env file
        if: inputs.REPO_CREDS == true
        run: |
          echo "export YUM0=${{ vars.ARTIFACTORY_USERNAME }}" >> .artifactory.env
          echo "export YUM1=${{ env.ARTIFACTORY_TOKEN }}" >> .artifactory.env
          echo "export YUM2=${{ vars.ARTIFACTORY_URL }}" >> .artifactory.env
      - name: Build the Debian sources.list file
        if: inputs.REPO_CREDS == true && inputs.DEBIAN == true
        run: |
          echo "deb [trusted=yes] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/debian-main bullseye main" >> .sources_list.env
          echo "deb [trusted=yes] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/debian-security bullseye-security main" >> .sources_list.env
          echo "deb [trusted=yes] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/debian-main bullseye-updates main" >> .sources_list.env
          echo "deb [trusted=yes] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/debian-postgresql bullseye-pgdg main" >> .sources_list.env
          echo "deb [signed-by=/etc/apt/keyrings/libcontainers-stable.gpg] https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@@${{ vars.ARTIFACTORY_URL }}/artifactory/libcontainers-stable-debian-11 /"
      - name: Add pip vars to the env file
        if: inputs.REPO_CREDS == true
        run: >
          for var in INDEX INDEX_URL ; do
          echo "export PIP_${var}=https://${{ vars.ARTIFACTORY_USERNAME }}:${{ env.ARTIFACTORY_TOKEN }}@${{ vars.ARTIFACTORY_URL }}/artifactory/api/pypi/pypi/simple"
          >> .artifactory.env ; done
      - name: Test ${{ inputs.IMAGE_NAME }}
        uses: docker/build-push-action@v4
        if: inputs.TEST_IMAGE == true
        with:
          context: ${{ inputs.CONTEXT }}
          file: ${{ inputs.DOCKERFILE }}
          push: false
          tags: ${{ inputs.IMAGE_NAME }}:${{ inputs.TAG }}
          load: true
      - name: Build and push ${{ inputs.IMAGE_NAME }}
        uses: docker/build-push-action@v4
        if: |
          inputs.PUSH_IMAGE == true &&
          inputs.REPO_CREDS == false
        with:
          context: ${{ inputs.CONTEXT }}
          file: ${{ inputs.DOCKERFILE }}
          push: true
          pull: true
          no-cache: true
          tags: ${{ vars.ARTIFACTORY_URL }}/${{ env.DOCKER_REPO_NAME }}/${{ inputs.IMAGE_NAME }}:${{ inputs.TAG }}
          build-args: ${{ inputs.BUILD_ARGS }}
          labels: ${{ inputs.LABELS }}
      - name: Build and push ${{ inputs.IMAGE_NAME }} with repo creds
        uses: docker/build-push-action@v4
        if: |
          inputs.PUSH_IMAGE == true &&
          inputs.REPO_CREDS == true
        with:
          context: ${{ inputs.CONTEXT }}
          file: ${{ inputs.DOCKERFILE }}
          push: true
          pull: true
          no-cache: true
          tags: ${{ vars.ARTIFACTORY_URL }}/${{ env.DOCKER_REPO_NAME }}/${{ inputs.IMAGE_NAME }}:${{ inputs.TAG }}
          build-args: ${{ inputs.BUILD_ARGS }}
          labels: ${{ inputs.LABELS }}
          secret-files: |
            "artifactory_env=./.artifactory.env"
      - name: Build and push ${{ inputs.IMAGE_NAME }} Debian-based image with repo creds
        uses: docker/build-push-action@v4
        if: |
          inputs.PUSH_IMAGE == true &&
          inputs.REPO_CREDS == true &&
          inputs.DEBIAN == true
        with:
          context: ${{ inputs.CONTEXT }}
          file: ${{ inputs.DOCKERFILE }}
          push: true
          pull: true
          no-cache: true
          tags: ${{ vars.ARTIFACTORY_URL }}/${{ env.DOCKER_REPO_NAME }}/${{ inputs.IMAGE_NAME }}:${{ inputs.TAG }}
          build-args: ${{ inputs.BUILD_ARGS }}
          labels: ${{ inputs.LABELS }}
          secret-files: |
            "artifactory_env=./.artifactory.env"
            "sources_list=./.sources_list.env"
      - name: Cleanup after build
        uses: colpal/actions-clean@v1
        if: always()

[crazy-max]

I'm not doing a secret but rather secret-files, so I'm not sure that the double escapes apply here:

Indeed this is not necessary

The second file, ./.sources_list.env is also being created (tested with an extra cat step in the workflow), but is not being mounted.

That's strange. Have you tried another mount path for testing like --mount=type=secret,id=sources_list,target=/tmp/sources.list?

[t-readyroc]

I have not yet tried another mount, but: 1. I need it to be at /etc/apt/sources.list, & 2. like I said, building locally mounts it there just fine.

[t-readyroc]

OK, I changed the mount in the Dockerfile to point to the following locations:

1. /tmp/sources.list
2. /var/sources.list

Both runs failed, since I added a cat into the Dockerfile to check that the file was actually present. They each failed with the same general error:

cat: /tmp/sources.list: No such file or directory
cat: /var/sources.list: No such file or directory

I'd really like to know why the GHA is not mounting the 2nd file.

[t-readyroc]

Another data point: re-ordering the listing of the secret-files has no effect. I switched the ordering to the below, and the file was still not mounted:

          secret-files: |
            "sources_list=./.sources_list.env"
            "artifactory_env=./.artifactory.env"

I tested its creation with the addition of a cat step in the workflow after the creation step, which returned successfully:

Run cat .sources_list.env
deb [trusted=yes] ***<our_artifactory>/artifactory/debian-main bullseye main
deb [trusted=yes] ***<our_artifactory>/artifactory/debian-security bullseye-security main
deb [trusted=yes] ***<our_artifactory>/artifactory/debian-main bullseye-updates main
deb [trusted=yes] ***<our_artifactory>/artifactory/debian-postgresql bullseye-pgdg main

But the cat command in the Dockerfile failed again. So again, the file is being created correctly, just not mounted.

[t-readyroc]

Thanks for looking - I've sorted it out. My conditionals weren't specific enough, & so the step that would actually mount the additional file was not running. I had to add another conditional to the previous step to ensure that it wouldn't run if the calling workflow needed the Debian step.