[v26] Build caching with type=cache mounts

[vitoyucepi]

Description

In moby v26.0.0 layers could be cached if side effect is stored in --mount=type=cache with different id, thus changing the previous behavior.

Reproduce

1. Dockerfile

   FROM ubuntu:22.04 as b1
   
   RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=b1 \
       apt-get update
   RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=b1 \
       apt-get install -y --no-install-recommends curl
   
   FROM ubuntu:22.04 as b2
   
   RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=b2 \
       apt-get update
   RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=b2 \
       apt-get install -y --no-install-recommends ca-certificates
   

2. docker build --progress plain --target b1 .
3. docker build --progress plain --target b2 .

Expected behavior

Both targets should be built.

docker version

Client:
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.22.1
 Git commit:        2ae903e86c
 Built:             Sat Mar 23 15:27:08 2024
 OS/Arch:           linux/amd64
 Context:           rootless

Server:
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.22.1
  Git commit:       8b79278316
  Built:            Sat Mar 23 15:27:08 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.15
  GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1.m
 runc:
  Version:          1.1.12
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /run/user/1000/dockerd-rootless
 slirp4netns:
  Version:          1.2.3
  GitCommit:        c22fde291bb35b354e6ca44d13be181c76a0a432

docker info

Client:
 Version:    26.0.0
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.13.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.26.1
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 926c9586fe4a6236699318391cd44976a98e31f1.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.8.2-arch2-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 31.28GiB
 Name: arch
 ID: 45fa93eb-3351-4ad8-b68a-c1ec6a7500ac
 Docker Root Dir: /home/main/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional Info

I bisect the repo to get the list of suggested bad commits.

• bc6d88c
• a79bb1e
• b04a2da
• 59ad169
• e68f712
• dd69926
• fa467ca
• 31545c3
• e01a1c5
• 951e42c

git bisect log

git bisect start
# status: waiting for both good and bad commits
# bad: [f3842ab533b6c04512354be573840e3bdddfd27b] Merge pull request #47671 from vvoland/v26.0-47670
git bisect bad f3842ab533b6c04512354be573840e3bdddfd27b
# status: waiting for good commit(s), bad commit known
# good: [9e075f3808a5a5fd54f0217ac114d3a02b16af2f] Merge pull request #47155 from thaJeztah/remove_deprecated_api_versions
git bisect good 9e075f3808a5a5fd54f0217ac114d3a02b16af2f
# bad: [460b4aebdf81512a3ea37c2a65826b890427537d] Merge pull request #47502 from vvoland/go-1.21.8
git bisect bad 460b4aebdf81512a3ea37c2a65826b890427537d
# good: [16aa7dd67fc20f15690b145f30e921d783746cb6] c8d/pull: Output truncated id for `Pulling fs layer`
git bisect good 16aa7dd67fc20f15690b145f30e921d783746cb6
# bad: [230cb53d3b6ba160a92cdaea6689392d231351b3] Merge pull request #47462 from vvoland/integration-testdaemonproxy-reset-otel
git bisect bad 230cb53d3b6ba160a92cdaea6689392d231351b3
# bad: [2d31532a001b196be6f09e871e484f8892a67a03] otel: Default metrics protocol to http/protobuf
git bisect bad 2d31532a001b196be6f09e871e484f8892a67a03
# skip: [b04a2dad6b8ab3eb5e6cb04c3a372abbc5cef535] builder/controller: Adjust NewWorkerOpt call
git bisect skip b04a2dad6b8ab3eb5e6cb04c3a372abbc5cef535
# bad: [951e42cd60f683d8e12fcae88e74c2ad8ca770ab] builder-next: Replace ResolveImageConfig with ResolveSourceMetadata
git bisect bad 951e42cd60f683d8e12fcae88e74c2ad8ca770ab
# skip: [a79bb1e83219959003d22e5ff1950a87b1dd2fde] builder-next/exporter: Sync with new signature
git bisect skip a79bb1e83219959003d22e5ff1950a87b1dd2fde
# skip: [dd6992617e3b3e7e5b16a4a41d53d207e34475df] integration/build: Use new buildkit progressui
git bisect skip dd6992617e3b3e7e5b16a4a41d53d207e34475df
# skip: [bc6d88c09a01d9d4b1d355b79b372be719f3f813] cmd/dockerd: Fix overriding OTEL resource
git bisect skip bc6d88c09a01d9d4b1d355b79b372be719f3f813
# skip: [fa467caf4dcce94048fa50f6151e439ee4d7f8dd] builder-next/mobyexporter: Use OptKeyName const
git bisect skip fa467caf4dcce94048fa50f6151e439ee4d7f8dd
# skip: [e68f71259ad27842c85cad3db31cec94c7668291] integration/build: Use fsutil.NewFS
git bisect skip e68f71259ad27842c85cad3db31cec94c7668291
# skip: [31545c3b6792e962437afdcc50bc999e7d3fb36b] vendor: github.com/moby/buildkit v0.13.0-rc2
git bisect skip 31545c3b6792e962437afdcc50bc999e7d3fb36b
# skip: [59ad1690f7482cf502f76a78ac0dbb127d3207df] builder-next: Adjust to source changes
git bisect skip 59ad1690f7482cf502f76a78ac0dbb127d3207df
# skip: [e01a1c5d09371b8397f06b20a1ebefacc5a3939c] builder/mobyexporter: Set `image.name` response key
git bisect skip e01a1c5d09371b8397f06b20a1ebefacc5a3939c
# good: [f90b03ee5d266a639cb28b9f204f5e23f47dc82e] go.mod: bump to go 1.21 and use local toolchain when vendoring
git bisect good f90b03ee5d266a639cb28b9f204f5e23f47dc82e
# only skipped commits left to test
# possible first bad commit: [951e42cd60f683d8e12fcae88e74c2ad8ca770ab] builder-next: Replace ResolveImageConfig with ResolveSourceMetadata
# possible first bad commit: [e01a1c5d09371b8397f06b20a1ebefacc5a3939c] builder/mobyexporter: Set `image.name` response key
# possible first bad commit: [fa467caf4dcce94048fa50f6151e439ee4d7f8dd] builder-next/mobyexporter: Use OptKeyName const
# possible first bad commit: [59ad1690f7482cf502f76a78ac0dbb127d3207df] builder-next: Adjust to source changes
# possible first bad commit: [b04a2dad6b8ab3eb5e6cb04c3a372abbc5cef535] builder/controller: Adjust NewWorkerOpt call
# possible first bad commit: [bc6d88c09a01d9d4b1d355b79b372be719f3f813] cmd/dockerd: Fix overriding OTEL resource
# possible first bad commit: [a79bb1e83219959003d22e5ff1950a87b1dd2fde] builder-next/exporter: Sync with new signature
# possible first bad commit: [e68f71259ad27842c85cad3db31cec94c7668291] integration/build: Use fsutil.NewFS
# possible first bad commit: [dd6992617e3b3e7e5b16a4a41d53d207e34475df] integration/build: Use new buildkit progressui
# possible first bad commit: [31545c3b6792e962437afdcc50bc999e7d3fb36b] vendor: github.com/moby/buildkit v0.13.0-rc2

[thaJeztah]

Both targets should be built.

Hm, I can double-check with the BuildKit maintainers, but I don't think the content of the mount should make a difference w.r.t. the build-cache; the expectation for cache is that the RUN command cannot depend on any state of the cache prior to the RUN command being executed, so the id should make no difference, other than to help with concurrency (two RUN instructions trying to modify content of the same mounts). https://docs.docker.com/reference/dockerfile/#run---mounttypecache

Contents of the cache directories persists between builder invocations without invalidating the instruction cache. Cache mounts should only be used for better performance. Your build should work with any contents of the cache directory as another build may overwrite the files or GC may clean it if more storage space is needed.

In other words, --mount type=cache is not taken into account for the build-cache, and given that both apt-cache update instructions start from the same base-image, both could share the same build-cache.

[thaJeztah]

In general, I would definitely recommend combining apt-get update with apt-get install into a single RUN though, so that updating the list of packages to install forces the cache to be invalidated (otherwise the apt-get update may never be updated)

[thaJeztah]

Had a quick chat on Slack, and it looks like there was a previous bug that caused the id to cause the cache to be invalidated; this was a bug, so it's possible you were depending on that bug here;

• Do not include a cache mount's ID in the ExecOp's cachemap buildkit#4585

[vitoyucepi]

If it was a bug, it was an absolutely amazing one. For me, it solves the problem of building multiple images that look the same on the local machine at the same time. In CI, I build each image in a separate namespace so they don't overlap.

Here are some examples

2.2/Dockerfile

FROM alpine:3.19.1@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b as downloader

# renovate: datasource=github-tags depName=savonet/liquidsoap-release-assets
ARG LIQUIDSOAP_VERSION=2.2.4

ARG LIQUIDSOAP_RELEASE="https://github.com/savonet/liquidsoap-release-assets/releases/download/v${LIQUIDSOAP_VERSION}"
ARG LIQUIDSOAP_DEB="liquidsoap_${LIQUIDSOAP_VERSION}-ubuntu-jammy-2_amd64.deb"
ARG LIQUIDSOAP_SHA512SUM="d16026b57ca9fbca6f89e1ba8523c09b6693868607c56119769cb0a3303b4328978f5e0a62ed9ea509bebb2fa5935a24cb03c6836c139cc2dae0ad55fdb6b036"

# hadolint ignore=DL3047,DL4006
RUN set -eux; \
      mkdir /packages; \
      wget "${LIQUIDSOAP_RELEASE}/${LIQUIDSOAP_DEB}" -O /packages/liquidsoap.deb; \
      echo "${LIQUIDSOAP_SHA512SUM}  /packages/liquidsoap.deb" | sha512sum -c -;

FROM ubuntu:22.04@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44eff17e

ENV TZ=UTC
ARG DEBIAN_FRONTEND=noninteractive

# hadolint ignore=DL3009
RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2 \
    apt-get update

# renovate: datasource=repology depName=ubuntu_22_04/bash versioning=loose
ARG BASH_VERSION=5.1-6ubuntu1.1
# renovate: datasource=repology depName=ubuntu_22_04/util-linux versioning=loose
ARG UTIL_LINUX_VERSION=2.37.2-4ubuntu3.3

# hadolint ignore=DL3015
RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2 \
    set -eux; \
      apt-get install -y --only-upgrade \
        bash="${BASH_VERSION}" \
        bsdutils="1:${UTIL_LINUX_VERSION}" \
        libblkid1="${UTIL_LINUX_VERSION}" \
        libmount1="${UTIL_LINUX_VERSION}" \
        libsmartcols1="${UTIL_LINUX_VERSION}" \
        libuuid1="${UTIL_LINUX_VERSION}" \
        mount="${UTIL_LINUX_VERSION}" \
        util-linux="${UTIL_LINUX_VERSION}" \
      ; \
      rm -f \
        /var/lib/dpkg/status-old \
        /var/cache/debconf/templates.dat-old \
        /var/cache/debconf/config.dat-old \
      ;

# renovate: datasource=repology depName=ubuntu_22_04/ca-certificates versioning=loose
ARG CA_CERTIFICATES_VERSION=20230311ubuntu0.22.04.1
# renovate: datasource=repology depName=ubuntu_22_04/gosu versioning=loose
ARG GOSU_VERSION=1.14-1
# renovate: datasource=repology depName=ubuntu_22_04/tzdata versioning=loose
ARG TZDATA_VERSION=2024a-0ubuntu0.22.04

RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2 \
    --mount=type=cache,target=/packages,source=/packages,from=downloader \
    set -eux; \
      apt-get install -y --no-install-recommends \
        ca-certificates="${CA_CERTIFICATES_VERSION}" \
        gosu="${GOSU_VERSION}" \
        tzdata="${TZDATA_VERSION}" \
        "/packages/liquidsoap.deb" \
      ; \
      rm -f \
        /var/lib/dpkg/status-old \
        /var/cache/debconf/templates.dat-old \
        /var/cache/debconf/config.dat-old \
      ;

COPY rootfs /

ENTRYPOINT [ "entrypoint" ]
CMD [ "liquidsoap", "/var/liquidsoap/default.liq" ]

2.2-minimal/Dockerfile

FROM alpine:3.19.1@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b as downloader

# renovate: datasource=github-tags depName=savonet/liquidsoap-release-assets
ARG LIQUIDSOAP_VERSION=2.2.4

ARG LIQUIDSOAP_RELEASE="https://github.com/savonet/liquidsoap-release-assets/releases/download/v${LIQUIDSOAP_VERSION}"
ARG LIQUIDSOAP_DEB="liquidsoap-minimal_${LIQUIDSOAP_VERSION}-ubuntu-jammy-2_amd64.deb"
ARG LIQUIDSOAP_SHA512SUM="a99e2739d4cec26bd1cbe21d1cb32204e290d05fe449f770c7a90dc935d46ed39b337667f10996acc1e6bb37a47ca3bb26fc0561a7f3a04d157525e278321b3f"

# hadolint ignore=DL3047,DL4006
RUN set -eux; \
      mkdir /packages; \
      wget "${LIQUIDSOAP_RELEASE}/${LIQUIDSOAP_DEB}" -O /packages/liquidsoap.deb; \
      echo "${LIQUIDSOAP_SHA512SUM}  /packages/liquidsoap.deb" | sha512sum -c -;

FROM ubuntu:22.04@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44eff17e

ENV TZ=UTC
ARG DEBIAN_FRONTEND=noninteractive

# hadolint ignore=DL3009
RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2-minimal \
    apt-get update

# renovate: datasource=repology depName=ubuntu_22_04/bash versioning=loose
ARG BASH_VERSION=5.1-6ubuntu1.1
# renovate: datasource=repology depName=ubuntu_22_04/util-linux versioning=loose
ARG UTIL_LINUX_VERSION=2.37.2-4ubuntu3.3

# hadolint ignore=DL3015
RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2-minimal \
    set -eux; \
      apt-get install -y --only-upgrade \
        bash="${BASH_VERSION}" \
        bsdutils="1:${UTIL_LINUX_VERSION}" \
        libblkid1="${UTIL_LINUX_VERSION}" \
        libmount1="${UTIL_LINUX_VERSION}" \
        libsmartcols1="${UTIL_LINUX_VERSION}" \
        libuuid1="${UTIL_LINUX_VERSION}" \
        mount="${UTIL_LINUX_VERSION}" \
        util-linux="${UTIL_LINUX_VERSION}" \
      ; \
      rm -f \
        /var/lib/dpkg/status-old \
        /var/cache/debconf/templates.dat-old \
        /var/cache/debconf/config.dat-old \
      ;

# renovate: datasource=repology depName=ubuntu_22_04/ca-certificates versioning=loose
ARG CA_CERTIFICATES_VERSION=20230311ubuntu0.22.04.1
# renovate: datasource=repology depName=ubuntu_22_04/gosu versioning=loose
ARG GOSU_VERSION=1.14-1
# renovate: datasource=repology depName=ubuntu_22_04/tzdata versioning=loose
ARG TZDATA_VERSION=2024a-0ubuntu0.22.04

RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2-minimal \
    --mount=type=cache,target=/packages,source=/packages,from=downloader \
    set -eux; \
      apt-get install -y --no-install-recommends \
        ca-certificates="${CA_CERTIFICATES_VERSION}" \
        gosu="${GOSU_VERSION}" \
        tzdata="${TZDATA_VERSION}" \
        "/packages/liquidsoap.deb" \
      ; \
      rm -f \
        /var/lib/dpkg/status-old \
        /var/cache/debconf/templates.dat-old \
        /var/cache/debconf/config.dat-old \
      ;

COPY rootfs /

ENTRYPOINT [ "entrypoint" ]
CMD [ "liquidsoap", "/var/liquidsoap/default.liq" ]

I build them both with the same command in parallel docker build --network host -t "${CONTAINER_BUILD_IMAGE}:${CONTAINER_TAG}" "./containers/${CONTAINER_TAG}", just using different environment variables.

P.S. I haven't figured out how to use AND invalidate cache using files or external registry, so I don't use cache at all when building in CI.

[thaJeztah]

it solves the problem of building multiple images that look the same on the local machine at the same time. In CI, I build each image in a separate namespace so they don't overlap.

I build them both with the same command in parallel (...) just using different environment variables.

The parts that depend on env-vars / build-args should still be invalidated if those parameters are different. I'm curious though why the common steps should be executed again in your situation? If the problem is that the apt-get update is not executed again when changing these options, then it's a better option to combine that with the step that performs the install (i.e., if you want apt-get update to be updated when you update the versions of packages to install).

Can you describe for which steps in your example you expected the cache to be invalidated, but that not happening? (And how they differ between the different Dockerfiles (or same Dockerfile, but different parameters)?

[vitoyucepi]

Can you describe for which steps in your example you expected the cache to be invalidated

I didn't design my dockerfiles for cache invalidation. On localhost I just remove the build cache. In CI I don't use build cache at all.
And now I'm in the process of refactoring my scripts.

[cpuguy83]

Why is it using "type=cache" for mounting the content of "downloader"?
Shouldn't that be "type=bind" (also the default)?