Cache miss with cargo/rust

[kpcyrd]

Behaviour

I'm trying to do docker build -t something . and use the github actions cache for caching.

Steps to reproduce this issue

FROM rust:1-alpine3.16
ENV RUSTFLAGS="-C target-feature=-crt-static"
RUN apk add --no-cache musl-dev
WORKDIR /app
COPY ./ /app
RUN --mount=type=cache,target=/var/cache/buildkit \
    CARGO_HOME=/var/cache/buildkit/cargo \
    CARGO_TARGET_DIR=/var/cache/buildkit/target \
    cargo build --release --locked && \
    cp -v /var/cache/buildkit/target/release/something .
RUN strip something

jobs:
  build:
    runs-on: ubuntu-22.04
    steps:
    - uses: actions/checkout@v3
    - uses: docker/setup-buildx-action@v2

    - name: Build Docker image
      uses: docker/build-push-action@v3
      with:
        tags: something
        load: true
        cache-from: type=gha
        cache-to: type=gha,mode=max

Expected behaviour

The cache is written to and read from github actions cache, the build finishes quickly.

Actual behaviour

It seems the cache is partially used, the RUN apk ... layer seems to be read from cache but after COPY ./ /app it's a cache miss and cargo build hangs on Updating crates.io index for a long time even though it should be read from cache because of RUN --mount=type=cache,target=/var/cache/buildkit CARGO_HOME=/var/cache/buildkit/cargo ....

Possibly related to #741, #735.

Configuration

• Repository URL (if public): https://github.com/kpcyrd/sh4d0wup
• Build URL (if public): https://github.com/kpcyrd/sh4d0wup/actions/runs/3652932149/jobs/6171852402

name: Docker Image CI

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  build:
    runs-on: ubuntu-22.04
    steps:
    - uses: actions/checkout@v3
    - uses: docker/setup-buildx-action@v2

    - name: Build Docker image
      uses: docker/build-push-action@v3
      with:
        tags: sh4d0wup
        load: true
        cache-from: type=gha
        cache-to: type=gha,mode=max

    - name: Test the Docker image
      run: docker run --rm sh4d0wup --help

    - name: Login to github container registry
      if: github.event_name != 'pull_request'
      uses: docker/login-action@v2
      with:
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Push the image to `edge`
      if: github.event_name != 'pull_request'
      run: |
        docker tag sh4d0wup ghcr.io/kpcyrd/sh4d0wup:edge
        docker push ghcr.io/kpcyrd/sh4d0wup:edge

Logs

logs_44.zip

[kpcyrd]

I started another build with debugging enabled:

jobs:
  build:
    runs-on: ubuntu-22.04
    steps:
    - uses: actions/checkout@v3
    - uses: docker/setup-buildx-action@v2
      with:
        buildkitd-flags: --debug

[...]

Build URL: https://github.com/kpcyrd/sh4d0wup/actions/runs/3653117586/jobs/6172218379

Logs

logs_46.zip

[crazy-max]

@kpcyrd Cache mounts through --mount=type=cache are not exported atm. They are just useful for incremental builds. So if you want this to be exported you have to remove this cache mount. This is tracked in moby/buildkit#1512

@jedevc @dvdksn I'm not sure if we have this properly documented atm?

[dvdksn]

Yeah I don't think we have covered this case. We've got some work to do on run mounts in general, but we could probably mention this in the docs for the gha backend.

[cemo]

What should I have in my docker file and github action configuration to have a proper cache?

[kpcyrd]

I'm also still interested in this, there was some discussion about this problem on reddit recently:

• https://www.reddit.com/r/rust/comments/126xeyx/exploring_the_problem_of_faster_cargo_docker/
• https://www.reddit.com/r/rust/comments/126whnc/better_support_of_docker_layer_caching_in_cargo/