Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache of COPY --from layers is not working as expected. #2615

Open
massimeddu-sj opened this issue Jul 7, 2023 · 10 comments · May be fixed by #2620
Open

Cache of COPY --from layers is not working as expected. #2615

massimeddu-sj opened this issue Jul 7, 2023 · 10 comments · May be fixed by #2620
Labels
area/caching For all bugs related to cache issues feat/cache-copy-layers feat/copy-from kind/bug Something isn't working more-information-needed needs-follow-up priority/p0 Highest priority. Break user flow. We are actively looking at delivering it.

Comments

@massimeddu-sj
Copy link

Actual behavior
I'm building a multi-target Docker image like this one:

FROM xxx as builder

COPY ./src. /src

RUN /src/build.sh

FROM yyy as deploy

COPY --from=builder /src/artifact /usr/local/bin/artifact

With the following build options --compressed-caching=true --cache-copy-layers=true --target deploy --reproducible --cache=true --skip-unused-stages=true --cache-run-layers=true --compression=zstd --compression-level=20 --cleanup --snapshot-mode=redo --use-new-run --log-timestamp=true

Now, if the content of the ./src directory is changed the build is re-run in the builder target, but Kaniko is using an (old) cached version of the COPY --from layer in the deploy target, so the new build is not reflecting the changes in the ./src directory.

Expected behavior
I expect that the kaniko do not use the cached layer but execute the COPY --from command.

To Reproduce
Steps to reproduce the behavior:

  1. Create a multi-target Dockerfile, with the first target has a COPY command and the second target has a COPY --from command
  2. Run kaniko build with --cache-copy-layers=true option
  3. Change the content of the files copied in the first target
  4. Re-run kaniko and see kaniko using the wrong cached layer.

N.B: Removing the --cache-copy-layers=true option fixed the issue.

Additional Information
Kaniko version 1.12.1

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile
@aaron-prindle
Copy link
Collaborator

Possibly related:
#2559

@aaron-prindle
Copy link
Collaborator

aaron-prindle commented Jul 7, 2023

@massimeddu-sonicjobs can you confirm that this is a regression from a previous version of kaniko (issue wasn't in version vX.Y.Z) and if so add the latest version of Kaniko this worked with or is this a bug that has always been the case? Thanks

@aaron-prindle aaron-prindle added area/caching For all bugs related to cache issues priority/p0 Highest priority. Break user flow. We are actively looking at delivering it. regression more-information-needed needs-follow-up feat/copy-from and removed priority/p0 Highest priority. Break user flow. We are actively looking at delivering it. regression labels Jul 7, 2023
@massimeddu-sj
Copy link
Author

Hi @aaron-prindle , thanks for having a look at this issue. I dug a bit into the codebase, and apparently the root cause of the issue is at:

if context.ExcludesFile(p) {

Basically it looks like that when kaniko calculates the cache key, it excludes the files that are in the .dockerignore file. Now, this is the correct behavior for normal COPY command, but is should not be the case for COPY --from commands.

Actually, removing /src/artifact references from my .dockerfile solved the issue and now the cache key is calculated correctly.

I'm not familiar with go and kaniko codebase, but I'll try to create a small PR to solve the issue in the next few days.

Thank you,

@kt315
Copy link
Contributor

kt315 commented Jul 19, 2023

when filename in the .dockerignore docker buildx dont copy to context this file and cant execute COPY .

~/docker_test$ sudo docker buildx build -t blabla .
[+] Building 0.1s (6/7)
 => [internal] load .dockerignore                                         0.0s
 => => transferring context: 49B                                          0.0s
 => [internal] load build definition from Dockerfile                      0.0s
 => => transferring dockerfile: 133B                                      0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04           0.0s
 => [internal] load build context                                         0.0s
 => => transferring context: 2B                                           0.0s
 => [stage1 1/2] FROM docker.io/library/ubuntu:20.04                      0.0s
 => => resolve docker.io/library/ubuntu:20.04                             0.0s
 => ERROR [stage1 2/2] COPY test.txt /                                    0.0s
------
 > [stage1 2/2] COPY test.txt /:
------
error: failed to solve: failed to solve with frontend dockerfile.v0: 
failed to build LLB: failed to compute cache key: "/test.txt" not found: not found




~/docker_test$ tree -a .
.
├── Dockerfile
├── .dockerignore
└── test.txt

0 directories, 3 files



~/docker_test$ cat Dockerfile
FROM ubuntu:20.04 as stage1

COPY test.txt /

FROM ubuntu:20.04

COPY --from=stage1 /test.txt /




~/docker_test$ cat .dockerignore
test.txt

@kt315
Copy link
Contributor

kt315 commented Jul 19, 2023

@massimeddu-sonicjobs can you create repo with example project?

@massimeddu-sj
Copy link
Author

Hi @kt-315 , sure. Please have a look at this example repo: https://github.com/massimeddu-sonicjobs/kaniko-copy-issue-test

@kt315
Copy link
Contributor

kt315 commented Jul 20, 2023

I checked it but dont see mistake. 🤔
When I change test.txt kaniko create new layer in final stage, what am I doing wrong?
log here https://pastebin.com/9MHkGT7C

@rgschmitz1
Copy link

I'm seeing similar issues of older cache being used even when files have been updated, using Kaniko 1.14.0.

The options I provide are --cache=true --cache-ttl=24h --cache-copy-layers=true --target release --skip-unused-stages=true. I can confirm that disabling --cache-copy-layers does partially resolve the issue but I'd expect layers following a COPY (e.g. RUN commands) to be invalidated (cache miss), but this doesn't appear to be happening.

@kt315
Copy link
Contributor

kt315 commented Dec 4, 2023

Hi @rgschmitz1! Can you make example Dockerfile please?

@rgschmitz1
Copy link

rgschmitz1 commented Dec 4, 2023

I can't share my exact source code but I'll try to work out an example, you're going to need a .dockerignore file also.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/caching For all bugs related to cache issues feat/cache-copy-layers feat/copy-from kind/bug Something isn't working more-information-needed needs-follow-up priority/p0 Highest priority. Break user flow. We are actively looking at delivering it.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants