GHA Cache Not Caching for Pre-Stages (multi-stage builds) #829
Labels
Comments
|
I don't know if this is the issue but git does not create deterministic |
|
That's important and useful info & I'll put together some fixes on my end for that. However, there are several steps prior to EDIT: I just reread my own logs and saw that stages 1-5 are, in fact, cached, so I think you're right and this is an issue with my own setup. Thank you! |
|
I'm sorry, but after digging into this more, I woudl like to re-open, because the logs I posted reveal an issue (if I should open a different task, let me know). In particular, check this out, from the log_read.zip build log: See how it says CACHED but then it runs it anyway? And the run takes LONGER than when it's uncached- here's the same step from log_write.zip: This seems to happen to arbitrary steps- while working on this, I had it happen with a step that literally runs a single command and normally takes less than a second: What does this mean? Why is it happening? |
|
It doesn't pull in the cache only for that command. It needs to pull in all the previous layers as well. Where do you see that step is cached but still executes as well? For the step that runs |
|
Hello, I apologize if this isn't exactly the right place but wanted to piggyback off of this issue for a question I feel might be relevant and potentially a similar issue, and discussions aren't available for this repo. I've successfully gotten a working cache locally (outside of actions context) but it breaks in CI. Effectively, I have a procedural workaround to get docker to cache the compiled Rust dependencies so changes to the codebase don't invalidate the docker cache and trigger all the dependencies to get recompiled from scratch (Dockerfile). Locally, like I said, this works great. I can load a build for one platform, make a small change somewhere, run another build and the caches make it a breeze. In CI, the only time the cache actually gets loaded for majority of the layers is on consecutive runs without any changes. If I make a tiny change to the code, most layers weren't recovered and all the deps get recompiled again. Here is the workflow.yml. If you feel this warrants a separate issue let me know and I'll make one |
|
Following on from #829 (comment) , we are also seeing this same issue with Python. We have a multi-stage build which does setup, copies requirements.txt, runs install etc. Using a standard ie if we do the following diff, our builds take a lot longer because 1. some of the layers arn't cached, and 2. those that are suddenly have a lot more cache layers seemingly downloaded to them: |
|
I see the same behavior, Are there any updates on this issue? |
|
I am also struggling with caching for multi stage builds or multi image builds. I tried both I tried caching both, but for some reason only base image is cached. I see cache being exported for both, but when I re-run The only way I was able to fully cache it, was using local docker instead of So if direct use of You can see my open issue about that: docker/bake-action#154 P.S. If I change it to build as single image without any base image, then it caches properly.. |
|
I also have the same issue when building the Next.js & Turborepo docker file example. It worked great locally but when using in Github actions it always rebuild the images in the first |
|
Interesting issue. I have a similar behavior but as opposed as the author's problem, the pre-stage is properly cached for me but the second part if systematically rebuilt even if there no file change. @moshest I have a Wondering if there is any optimization I could do since the build action takes hella time. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Troubleshooting
Before submitting a bug report please read the Troubleshooting doc.
Behaviour
Steps to reproduce this issue
Expected behaviour
All operations should be cached, since nothing in the build has changed since the previous run
Actual behaviour
The final image's operations are cached, but not the pre-stage where most of the work is
Configuration
Logs
The text was updated successfully, but these errors were encountered: