Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Save BuildKit state on client for cache support #138

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Save BuildKit state on client for cache support #138

wants to merge 1 commit into from
+127 −8

Conversation

crazy-max
Copy link
Member

fixes #122

Signed-off-by: CrazyMax crazy-max@users.noreply.github.com

@crazy-max
Save BuildKit state on client for cache support
f3f23a5 
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
@underyx
Copy link

underyx commented Apr 27, 2022

This is the most wonderful PR to ever have graced GitHub with its existence 🥳

@crazy-max
Copy link
Member Author

@underyx

This is the most wonderful PR to ever have graced GitHub with its existence 🥳

Unfortunately there are some issues to copy back from the volume to a local folder for the GitHub Action cache.

One of them is some snapshots have device based permissions: 8/work/work/#b0

    │       │   └── [drwx--x--x root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/8/work
    │       │       └── [d--------- root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/8/work/work
    │       │           └── [c--------- root     root        0 Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/8/work/work/#b0
    │       └── [drwx------ root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/9
    │           ├── [drwxr-xr-x root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/snapshots/snapshots/9/fs

Others are perms on runc-overlayfs dir and cache.db:

├── [-rw------- root     root      64K Apr 22 11:42]  /tmp/.buildkit-state/cache.db
└── [drwx------ root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs
    ├── [drwx------ root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/cachemounts
    ├── [-rw-r--r-- root     root     512K Apr 22 11:43]  /tmp/.buildkit-state/runc-overlayfs/containerdmeta.db
    ├── [drwxrwxrwx root     root     4.0K Apr 22 11:42]  /tmp/.buildkit-state/runc-overlayfs/content

So cache fails to be uploaded:

Post job cleanup.
/usr/bin/tar --posix --use-compress-program zstd -T0 -cf cache.tzst -P -C /home/runner/work/test-docker-action/test-docker-action --files-from manifest.txt
/usr/bin/tar: ../../../../../tmp/.buildkit-state/runc-overlayfs: Cannot open: Permission denied
/usr/bin/tar: ../../../../../tmp/.buildkit-state/cache.db: Cannot open: Permission denied
/usr/bin/tar: Exiting with failure status due to previous errors
Warning: Tar failed with error: The process '/usr/bin/tar' failed with exit code 2

As every actions are executed with a standard user it seems that's something we can't workaround atm.

But even after succeeding to fix the perms issues, loading the cache back and forth from the official cache action that does tar comp/uncomp, it takes more time than the build itself 😞.

After discussing with @tonistiigi about this our best shot is to be able to export cache mounts. See moby/buildkit#1512

@crazy-max crazy-max mentioned this pull request Apr 19, 2023
Open
@shunkakinoki shunkakinoki mentioned this pull request Aug 28, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cache BuildKit state volume
2 participants
@crazy-max @underyx