Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support eStargz TOC conversion and run for RAFS v6 #486

Merged
merged 12 commits into from
Jun 22, 2022
Merged

Support eStargz TOC conversion and run for RAFS v6 #486

merged 12 commits into from
Jun 22, 2022

Conversation

imeoer
Copy link
Collaborator

@imeoer imeoer commented Jun 12, 2022

No description provided.

@hsiangkao hsiangkao mentioned this pull request Jun 12, 2022
Closed
@hsiangkao
Copy link
Contributor

hsiangkao commented Jun 12, 2022 •
edited

Moved as a common branch for better cooperation.
related topic: opencontainers/image-spec#877 and #463

@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch 8 times, most recently from a2411ee to 2281c2d Compare June 18, 2022 09:02
@yqleng1987
Copy link
Contributor

@imeoer , Your pull request has been updated. A new test job has been submitted. Please wait in patience.

jiangliu
jiangliu reviewed Jun 18, 2022
View reviewed changes
storage/src/meta/mod.rs Outdated Show resolved Hide resolved
jiangliu
jiangliu reviewed Jun 18, 2022
View reviewed changes
storage/src/meta/mod.rs Outdated Show resolved Hide resolved
jiangliu
jiangliu reviewed Jun 18, 2022
View reviewed changes
storage/src/meta/mod.rs Outdated Show resolved Hide resolved
@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch 4 times, most recently from 253bd20 to 76a5a8e Compare June 21, 2022 03:04
@yqleng1987
Copy link
Contributor

@imeoer , Your pull request has been updated. A new test job has been submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , Test job has been submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , The CI test is completed, please check result:

Test CaseTest Result
merge-target-branch✅SUCCESS
build-docker-image✅SUCCESS
compile-nydus❌FAIL

Sorry, your test job failed. Please get the details in the link.

@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch from 76a5a8e to d8f0f53 Compare June 21, 2022 03:20
@yqleng1987
Copy link
Contributor

@imeoer , Your pull request has been updated. A new test job has been submitted. Please wait in patience.

@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch from a1e1b65 to cf0f18e Compare June 21, 2022 11:44
@yqleng1987
Copy link
Contributor

@imeoer , your pull request has been updated. A new test job will be submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , your test job has passed, and no need to test again.

@imeoer
Copy link
Collaborator Author

imeoer commented Jun 22, 2022

The smoke test passed but docker cache export hang, I think we can move on first. cc @jiangliu

@imeoer imeoer mentioned this pull request Jun 22, 2022
Closed
imeoer added 12 commits June 22, 2022 16:18
@imeoer
smoke: build estargz toc to rafs v6
ebf1dd9 
Add stargz TOC build and run in smoke test for rafs v6.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
builder: support convert estargz toc to rafs v6
271c970 
Set rafs v6's blob meta header for stargz chunks.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
builder: handle estargz's chunk info for rafs v6
d40e9c3 
1. Remove the chunk size limitation RAFS_DEFAULT_CHUNK_SIZE.
2. Fill stargz chunk's uncompressed_offset to 4k aligned.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
builder: support dump blob meta to local disk
b94c254 
Add `--blob-meta` option to allow dump a separate blob meta file
to local disk, so that we can convert stargz TOC to bootstrap and
generate blob meta file for rafs v6 runtime.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
nydusd: support local blob meta file
aa6bd4a 
Allow loading separate blob meta file from local cache directory
for stargz image, this commit just remove some useless checks.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
builder: align to 4k for uncompressed blob meta
f288ff9 
For uncompressed blob meta, keeping 4k alignment to make compatible
with nydusd runtime, allowing runtime to use the blob meta in blob
cache directory directly.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
builder: set chunk size for merge
5cf2155 
Merge should inherit the chunk size stored in parent bootstrap,
not the default value.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
smoke: update stargz & rafs v6 testing by merging method
e2d62de 
Convert all TOC to multiple bootstraps for each stargz layers,
and then merge all bootstrap into final bootstrap. This method
has better performance than loading parent bootstrap layer by
layer.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
storage: extend BlobChunkInfoOndisk struct
c5664e2 
The stargz chunk size is 4MB by default, and we need store
its compressed/uncompressed size into BlobChunkInfoOndisk.

So wen need use reserved 4bits as the high bits of compressed_size
and uncompressed_size.

This change is still compatible with old format, update ut
case to ensure that.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
builder: prepend stargz root entry automatically
1cf927a 
The estargz image format has changed a lot, for a real estargz image,
the TOC file in each layer does not contain root inode, so we need
prepend one before dump bootstrap.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
builder: fix lost blob meta for stargz build
be414e5 
For stargz chunk, it does not provide compressed size, so the
value of `compressed_blob_size` is always 0, we need to check
decompressed_blob_size here.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer
storage: a series fixups for stargz chunks
8b746e9 
For rafs v6 in fscache daemon, we must make compatible with
stargz chunks.

Signed-off-by: Yan Song <yansong.ys@antfin.com>
@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch from cf0f18e to 8b746e9 Compare June 22, 2022 08:21
@yqleng1987
Copy link
Contributor

@imeoer , your pull request has been updated. A new test job will be submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , the test job has been submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , The CI test is completed, please check result:

Test CaseTest Result
merge-target-branch✅SUCCESS
build-docker-image✅SUCCESS
compile-nydus✅SUCCESS
compile-ctr-remote✅SUCCESS
compile-nydus-snapshotter✅SUCCESS
start-nydus-snapshotter-config-containerd✅SUCCESS
run-container-with-nydus-image✅SUCCESS

Congratulations, your test job passed!

@hsiangkao hsiangkao self-requested a review June 22, 2022 14:48
@jiangliu jiangliu merged commit c839875 into master Jun 22, 2022
@jiangliu jiangliu deleted the support-estargz-in-rafs-v6 branch September 30, 2022 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiangliu jiangliu jiangliu approved these changes

@hsiangkao hsiangkao hsiangkao approved these changes

Assignees
No one assigned
Labels
anolis_test_pass
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@imeoer @hsiangkao @yqleng1987 @jiangliu