Add option to enable sparse indexes #3536

Jamstah · 2021-11-23T12:14:25Z

Enable configuration options that can selectively disable validation that dependencies exist within the registry before the image index is uploaded.

This enables sparse indexes, where a registry holds a manifest index that could be signed (so the digest must not change) but does not hold every referenced image in the index. The use case for this is when a registry mirror does not need to mirror all platforms, but does need to maintain the digests of all manifests either because they are signed or because they are pulled by digest.

The registry administrator can also select specific image architectures that must exist in the registry, enabling a registry operator to select only the platforms they care about and ensure all image indexes uploaded to the registry are valid for those platforms.

Closes #3628

dmesser · 2021-11-24T12:13:08Z

One particular aspect of this is the requirement that a sparse index / manifest list has the same digest as the full index / manifest list.

Jamstah · 2021-11-24T13:37:51Z

Hi Daniel!

One particular aspect of this is the requirement that a sparse index / manifest list has the same digest as the full index / manifest list.

Yes, absolutely, but from a PR perspective I would consider that to be a client problem. I've got a PR open for skopeo [1] as an example client implementation and a PR open for the image library [2] to enable a stronger "don't change the digest" option on copies, which once merged could be used by skopeo and oc image mirror.

[1] containers/skopeo#1511
[2] containers/image#1413

configuration/configuration.go

docs/configuration.md

sudo-bmitch · 2021-11-28T01:10:52Z

For what it's worth, we allow "sparse indexes" to be on the registry today via some convoluted means of copying the entire set of images and index, and then deleting the unneeded manifests:

$ regctl manifest get --list localhost:5000/library/busybox:latest
Name:        localhost:5000/library/busybox:latest
MediaType:   application/vnd.docker.distribution.manifest.list.v2+json
Digest:      sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353

Manifests:

  Name:      localhost:5000/library/busybox@sha256:34efe68cca33507682b1673c851700ec66839ecf94d19b928176e20d20e02413
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      localhost:5000/library/busybox@sha256:936a83ff87837f54e46218008fd04ff79bf743bc11b8e9a9773c2a0b11c571e0
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v5

  Name:      localhost:5000/library/busybox@sha256:368f923b2fde7a9fcb461e856ed7807c5fc8bf2524d5feb0365a86288d959d42
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v6

  Name:      localhost:5000/library/busybox@sha256:b98e846c227d038ce59a86f480a57c13eefcd41cc152cfb4a2b8d0c32c423f75
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7

  Name:      localhost:5000/library/busybox@sha256:1bab66c8e73ab581941fc7ba417338214b0373384f1fdf03e02a65c550f818a1
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64/v8

  Name:      localhost:5000/library/busybox@sha256:be0b3422c99de66c5bb4808cd74a30870e68252fdd361d84250f9691797a2665
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/386

  Name:      localhost:5000/library/busybox@sha256:e54f3046086cfda58b89f87763245db08d93418f204629b9e875d089007f0bb9
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/mips64le

  Name:      localhost:5000/library/busybox@sha256:a306c64b6a97a94ed0df1598c4f3a7ed6057393bc114b7e8eb0da67352879725
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/ppc64le

  Name:      localhost:5000/library/busybox@sha256:02eb46ac1dbd9302135da448098a66a33c10a677f5946487185aa29d2a7700ce
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/riscv64

  Name:      localhost:5000/library/busybox@sha256:54339d461e3be5eef03b0a444af67e595c7c9513b18bf6b8f343d52414ce8ec0
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/s390x

$ regctl manifest get localhost:5000/library/busybox@sha256:e54f3046086cfda58b89f87763245db08d93418f204629b9e875d089007f0bb9
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1459,
    "digest": "sha256:cce65d13bdbe6fe8cb81f41ff0a81caa86ddf78f5ea9218b4d3a116b123908e7"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 960456,
      "digest": "sha256:7058006986d854f35d52d4635c8e1a8f4f4ce4aec92df80c467595dfcf15189a"
    }
  ]
}

$ regctl manifest rm localhost:5000/library/busybox@sha256:e54f3046086cfda58b89f87763245db08d93418f204629b9e875d089007f0bb9

$ regctl manifest get localhost:5000/library/busybox@sha256:e54f3046086cfda58b89f87763245db08d93418f204629b9e875d089007f0bb9
Failed to get manifest localhost:5000/library/busybox@sha256:e54f3046086cfda58b89f87763245db08d93418f204629b9e875d089007f0bb9: Not found [http 404]: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"library/busybox","Revision":"sha256:e54f3046086cfda58b89f87763245db08d93418f204629b9e875d089007f0bb9"}}]}


$ regctl manifest get --list localhost:5000/library/busybox:latest
Name:        localhost:5000/library/busybox:latest
MediaType:   application/vnd.docker.distribution.manifest.list.v2+json
Digest:      sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353

Manifests:

  Name:      localhost:5000/library/busybox@sha256:34efe68cca33507682b1673c851700ec66839ecf94d19b928176e20d20e02413
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      localhost:5000/library/busybox@sha256:936a83ff87837f54e46218008fd04ff79bf743bc11b8e9a9773c2a0b11c571e0
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v5

  Name:      localhost:5000/library/busybox@sha256:368f923b2fde7a9fcb461e856ed7807c5fc8bf2524d5feb0365a86288d959d42
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v6

  Name:      localhost:5000/library/busybox@sha256:b98e846c227d038ce59a86f480a57c13eefcd41cc152cfb4a2b8d0c32c423f75
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7

  Name:      localhost:5000/library/busybox@sha256:1bab66c8e73ab581941fc7ba417338214b0373384f1fdf03e02a65c550f818a1
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64/v8

  Name:      localhost:5000/library/busybox@sha256:be0b3422c99de66c5bb4808cd74a30870e68252fdd361d84250f9691797a2665
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/386

  Name:      localhost:5000/library/busybox@sha256:e54f3046086cfda58b89f87763245db08d93418f204629b9e875d089007f0bb9
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/mips64le

  Name:      localhost:5000/library/busybox@sha256:a306c64b6a97a94ed0df1598c4f3a7ed6057393bc114b7e8eb0da67352879725
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/ppc64le

  Name:      localhost:5000/library/busybox@sha256:02eb46ac1dbd9302135da448098a66a33c10a677f5946487185aa29d2a7700ce
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/riscv64

  Name:      localhost:5000/library/busybox@sha256:54339d461e3be5eef03b0a444af67e595c7c9513b18bf6b8f343d52414ce8ec0
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/s390x

This does create some weird issues though, like not being able to push the same manifest with a different tag. And many clients will report the 404 error on pulling the platform manifest while that tag is listed in API's and probably most GUI's, leading to confusing behavior for users.

Allowing a sparse index is going to create some issues for tooling that isn't expecting this, e.g. tools that do a full copy of an index and images. I'd also question how vulnerability scanners should react if parts of the image cannot be pulled to scan.

Overall, I'm interested in this feature and would be very likely to add it to regclient/regsync for more efficient image mirroring. But I'd want to move cautiously considering the potential issues it will create.

Jamstah · 2021-11-28T10:52:00Z

Many clients will report the 404 error on pulling the platform manifest while that tag is listed in API's and probably most GUI's, leading to confusing behavior for users.

I agree, its one of the reasons I thought this should go in as an optional config option, even though I think it makes sense as the default behaviour. On my list next is to go around the clients (cmdline and CRI) and see how they respond, and raise bugs/prs where the UX isn't as good as it could be. I didn't want to raise those without a supporting registry, and preferably this too: opencontainers/distribution-spec#310

Allowing a sparse index is going to create some issues for tooling that isn't expecting this, e.g. tools that do a full copy of an index and images. I'd also question how vulnerability scanners should react if parts of the image cannot be pulled to scan.

Two good points here, I hadn't considered vuln scanners (will have to try those) and copying when the index is sparse. I already raised something for skopeo to enable sparse copying, but will need to look at another PR to handle copying when the index is already sparse.

Overall, I'm interested in this feature and would be very likely to add it to regclient/regsync for more efficient image mirroring. But I'd want to move cautiously considering the potential issues it will create.

👍

Jamstah · 2021-12-07T10:15:32Z

Hi to all subscribed.

Based on feedback around the configuration options, I've revisited this one to change the language in the config and to add the suggestion from @SteveLasker to validate a subset of platforms.

Also converted to draft as I haven't finished writing the tests yet, but wanted to get it out for an initial look to make sure people agree with the approach.

Jamstah · 2022-03-22T13:50:28Z

This took me longer to get back to than I hoped, but bringing this back again for review. I've extended the functionality to be able to validate a subset of platforms.

Because the commit is substantially different I resolved all the previous conversations too.

Skopeo from 1.6.0 includes the command-line option (--multi-arch index-only) for being able to push image indexes unaltered without also first copying the referenced, which allows for testing this.

codecov-commenter · 2022-03-27T17:05:53Z

Codecov Report

Merging #3536 (60c12c7) into main (3413bf8) will increase coverage by 0.15%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##             main    #3536      +/-   ##
==========================================
+ Coverage   57.43%   57.59%   +0.15%     
==========================================
  Files         105      105              
  Lines       10839    10889      +50     
==========================================
+ Hits         6225     6271      +46     
- Misses       3930     3934       +4     
  Partials      684      684

Impacted Files	Coverage Δ
registry/handlers/app.go	`47.61% <42.85%> (-0.74%)`	⬇️
configuration/configuration.go	`64.15% <61.53%> (-0.24%)`	⬇️
registry/storage/manifestlisthandler.go	`68.57% <100.00%> (+25.71%)`	⬆️
registry/storage/registry.go	`89.47% <100.00%> (+0.79%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Jamstah · 2022-04-04T14:24:16Z

I thought I'd look at the client output for pulling a sparse manifest list where the platform image is missing. I copied the latest alpine index and s390x image to my registry, leaving out the other platforms, and am now attempting to use the manifest list on amd64.

Overall, everything fails in sensible ways but the error messages could be improved in the clients.

skopeo inspect - error message could be clearer but you can tell its looking for the "target platform"

[jammy@ibm007470 registry]$ skopeo inspect --raw docker://localhost:5000/alpine:latest
{"manifests":[{"digest":"sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801","mediaType":"application\/vnd.docker.distribution.manifest.v2+json","platform":{"architecture":"amd64","os":"linux"},"size":528},{"digest":"sha256:1c900f718a091aa74ef0464fd33540ba1b7c0406a7fc5a438852c701ff58d42e","mediaType":"application\/vnd.docker.distribution.manifest.v2+json","platform":{"architecture":"arm","os":"linux","variant":"v6"},"size":528},{"digest":"sha256:5d4a549d1a0d42c790cb7b848485404e6d32a1f6256d6174adda83c300a0456b","mediaType":"application\/vnd.docker.distribution.manifest.v2+json","platform":{"architecture":"arm","os":"linux","variant":"v7"},"size":528},{"digest":"sha256:d72e2d383f2d5fb1e8186ebfd1fbb22a87c04f52ac12fc379d21abb368d373df","mediaType":"application\/vnd.docker.distribution.manifest.v2+json","platform":{"architecture":"arm64","os":"linux","variant":"v8"},"size":528},{"digest":"sha256:b66bccf2e0cca8e5fb79f7d3c573dd76c4787d1d883f5afe6c9d136a260bba07","mediaType":"application\/vnd.docker.distribution.manifest.v2+json","platform":{"architecture":"386","os":"linux"},"size":528},{"digest":"sha256:b13eb315f8d65585548a683bf610c9a1f21a5903f6889aa0bac78a1bc1a7e907","mediaType":"application\/vnd.docker.distribution.manifest.v2+json","platform":{"architecture":"ppc64le","os":"linux"},"size":528},{"digest":"sha256:1c3a5f2d660d191dd74de5a67007c0180f52e33dbd7a2d3e1132866df30fae3a","mediaType":"application\/vnd.docker.distribution.manifest.v2+json","platform":{"architecture":"s390x","os":"linux"},"size":528}],"mediaType":"application\/vnd.docker.distribution.manifest.list.v2+json","schemaVersion":2}
[jammy@ibm007470 registry]$ skopeo inspect docker://localhost:5000/alpine:latest
FATA[0000] Error parsing manifest for image: loading manifest for target platform: reading manifest sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801 in localhost:5000/alpine: manifest unknown: manifest unknown 
[jammy@ibm007470 registry]$ skopeo inspect docker://localhost:5000/alpine:missing
FATA[0000] Error parsing image name "docker://localhost:5000/alpine:missing": reading manifest missing in localhost:5000/alpine: manifest unknown: manifest unknown

skopeo copy - It will fail on the first missing image when trying to copy them all, would be better to copy those that exist. Doesn't mention platform when pulling based on system, which could be improved.

[jammy@ibm007470 registry]$ skopeo copy docker://localhost:5000/alpine:latest docker://localhost:5000/copytarget:latest
FATA[0000] determining manifest MIME type for docker://localhost:5000/alpine:latest: reading manifest sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801 in localhost:5000/alpine: manifest unknown: manifest unknown 
[jammy@ibm007470 registry]$ skopeo copy docker://localhost:5000/alpine:missing docker://localhost:5000/copytarget:latest
FATA[0000] initializing source docker://localhost:5000/alpine:missing: reading manifest missing in localhost:5000/alpine: manifest unknown: manifest unknown 
[jammy@ibm007470 registry]$ skopeo copy docker://localhost:5000/alpine:latest docker://localhost:5000/copytarget:latest --multi-arch all
Getting image list signatures
Copying 7 of 7 images in list
Copying image sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801 (1/7)
FATA[0000] determining manifest MIME type for docker://localhost:5000/alpine:latest: reading manifest sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801 in localhost:5000/alpine: manifest unknown: manifest unknown

docker pull - error is different, but doens't mention platform or a manifest list.

[jammy@ibm007470 registry]$ docker pull localhost:5000/alpine:latest
latest: Pulling from alpine
manifest for localhost:5000/alpine:latest not found: manifest unknown: manifest unknown
[jammy@ibm007470 registry]$ docker pull localhost:5000/alpine:missing
Error response from daemon: manifest for localhost:5000/alpine:missing not found: manifest unknown: manifest unknown

docker build - as per docker pull

[jammy@ibm007470 df]$ docker build -t whatever .
Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM localhost:5000/alpine:latest
latest: Pulling from alpine
manifest for localhost:5000/alpine:latest not found: manifest unknown: manifest unknown
[jammy@ibm007470 df]$ vi Dockerfile
[jammy@ibm007470 df]$ docker build -t whatever .
Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM localhost:5000/alpine:missing
manifest for localhost:5000/alpine:missing not found: manifest unknown: manifest unknown

podman pull - as per skopeo copy

[jammy@ibm007470 df]$ podman pull localhost:5000/alpine:latest
Trying to pull localhost:5000/alpine:latest...
Error: determining manifest MIME type for docker://localhost:5000/alpine:latest: reading manifest sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801 in localhost:5000/alpine: manifest unknown: manifest unknown
[jammy@ibm007470 df]$ podman pull localhost:5000/alpine:missing
Trying to pull localhost:5000/alpine:missing...
Error: initializing source docker://localhost:5000/alpine:missing: reading manifest missing in localhost:5000/alpine: manifest unknown: manifest unknown

podman build - as per podman pull

[jammy@ibm007470 df]$ podman build -t whatever .
STEP 1/1: FROM localhost:5000/alpine:latest
Trying to pull localhost:5000/alpine:latest...
Error: error creating build container: determining manifest MIME type for docker://localhost:5000/alpine:latest: reading manifest sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801 in localhost:5000/alpine: manifest unknown: manifest unknown
[jammy@ibm007470 df]$ vi Dockerfile
[jammy@ibm007470 df]$ podman build -t whatever .
STEP 1/1: FROM localhost:5000/alpine:missing
Trying to pull localhost:5000/alpine:missing...
Error: error creating build container: initializing source docker://localhost:5000/alpine:missing: reading manifest missing in localhost:5000/alpine: manifest unknown: manifest unknown

buildah - as per skopeo copy

[jammy@ibm007470 df]$ buildah from localhost:5000/alpine:latest
Trying to pull localhost:5000/alpine:latest...
determining manifest MIME type for docker://localhost:5000/alpine:latest: reading manifest sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801 in localhost:5000/alpine: manifest unknown: manifest unknown
ERRO[0000] exit status 125                              
[jammy@ibm007470 df]$ buildah from localhost:5000/alpine:missing
Trying to pull localhost:5000/alpine:missing...
initializing source docker://localhost:5000/alpine:missing: reading manifest missing in localhost:5000/alpine: manifest unknown: manifest unknown
ERRO[0000] exit status 125

grype - doesn't mention platforms in the error message

grype doesn't seem to have options specifically for manifest lists anyway, and will only scan the system image. There are calls to improve already: anchore/grype#485

[jammy@ibm007470 Downloads]$ grype registry:localhost:5000/alpine:latest
 ✔ Vulnerability DB        [updated]
[0000]  WARN scheme "registry" specified, but it coincides with a common image name; re-examining user input "registry:localhost:5000/alpine:latest" without scheme parsing because image retrieval using scheme parsing was unsuccessful: unable to use OciRegistry source: failed to get image from registry: GET http://localhost:5000/v2/alpine/manifests/sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801: MANIFEST_UNKNOWN: manifest unknown; map[Name:alpine Revision:sha256:1e014f84205d569a5cc3be4e108ca614055f7e21d11928946113ab3f36054801] from-lib=syft
1 error occurred:
	* failed to catalog: could not fetch image "localhost:5000/alpine:latest": unable to use DockerDaemon source: unable to inspect existing image: Error response from daemon: no such image: registry:localhost:5000/alpine:latest: invalid reference format

[jammy@ibm007470 Downloads]$ grype registry:localhost:5000/alpine:missing
 ⠹ Vulnerability DB        [checking for update]
[0000]  WARN scheme "registry" specified, but it coincides with a common image name; re-examining user input "registry:localhost:5000/alpine:missing" without scheme parsing because image retrieval using scheme parsing was unsuccessful: unable to use OciRegistry source: failed to get image descriptor from registry: GET http://localhost:5000/v2/alpine/manifests/missing: MANIFEST_UNKNOWN: manifest unknown; map[Tag:missing] from-lib=syft
1 error occurred:
	* failed to catalog: could not fetch image "localhost:5000/alpine:missing": unable to use DockerDaemon source: unable to inspect existing image: Error response from daemon: no such image: registry:localhost:5000/alpine:missing: invalid reference format

Jamstah · 2022-04-06T11:13:36Z

@dmage @bparees as you both very kindly reviewed this for me for my first version, any chance you'd be happy to review again?

bparees

i've added some comments on the proposed api, but i should note that i am not a contributor to this repo, so i don't expect my input to carry any particular weight, i may be overlooking general best practices/guidance/philosophies that this repo follows for its apis.

configuration/configuration.go

docs/configuration.md

Jamstah · 2022-04-06T21:15:56Z

i've added some comments on the proposed api, but i should note that i am not a contributor to this repo, so i don't expect my input to carry any particular weight, i may be overlooking general best practices/guidance/philosophies that this repo follows for its apis.

Thanks for the review, right now I'm just trying to get something that feels right so any input is valued!

Jamstah · 2023-10-02T14:59:20Z

@milosgajdos @deleteriousEffect I have added a warning to say that this is an experimental option to help with original concerns around the graph. I'd really like to get this one in. If you still have concerns can you please state them so we can discuss?

milosgajdos · 2023-10-02T21:44:18Z

I would love to give this another thought and review soon

Jamstah · 2023-10-10T10:40:07Z

As you've been active recently, @thaJeztah and @davidspek I'd love a review on this one :)

milosgajdos · 2023-10-24T16:07:06Z

Hey @neersighted, would you mind weighing into this PR, please?

milosgajdos · 2023-11-29T22:12:55Z

This needs a rebase @Jamstah also, friendly ping to @neersighted 😄

milosgajdos

This LGTM and I feel would be valuable to end users, but I've some configuration UX/stylistic questions/suggestions we could discuss before we move forward with this.

Curious what you think @Jamstah

docs/content/about/configuration.md

neersighted · 2023-12-06T13:22:09Z

I'm still personally not experienced with distribution/distribution's code to say whether or not this is a good idea for this codebase... I think that as long as the maintainers don't forsee integrity/architectural issues arising from intentionally allowing sparse indexes (something that can already arise organically today, as I understand it) that this very much makes sense and is in demand by a small but significant minority of users.

milosgajdos · 2023-12-06T13:29:20Z

I think that as long as the maintainers don't forsee integrity/architectural issues arising from intentionally allowing sparse indexes (something that can already arise organically today, as I understand it) that this very much makes sense and is in demand by a small but significant minority of users.

The key in adding this in is, it's entirely optional by the operators themselves. It's not enforced - so if say user X makes a deliberate decision to do this, knowing that the integrity of the store will not be maintained, then IMHO they should be ok to make that decision. We need to make sure this is properly highlighted in the docs

In other words: this isn't forced upon anyone - it's entirely optional.

Enable configuration options that can selectively disable validation that dependencies exist within the registry before the image index is uploaded. This enables sparse indexes, where a registry holds a manifest index that could be signed (so the digest must not change) but does not hold every referenced image in the index. The use case for this is when a registry mirror does not need to mirror all platforms, but does need to maintain the digests of all manifests either because they are signed or because they are pulled by digest. The registry administrator can also select specific image architectures that must exist in the registry, enabling a registry operator to select only the platforms they care about and ensure all image indexes uploaded to the registry are valid for those platforms. Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>

Jamstah · 2023-12-19T16:59:40Z

@milosgajdos ready for review again now I finally got to the config reshuffle.

milosgajdos

I had a look at this today and left some comments. Sorry for dragging this @Jamstah. I'm all in for this feature to get in, just to make it easier to understand 😄

registry/handlers/app.go

milosgajdos · 2023-12-22T11:15:12Z

registry/storage/manifestlisthandler.go

@@ -74,7 +76,7 @@ func (ms *manifestListHandler) Put(ctx context.Context, manifestList distributio
 func (ms *manifestListHandler) verifyManifest(ctx context.Context, mnfst distribution.Manifest, skipDependencyVerification bool) error {
 	var errs distribution.ErrManifestVerification

-	if !skipDependencyVerification {
+	if ms.validateImagesExist && !skipDependencyVerification {


suggestion: should we swap these two so we break the logic circuit on skipDependencyVerification by default

For what purpose? I'm not sure it matters.

registry/handlers/app.go

registry/storage/manifeststore_test.go

registry/storage/registry.go

milosgajdos · 2023-12-22T12:02:01Z

registry/storage/manifestlisthandler.go

@@ -101,3 +107,20 @@ func (ms *manifestListHandler) verifyManifest(ctx context.Context, mnfst distrib

 	return nil
 }
+
+func (ms *manifestListHandler) platformMustExist(descriptor distribution.Descriptor) bool {


I'm finding it a bit hard to follow the boolean logic in the flow of verifyManifest that calls this func but maybe it's the naming of the functions and the boolean logic I must do when reading the the func flow that slightly confuse me.

I thought the validateImagesExistPlatforms list would work as follows:

if validateImagesExistPlatforms is empty then I'd expect we are saying no platform needs to exist in which case I'd expect the if len(validateImagesExistPlatforms) == 0 to return false which would then lead to !ms.platformMustExist(manifestDescriptor) skipping the validation 🤔

Maybe renaming this func to platformSkipExist and changing the logic would make it easier to read/understand to me personally

I think switching it to platformSkipExist would make it even more confusing.

I've added a comment and changed the check to not use continue, which I think helps with clarity. wdyt?

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>

Jamstah · 2024-02-06T23:41:27Z

@milosgajdos thakns for the further review, somehow I completely missed it. Good suggestions, mostly implemented, as above.

No hurry, I know you're out at the moment :)

milosgajdos

This change looks good to me and I don't want to keep blocking this PR. PTAL @thaJeztah

milosgajdos · 2024-05-01T14:22:51Z

CC: @thaJeztah @squizzi @corhere you folks mind having a look at this? It's been sitting here a while; would be good put it into the beta release

bparees reviewed Nov 24, 2021

View reviewed changes

configuration/configuration.go Outdated Show resolved Hide resolved

Jamstah force-pushed the sparse-indexes branch from 2a72348 to fa5d3cb Compare November 24, 2021 16:29

dmage reviewed Nov 25, 2021

View reviewed changes

configuration/configuration.go Outdated Show resolved Hide resolved

dmage reviewed Nov 25, 2021

View reviewed changes

docs/configuration.md Outdated Show resolved Hide resolved

Jamstah force-pushed the sparse-indexes branch from fa5d3cb to 166fcf2 Compare November 25, 2021 18:12

Jamstah force-pushed the sparse-indexes branch from 166fcf2 to f89f05b Compare December 7, 2021 10:12

Jamstah marked this pull request as draft December 7, 2021 10:12

Jamstah force-pushed the sparse-indexes branch from f89f05b to 311ea2e Compare February 8, 2022 15:52

Jamstah force-pushed the sparse-indexes branch 3 times, most recently from 52a020c to 52bc959 Compare March 22, 2022 13:44

Jamstah marked this pull request as ready for review March 22, 2022 13:50

Jamstah force-pushed the sparse-indexes branch 4 times, most recently from 21d8e19 to fd8909b Compare March 27, 2022 16:57

Jamstah mentioned this pull request Apr 6, 2022

Improve error messages when system image missing containers/skopeo#1615

Closed

bparees reviewed Apr 6, 2022

View reviewed changes

configuration/configuration.go Outdated Show resolved Hide resolved

configuration/configuration.go Outdated Show resolved Hide resolved

docs/configuration.md Outdated Show resolved Hide resolved

Jamstah force-pushed the sparse-indexes branch 2 times, most recently from daeef49 to 3f33878 Compare April 10, 2022 17:07

Jamstah force-pushed the sparse-indexes branch from b7e64b7 to fd46d84 Compare October 2, 2023 14:56

Jamstah force-pushed the sparse-indexes branch 2 times, most recently from ab46a89 to 4e5267a Compare October 2, 2023 15:39

Jamstah force-pushed the sparse-indexes branch from 4e5267a to 337d133 Compare October 17, 2023 14:58

Jamstah force-pushed the sparse-indexes branch from 337d133 to 165b247 Compare December 1, 2023 10:54

milosgajdos reviewed Dec 6, 2023

View reviewed changes

docs/content/about/configuration.md Outdated Show resolved Hide resolved

docs/content/about/configuration.md Outdated Show resolved Hide resolved

Jamstah force-pushed the sparse-indexes branch from 165b247 to 2b3ce73 Compare December 11, 2023 15:38

Jamstah added 2 commits December 11, 2023 15:41

Refactor config

cac474e

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>

Jamstah force-pushed the sparse-indexes branch from 2b3ce73 to cac474e Compare December 11, 2023 15:42

Shift configuration hierarchy to be more sensible

40941a4

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>

milosgajdos reviewed Dec 22, 2023

View reviewed changes

Review comments

b7ae47b

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>

github-actions bot added area/config Related to registry config area/docs area/api labels Feb 6, 2024

milosgajdos approved these changes Mar 2, 2024

View reviewed changes

milosgajdos requested review from corhere, squizzi and thaJeztah May 1, 2024 20:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to enable sparse indexes #3536

Add option to enable sparse indexes #3536

Jamstah commented Nov 23, 2021 •

edited

dmesser commented Nov 24, 2021

Jamstah commented Nov 24, 2021

sudo-bmitch commented Nov 28, 2021

Jamstah commented Nov 28, 2021 •

edited

Jamstah commented Dec 7, 2021 •

edited

Jamstah commented Mar 22, 2022

codecov-commenter commented Mar 27, 2022 •

edited

Jamstah commented Apr 4, 2022 •

edited

Jamstah commented Apr 6, 2022

bparees left a comment

Jamstah commented Apr 6, 2022

Jamstah commented Oct 2, 2023

milosgajdos commented Oct 2, 2023

Jamstah commented Oct 10, 2023

milosgajdos commented Oct 24, 2023

milosgajdos commented Nov 29, 2023

milosgajdos left a comment

neersighted commented Dec 6, 2023

milosgajdos commented Dec 6, 2023

Jamstah commented Dec 19, 2023

milosgajdos left a comment

milosgajdos Dec 22, 2023

Jamstah Feb 6, 2024

milosgajdos Dec 22, 2023

Jamstah Feb 6, 2024

Jamstah commented Feb 6, 2024

milosgajdos left a comment

milosgajdos commented May 1, 2024

Add option to enable sparse indexes #3536

Are you sure you want to change the base?

Add option to enable sparse indexes #3536

Conversation

Jamstah commented Nov 23, 2021 • edited

dmesser commented Nov 24, 2021

Jamstah commented Nov 24, 2021

sudo-bmitch commented Nov 28, 2021

Jamstah commented Nov 28, 2021 • edited

Jamstah commented Dec 7, 2021 • edited

Jamstah commented Mar 22, 2022

codecov-commenter commented Mar 27, 2022 • edited

Codecov Report

Jamstah commented Apr 4, 2022 • edited

Jamstah commented Apr 6, 2022

bparees left a comment

Choose a reason for hiding this comment

Jamstah commented Apr 6, 2022

Jamstah commented Oct 2, 2023

milosgajdos commented Oct 2, 2023

Jamstah commented Oct 10, 2023

milosgajdos commented Oct 24, 2023

milosgajdos commented Nov 29, 2023

milosgajdos left a comment

Choose a reason for hiding this comment

neersighted commented Dec 6, 2023

milosgajdos commented Dec 6, 2023

Jamstah commented Dec 19, 2023

milosgajdos left a comment

Choose a reason for hiding this comment

milosgajdos Dec 22, 2023

Choose a reason for hiding this comment

Jamstah Feb 6, 2024

Choose a reason for hiding this comment

milosgajdos Dec 22, 2023

Choose a reason for hiding this comment

Jamstah Feb 6, 2024

Choose a reason for hiding this comment

Jamstah commented Feb 6, 2024

milosgajdos left a comment

Choose a reason for hiding this comment

milosgajdos commented May 1, 2024

Jamstah commented Nov 23, 2021 •

edited

Jamstah commented Nov 28, 2021 •

edited

Jamstah commented Dec 7, 2021 •

edited

codecov-commenter commented Mar 27, 2022 •

edited

Jamstah commented Apr 4, 2022 •

edited