Non-tarball Layers

[jonjohnsonjr]

I often want to upload some bytes to the registry, as is, and not have them be compressed. Or maybe I do want them to be compressed, but they're not a tarball. Sometimes I want to just set a different mediaType.

We don't have great support for that. We can add it, and it's not too hard, but we need to fix up some things:

1. validate assumes layers are tarballs.
2. mutate.Time assumes layers are tarballs.
3. mutate.Extract assumes layers are tarballs.

We probably want to add some support for setting the mediaType in tarball and stream for layers.

We also probably want to add support for disabling gzip altogether.

We may want to automagically append +gzip to the mediaType when compressing it, in certain cases.

Very relevant: #934

X-ref: we should never have put the layer logic in tarball: #413
X-ref: we don't abstract away tarball with Layer: #487

[dekkagaijin]

+1, We're using uncompressed, non-tarball blobs in cosign

[imjasonh]

+1 to removing tarball assumptions. Also +1 to moving func NewStaticLayer([]byte) v1.Layer into ggcr somewhere, I think that would be useful. I'm not sure if it belongs in stream? Maybe pkg/v1 directly?

We've also had some need for a func NewLayerFromFile(os.File) v1.Layer or func NewLayerFromPath(path string) (v1.Layer, error), separate from tarballiness. Maybe also pkg/v1?

[jonjohnsonjr]

I'm not in love with putting more logic in pkg/v1 but I don't have a great idea for a package name that wouldn't collide with lots of variable names.

The best I've come up with would be pkg/v1/static or pkg/v1/buffered which would distinguish it from pkg/v1/stream.

We could have the pkg/v1/tarball layer implementations just alias to pkg/v1/static.

[dprotaso]

v2!

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Keep fresh with the 'lifecycle/frozen' label.

[jonjohnsonjr]

Static layer added in #1093 but we should try to clean up some of those assumptions still.

[bmoylan]

Hi @jonjohnsonjr I'm running into a specific case of this in #1799 where I have a reader for an already-compressed layer (coming from an http request body) and want to upload it. A possible path forward is in #1800 to add another reader to the stream package. Hope you can take a look!

This assumes the caller knows at development-time what these layers are going to include. Alternatives would be to use ggzip.Is to detect whether the reader has the gzip header bytes, or additional stream.Options to indicate what the input will be (instead of the second constructor)

[deitch]

Ah, I was wondering if there is an issue for this (also see #1826 , which is not mine but I think is relevant).

Also, much more importantly.. hi @imjasonh and @jonjohnsonjr 👋

I had to process an index off of a registry that had the buildkit-style attestation. Not getting into whether we like it or not; it is there, so I have to work with it.

It looks like:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:b7796d3d1d633a1abf8d69582d301387e65052ca848432a6631b01c9e6271871",
    "size": 167
  },
  "layers": [
    {
      "mediaType": "application/vnd.in-toto+json",
      "digest": "sha256:4377f521e1eae97005e106fe8d4aa37f2d49ed73c196d78f8297f395fca4be2d",
      "size": 60722,
      "annotations": {
        "in-toto.io/predicate-type": "https://spdx.dev/Document"
      }
    }
  ]
}

When I run validate.Image(), it assumes the layer is gzip, tries to reach it, returns an error.

It could be I am misusing validate.Image(). I am just trying to make sure everything in a given hash exists. i.e. given a v1.ImageIndex, can I check that all of the components are retrievable? I am not actually trying to retrieve them, just check that every hash for a manifest can be retrieved. I do it both for remote registry and for v1-layout.