Detect layer compression #957

tych0 · 2020-06-09T14:45:45Z

A continuation of #947, this changes the OCI/docker backends to only try to compress layer types that they know about.

rhatdan · 2020-06-10T19:28:54Z

directory/directory_dest.go

@@ -100,7 +100,7 @@ func (d *dirImageDestination) SupportsSignatures(ctx context.Context) error {
 	return nil
 }

-func (d *dirImageDestination) DesiredLayerCompression() types.LayerCompression {
+func (d *dirImageDestination) DesiredLayerCompression(info types.BlobInfo) types.LayerCompression {


This is a breaking change to the API. Would be easier to add a new Endpoint. Then we would not have to ref container/image version.

Ok, and then mark this one as Deprecated?

Any thoughts about a name? DesiredBlobCompression()?

Ok, I just updated it per the above.

It’s worse than that: callers of e.g. copy.Image can, in principle, submit their own ImageReference implementation that returns their own ImageDestination implementation, so adding any new methods to ImageDestination is also an API-breaking change. (And there is at least Buildah’s blob caching that actually does provide its own implementation/wrapper.)

We’ve been talking about the need to make ImageDestination and several other types (mostly) private for some time now; between this, the delta support, and enhancing BlobInfoCache, we’ll need to figure this out now.

I’ll read through the various PRs and try to figure out a way.

… but, looking at this PR more, this might not be quite the concern for this one.

Conceptually, whether a MIME type can be compressed is a property of either that MIME type only, or of a combination of that MIME type and the image format ~ manifest schema (types.Image implementation); AFAICS is not really the property of the image transport (e.g. dir: and containers-storage: can accept/supply several different image formats).

So I think this should conceptually be a new method on types.Image, not on ImageDestination, unless there’s something that would make that impractical.

That still has the concern of breaking API by adding extra methods to types.Image, but the only implementation the copy pipeline cares about is the one returned by image.FromUnparsedImage, so we don’t have to add anything to the public API. (The ImageReference.NewImage method is somewhat a historical artifact.)

Tentatively I’ve been thinking:

Define an c/image/internal/image.Image, which embeds c/image/types.Image but adds the necessary new method.

Move the implementation of c/image.sourcedImage to c/image/internal/image; add the necessary new method there. Make the existing public API in c/image/sourced.go compatibility wrappers for the internal versions.

(From a quick look it seems that the actual implementation would be a added to the various c/manifest types, which are not interfaces, so we can add methods safely.)

… but I still need to examine the other PRs, or try doing this myself — there may well be something about this plan that doesn’t work.

So it seems like maybe the best path forward is the original patch?

The original patch still not solving the problem directly.

c/image needs to deal with the API anyway. It’s completely fair that you shouldn’t have to sign up for that work just to get this fix in — I’m going to be working in this area now, but I can’t promise that it will be done by any specific date.

rhatdan · 2020-06-10T19:29:18Z

docker/archive/dest.go

@@ -48,7 +48,7 @@ func newImageDestination(sys *types.SystemContext, ref archiveReference) (types.
 }

 // DesiredLayerCompression indicates if layers must be compressed, decompressed or preserved
-func (d *archiveImageDestination) DesiredLayerCompression() types.LayerCompression {
+func (d *archiveImageDestination) DesiredLayerCompression(info types.BlobInfo) types.LayerCompression {


API Change see above.

rhatdan · 2020-06-10T19:29:26Z

docker/daemon/daemon_dest.go

@@ -88,7 +88,7 @@ func imageLoadGoroutine(ctx context.Context, c *client.Client, reader *io.PipeRe
 }

 // DesiredLayerCompression indicates if layers must be compressed, decompressed or preserved
-func (d *daemonImageDestination) DesiredLayerCompression() types.LayerCompression {
+func (d *daemonImageDestination) DesiredLayerCompression(info types.BlobInfo) types.LayerCompression {


API Change.

tych0 · 2020-06-10T20:11:03Z

As for the test, I can't reproduce. When I run make test-skopeo locally I get:

panic: test timed out after 15m0s

goroutine 540 [running]:
testing.(*M).startAlarm.func1()
	/usr/lib/golang/src/testing/testing.go:1377 +0xdf
created by time.goFunc
	/usr/lib/golang/src/time/sleep.go:168 +0x44

goroutine 1 [chan receive, 14 minutes]:
testing.(*T).Run(0xc00043c400, 0xc5bcd9, 0x4, 0xc92c38, 0x48e1d6)
	/usr/lib/golang/src/testing/testing.go:961 +0x377
testing.runTests.func1(0xc00043c300)
	/usr/lib/golang/src/testing/testing.go:1202 +0x78
testing.tRunner(0xc00043c300, 0xc0001bbdc0)
	/usr/lib/golang/src/testing/testing.go:909 +0xc9
testing.runTests(0xc000470fc0, 0x12372e0, 0x1, 0x1, 0x0)
	/usr/lib/golang/src/testing/testing.go:1200 +0x2a7
testing.(*M).Run(0xc00046c180, 0x0)
	/usr/lib/golang/src/testing/testing.go:1117 +0x176
main.main()
	_testmain.go:44 +0x135

goroutine 9 [chan receive]:
github.com/go-check/check.(*suiteRunner).runTest(...)
	/go/src/github.com/containers/skopeo/vendor/github.com/go-check/check/check.go:819
github.com/go-check/check.(*suiteRunner).run(0xc00046c400, 0xc000470fa0)
	/go/src/github.com/containers/skopeo/vendor/github.com/go-check/check/check.go:624 +0x11d
github.com/go-check/check.Run(0xc2c580, 0xc000470fa0, 0xc000065f08, 0xc00048a000)
	/go/src/github.com/containers/skopeo/vendor/github.com/go-check/check/run.go:92 +0x4d
github.com/go-check/check.RunAll(0xc000065f08, 0x0)
	/go/src/github.com/containers/skopeo/vendor/github.com/go-check/check/run.go:84 +0x96
github.com/go-check/check.TestingT(0xc00043c400)
	/go/src/github.com/containers/skopeo/vendor/github.com/go-check/check/run.go:72 +0x395
github.com/containers/skopeo/integration.Test(0xc00043c400)
	/go/src/github.com/containers/skopeo/integration/check_test.go:18 +0x2b
testing.tRunner(0xc00043c400, 0xc92c38)
	/usr/lib/golang/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
	/usr/lib/golang/src/testing/testing.go:960 +0x350

goroutine 438 [select]:
github.com/go-check/check.(*resultTracker)._loopRoutine(0xc000762120)
	/go/src/github.com/containers/skopeo/vendor/github.com/go-check/check/check.go:470 +0xc4
created by github.com/go-check/check.(*resultTracker).start
	/go/src/github.com/containers/skopeo/vendor/github.com/go-check/check/check.go:450 +0x3f

goroutine 139 [select, 14 minutes]:
net/http.(*persistConn).writeLoop(0xc00038c000)
	/usr/lib/golang/src/net/http/transport.go:2210 +0x123
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1581 +0xb32

goroutine 35 [select, 14 minutes]:
net/http.(*persistConn).readLoop(0xc0000c7c20)
	/usr/lib/golang/src/net/http/transport.go:2032 +0x999
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1580 +0xb0d

goroutine 36 [select, 14 minutes]:
net/http.(*persistConn).writeLoop(0xc0000c7c20)
	/usr/lib/golang/src/net/http/transport.go:2210 +0x123
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1581 +0xb32

goroutine 484 [IO wait, 7 minutes]:
internal/poll.runtime_pollWait(0x7fc83e0ec478, 0x72, 0xffffffffffffffff)
	/usr/lib/golang/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0004b2f18, 0x72, 0x401, 0x400, 0xffffffffffffffff)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0004b2f00, 0xc00013cc00, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/internal/poll/fd_unix.go:169 +0x1cf
os.(*File).read(...)
	/usr/lib/golang/src/os/file_unix.go:259
os.(*File).Read(0xc00001e128, 0xc00013cc00, 0x400, 0x400, 0x1, 0x1, 0xc000068040)
	/usr/lib/golang/src/os/file.go:116 +0x71
github.com/containers/skopeo/integration.consumeAndLogOutputStream.func1(0xd6c120, 0xc00001e128, 0xc00048c2d0, 0xc0001c4840, 0x1e)
	/go/src/github.com/containers/skopeo/integration/utils.go:30 +0x19e
created by github.com/containers/skopeo/integration.consumeAndLogOutputStream
	/go/src/github.com/containers/skopeo/integration/utils.go:22 +0xc2
[lots more spew]

I think the problem is just that there's another zstd mime type somewhere that doesn't match, but I can't really tell. Is there some trick to getting the test suite to run locally?

The various ImageDestinations know what types of layers are supported natively and how they should be compressed, but some (e.g. the OCI distribution repo or the OCI image spec) allow arbitrary blobs to be uploaded. We should not compress these blobs by default, since we don't know what they are and may already be compressed (e.g. squashfs). So, let's introduce a new DesiredBlobCompression() and deprecate DesiredLayerCompression() to allow the various backends to tell the core code about what kind of compression they want. We implement mime-type-based compression detection for the docker and OCI backends. As above, OCI supports arbitrary blobs so we shouldn't interfere with ones we don't understand. The docker backend is intended to be the way we interact with OCI dist spec repos, so we should also not compress unknown media types when talking to the docker backend. Signed-off-by: Tycho Andersen <tycho@tycho.ws>

mtrmac · 2020-06-15T12:37:16Z

As for the test, I can't reproduce. When I run make test-skopeo locally I get:
panic: test timed out after 15m0s

You can use e.g. TESTFLAGS+=" -test.timeout=25m" in hack/make.sh. Ideally, we should instead make the test runs shorter, but that would be mostly in the Skopeo repo.

mtrmac · 2022-02-03T23:18:33Z

More work on handling unknown blobs / SquashFS is now happening in #1408. This specific approach is not mergeable as is due to the API break, and not correctly targeted, so closing this PR.

tych0 force-pushed the detect-layer-compression branch 2 times, most recently from b1dc3f7 to 4fe708e Compare June 9, 2020 18:49

rhatdan approved these changes Jun 10, 2020

View reviewed changes

tych0 force-pushed the detect-layer-compression branch from 4fe708e to d0d543d Compare June 12, 2020 15:56

tych0 force-pushed the detect-layer-compression branch from d0d543d to 4c5851b Compare June 12, 2020 16:09

rchincha mentioned this pull request Nov 17, 2021

squashfs: passthrough squashfs compressed layers #1408

Closed

mtrmac closed this Feb 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect layer compression #957

Detect layer compression #957

tych0 commented Jun 9, 2020

rhatdan Jun 10, 2020

tych0 Jun 10, 2020

tych0 Jun 12, 2020

mtrmac Jun 15, 2020

mtrmac Jun 15, 2020

tych0 Jun 15, 2020

mtrmac Jun 15, 2020

rhatdan Jun 10, 2020

rhatdan Jun 10, 2020

tych0 commented Jun 10, 2020

mtrmac commented Jun 15, 2020

mtrmac commented Feb 3, 2022

Detect layer compression #957

Detect layer compression #957

Conversation

tych0 commented Jun 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tych0 commented Jun 10, 2020

mtrmac commented Jun 15, 2020

mtrmac commented Feb 3, 2022