Implement private.ImageDestination in all transports, improve transport helpers

[mtrmac]

This is a superset of the remaining part of #1439.

• Most importantly, have all transports implement private.ImageDestination, leaving the task of compatibility with public types.ImageDestination to an internal/imagedestination/impl.Compat helper. All transports will now think purely in terms of the private interface.
  • BlobInfoCache is not yet included: that needs to happen, but that will be a later PR (notably because it needs to deal with forwarding between private and public-only users).
• Try to significantly reduce copy&pasted boilerplate in transports. This doesn’t change the provided interface, just the code.
  • Instead of a bunch of methods that just return a single value, consolidate them into imagedestination/impl.Properties, and let the transport just provide the values to return.
  • For possibly-missing features like writing signatures, provide a stub that allows one-line opt-in into a feature, and a stub that provides a short opt-out (without having to individually implement every single method).

The move to private interfaces is, I think, generally desirable. We will soon need to extend the signatures interface, so consolidating the compatibility concerns into a single location will help.

The boilerplate reduction part is RFC. Go is really not well suited handling some of the design concerns, primarily because it allows silent zero-initialization of struct members (all of simple values, structs, and interface pointers), without requiring the user to provide a value or to call a constructor. The previously-used interface with mandatory method implementation is really the only way to force a transport author to deal with a value; impl.Properties weakens that compile-time enforcement. I’m not quite sure that it is worth it.

[mtrmac]

(Updated the PR description after an accidental early submit.)

[mtrmac]

I would very much appreciate an independent opinion. I’m really not sure whether using mix-ins like this is is more succinct and convenient, or surprising and inscrutable.

[vrothberg]

To be honest, I found it exotic at the beginning but started to like it. I still have concerns how approachable the code is for new pairs of eyes though. I think that one central structure with boolean fields would be easier to digest. At the current state, properties are implicitly embedded into the structures.

In other words: I am completely OK to merge it as is. It's a dramatic improvement!

[mtrmac]

I think that one central structure with boolean fields would be easier to digest. At the current state, properties are implicitly embedded into the structures.

I’m not quite sure what this refers to. My best guess is that this says that it’s very hard to find the SupportsPutBlobPartial value for a transport just by text search (quite true), and perhaps that it would be nice to have something broadly like

type private.ImageSource interface {
    HasThreadSafeGetBlob bool
    SupportsPutBlobPartial bool
}

…
func /*docker.*/newImageSource(…) private.ImageSource {
    return &dockerImageSource{
      HasThreadSafeGetBlob: true,
      SupportsPutBlobPartial: true,
    }
}

Is that what you mean? The immediate problem is that Go interfaces only contain methods, not fields. Secondarily, and less importantly, SupportsPutBlobPartial could get out of sync with the actual implementation.[1]

The immediate problem could be handled by making private.ImageSource a struct

type private.ImageSource struct {
    HasThreadSafeGetBlob bool
    SupportsPutBlobPartial bool
    BasicImageSourceOperations // an interface
    PutBlobPartialOperations // an interface, or even a *PutBlobPartialOperations and remove a separate SupportsPutBlobPartial
}

which is quite attractive in some respects, OTOH calling methods would be more annoying, and to an extent this is replacing compile-time interfaces with run-time dynamic function pointers. We would also have a separate private.ImageSource struct and some separate dockerImageSource struct carrying the actual transport’s state, so that doesn’t help tracing the relationship between transport’s properties and methods much.

With the mix-ins embedded into interface implementations, questions like “what implements $method for $transport” are compiler-visible and can be statically answered, though probably hard to follow with an IDE. With a struct pointing to separate interfaces, all of that data only exists dynamically at runtime.

---

[1] The primary motivation for the No/Implements… stubs is to make it a compile-time failure to forget to add a method, or to fail to keep up with an interface change. Without that, it’s IIRC possible to change the type of a parameter in an interface, and implementations suddenly don’t match without any notice, so a run-time check like source.(private.ImageDestinationWithPutBlobPartial) can fail. With the stubs, and a single compile-time type check against a private.ImageDestination, we can guarantee that one of the two stubs are present and that the type signatures are correct.

Admittedly this might be overthinking it — there are only 10 transports after all, and we could just do careful type checks in unit tests, and review API additions carefully. Then we wouldn’t the Supports… methods, and we could just do interface type checks.

---

WRT the non-Supports values, one possible alternative is

type Properties struct {
   HasThreadSafeGetBlob bool
   … 
}

type ImageSource interface {
    func Properties() Properties
}

and then we don’t have a PropertyMethodsInitialize, just a single method implemented in every transport (+ compat.Impl providing the single-value getters in the existing interface).

One downside to this is that actual users of the transport would be more clumsy — everything would be source.Properties().SomeValue. A larger downside is that it would be harder and more expensive to implement “forwarding” transports like oci/archive forwarding to oci/layout, or BlobCache, especially if the forwarding transport only wanted to override some of the values — and the forwarding transport would have to collect all of the values although the caller only wants one of them.

---

I’m fairly certain there is a better design to be found, and I’m very open to suggestions.

[mtrmac]

(Also, the …Initialize convention kind of drives me nuts. It seems somewhat attractive to me: to move that to a run-time check (allow each of those stubs to check if it has been initialized as a private method, and run that check in a unit test of the transport):

type privateSourceInitializationChecks interface {
   initializedPutBlobPartialStub()
   initialized…
}

func (stub ImplementsPutBlobPartial) initializedPutBlobPartialStub() bool {}

func (stub NoPutBlobPartialInitialize) initializedPutBlobPartialStub() bool {
	if !initialized { panic() }
}

func ValidatePrivateSourceImplementation(src private.ImageSource) { // called from a unit test
   inits, ok := src.(privateSourceInitializationChecks)
   inits.initializedPutBlobPartialStub()
   inits.initialized…
}

That has some run-time cost, but much worse it only works for transports where we can ~successfully create the source/destination object in a unit test. Which is awkward for remote clients like docker-daemon.

Oh well, I suppose hating Go might count as a hobby…)

[vrothberg]

Thanks for the detailed elaboration, @mtrmac.

I am supportive of the idea - and think it's a really good one. I very much enjoy the fact that build tests can help review changes :)

[vrothberg]

LGTM

Thanks for putting the changes into easy to digest small(er) commits.

Feel free to merge at will.

[rhatdan]

LGTM