introduce publish (alpha) command #10949

ndeloof · 2023-08-30T08:34:48Z

This introduce new docker compose (alpha) publish REPOSITORY command to publish a compose.yaml file as an OCI artifact, and support for docker:<REPOSITORY> remote resource to be used with include

illustration example:

include:
  - docker:ndeloof/test:demo

$ COMPOSE_EXPERIMENTAL_OCI_REMOTE=true docker compose config --no-normalize
name: truc
services:
  test:
    image: nginx

note: most of the changes in this PR is about passing dockerCli to the project loader

codecov · 2023-08-30T08:54:07Z

Codecov Report

Patch coverage: 45.37% and project coverage change: -0.57% ⚠️

Comparison is base (aeb835a) 58.30% compared to head (057c273) 57.73%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10949      +/-   ##
==========================================
- Coverage   58.30%   57.73%   -0.57%     
==========================================
  Files         124      127       +3     
  Lines       10912    11037     +125     
==========================================
+ Hits         6362     6372      +10     
- Misses       3922     4034     +112     
- Partials      628      631       +3

Files Changed	Coverage Δ
pkg/api/api.go	`28.84% <ø> (ø)`
pkg/compose/publish.go	`0.00% <0.00%> (ø)`
pkg/remote/git.go	`1.69% <0.00%> (-8.31%)`	⬇️
pkg/remote/oci.go	`2.46% <2.46%> (ø)`
cmd/compose/events.go	`28.20% <16.66%> (ø)`
cmd/compose/kill.go	`48.14% <25.00%> (ø)`
cmd/compose/push.go	`41.37% <25.00%> (ø)`
cmd/compose/viz.go	`51.02% <25.00%> (ø)`
cmd/compose/watch.go	`31.03% <25.00%> (ø)`
cmd/compose/config.go	`33.95% <28.57%> (-0.19%)`	⬇️
... and 26 more

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

milas

Logic LGTM, just a couple higher-level things

cmd/compose/compose.go

milas · 2023-08-30T13:12:42Z

pkg/remote/oci.go

+}
+
+func (g ociRemoteLoader) Accept(path string) bool {
+	return strings.HasPrefix(path, "docker:")


I believe elsewhere (e.g. build) we've used docker-image:// as the format

I guess this isn't technically an image, but a new artifact type, at which point I wonder if oci:// would be better?

(I know flux2 uses that format for example: https://fluxcd.io/flux/cheatsheets/oci-artifacts/#consuming-artifacts)

oci would be a better terminology from a pure implementation point of view, but I don't expect more than 1% Docker users know about Open Container Initiative, and just have a basic understanding of Docker images

Definitely agree that most people don't know what OCI is 😛

I would still say that:

We should at least use a URL scheme, i.e. foo://bar instead of foo:bar, being able to do a cheap check for :// to distinguish between local & remote (without awareness of loader) is a helpful property

docker isn't really descriptive and could mean many different things/is un-Googleable, but "Docker Compose OCI include" will [hopefully] get results

helm also uses oci:// for modern helm push/pull

milas · 2023-09-01T18:37:13Z

cmd/compose/compose.go

 	if err != nil {
 		return "", err
 	}
 	return project.Name, nil
 }

-func (o *ProjectOptions) ToProject(services []string, po ...cli.ProjectOptionsFn) (*types.Project, error) {
+func (o *ProjectOptions) ToProject(dockerCli command.Cli, services []string, po ...cli.ProjectOptionsFn) (*types.Project, error) {
+	if !o.Offline {


Oh this was a very elegant solution! ✨

milas · 2023-09-01T18:38:58Z

Oh no, I caused a bunch of conflicts with the build changes...sorry 😭

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

glours

Sounds good to me

milas

LGTM, few last things but mostly the distribution/distribution/v3 dep that snuck back 😅

milas · 2023-09-06T14:50:47Z

go.mod

@@ -50,6 +50,8 @@ require (
 	gotest.tools/v3 v3.5.0
 )

+require github.com/distribution/distribution/v3 v3.0.0-20230214150026-36d8c594d7aa


See #10954 - will need to swap things to github.com/distribution/reference to keep this from reappearing.

milas · 2023-09-06T14:52:07Z

pkg/api/api.go

@@ -74,6 +74,8 @@ type Service interface {
 	Events(ctx context.Context, projectName string, options EventsOptions) error
 	// Port executes the equivalent to a `compose port`
 	Port(ctx context.Context, projectName string, service string, port uint16, options PortOptions) (string, int, error)
+	// Publish executes the equivalent to a `compose publish`
+	Publish(ctx context.Context, project *types.Project, repository string) error


Minor: we often end up refactoring the API methods to add an options object later...maybe worth having PublishOptions with a Repository string field upfront

e.g. I'm guessing this will need Quiet bool at some point

Repository being mandatory parameter should not be part of the Options struct imho

milas · 2023-09-06T14:53:36Z

pkg/remote/oci.go

+	offline   bool
+}
+
+const prefix = "oci:"


Shouldn't this be oci://?

This is not an URL, just a prefix we decide on our own. There's no canonical URL style for OCI artifacts, as this ends being transported over http

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

jaydrogers · 2024-02-08T21:46:46Z

This feature seems very exciting! I found the documentation on this feature (https://docs.docker.com/engine/reference/commandline/compose_alpha_publish/) but are there more resources to learn how to use it and play around with it?

My questions:

Do I need to do anything special to create a template?
When I publish it, where does it go?
How can I have someone else use it?
Are there plans for it to work with Swarm?

ndeloof · 2024-02-09T12:47:28Z

docker compose -f <your compose file> alpha publish <registry>/<repo>:<tag>

then you can use this published model from another compose file using include: - oci:<registry>/<repo>:<tag>

Are there plans for it to work with Swarm?

swarm doesn't use the compose specification nor compose-go to parse compose files.
.. at least until we get docker/cli#4863 approved

ndeloof force-pushed the publish branch 2 times, most recently from e9622c9 to ee6e7b9 Compare August 30, 2023 08:39

ndeloof force-pushed the publish branch 2 times, most recently from 1b302de to c74d06c Compare August 30, 2023 09:11

ndeloof requested review from a team, nicksieger, StefanScherer, ulyssessouza, glours, milas and laurazard and removed request for a team August 30, 2023 09:25

milas reviewed Aug 30, 2023

View reviewed changes

ndeloof force-pushed the publish branch from c74d06c to e6a4567 Compare August 31, 2023 09:06

milas approved these changes Sep 1, 2023

View reviewed changes

introduce publish (alpha) command

2772913

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

glours approved these changes Sep 6, 2023

View reviewed changes

ndeloof force-pushed the publish branch 2 times, most recently from 00dad5e to 08a7873 Compare September 6, 2023 14:05

milas reviewed Sep 6, 2023

View reviewed changes

ndeloof force-pushed the publish branch 2 times, most recently from 04b3fe6 to 998a7ea Compare September 6, 2023 15:24

pull OCI remote resource

057c273

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

ndeloof force-pushed the publish branch from 998a7ea to 057c273 Compare September 6, 2023 20:45

ndeloof merged commit e0f39eb into docker:main Sep 7, 2023
25 checks passed

ndeloof deleted the publish branch September 7, 2023 05:27

dnhn mentioned this pull request Sep 21, 2023

docker-compose 2.22.0 Homebrew/homebrew-core#144591

Merged

chenrui333 mentioned this pull request Sep 22, 2023

docker-compose 2.22.0 Homebrew/homebrew-core#144705

Closed

ndeloof mentioned this pull request Oct 2, 2023

[BUG] include from git repo works with config but not pull, up #11022

Closed

parisk mentioned this pull request Jan 24, 2024

[BUG] Getting No such file or directory when including OCI artifact using Docker remote #11398

Closed

qclaogui mentioned this pull request Mar 23, 2024

Simple use case qclaogui/codelab-monitoring#67

Closed

2 tasks

qclaogui mentioned this pull request Apr 5, 2024

[docker compose]: (WIP)Add compose publish (alpha) command qclaogui/codelab-monitoring#91

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce publish (alpha) command #10949

introduce publish (alpha) command #10949

ndeloof commented Aug 30, 2023 •

edited

codecov bot commented Aug 30, 2023 •

edited

milas left a comment

milas Aug 30, 2023

milas Aug 30, 2023

ndeloof Aug 30, 2023

milas Aug 31, 2023

BretFisher Sep 5, 2023

milas Sep 1, 2023

milas commented Sep 1, 2023

glours left a comment

milas left a comment

milas Sep 6, 2023

milas Sep 6, 2023

milas Sep 6, 2023

ndeloof Sep 6, 2023

milas Sep 6, 2023

ndeloof Sep 6, 2023

jaydrogers commented Feb 8, 2024

ndeloof commented Feb 9, 2024

introduce publish (alpha) command #10949

introduce publish (alpha) command #10949

Conversation

ndeloof commented Aug 30, 2023 • edited

codecov bot commented Aug 30, 2023 • edited

Codecov Report

milas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

milas commented Sep 1, 2023

glours left a comment

Choose a reason for hiding this comment

milas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaydrogers commented Feb 8, 2024

ndeloof commented Feb 9, 2024

ndeloof commented Aug 30, 2023 •

edited

codecov bot commented Aug 30, 2023 •

edited