Add metadata-file flag

[crazy-max]

cc. @tonistiigi

Signed-off-by: CrazyMax crazy-max@users.noreply.github.com

[tonistiigi]

I think this would be better as metadata-file (eg. moby/buildkit#2095) and one per bake that lists all properties for all built targets instead of a file per target.

@AkihiroSuda thoughts?

[AkihiroSuda]

SGTM

[tonistiigi]

^

[crazy-max]

Ok here is the current state on my side:

• build command
  • --iidfile same behavior
  • --metadatafile write ExporterResponse (single target)
• bake command
  • --iidfile remove this flag introduced here in favor of the new flag --metadatafile
  • --metadatafile write all build options and ExporterResponse for all targets. In this case I don't think we should handle a metadatafile target field.

[tonistiigi]

@crazy-max SGTM. Having both iidfile and metadata-file in build may be a bit confusing atm but I guess it is fine. We could deprecate iidfile later or make it hidden.

https://github.com/moby/buildkit/pull/2095/files added flag with a dash. I think the dash is better.

[crazy-max]

@tonistiigi

Having both iidfile and metadata-file in build may be a bit confusing atm but I guess it is fine. We could deprecate iidfile later or make it hidden.

Agree about deprecation of iidfile.

https://github.com/moby/buildkit/pull/2095/files added flag with a dash. I think the dash is better.

Yes, I was wondering if we were keeping the same format as for iidfile but since we could deprecate it I will modify it to metadata-file.

[crazy-max]

@tonistiigi Ok now metadata-file handles build/solve. Here is an example:

variable "DEFAULT_TAG" {
  default = "7zip:local"
}

// Special target: https://github.com/docker/metadata-action#bake-definition
target "docker-metadata-action" {
  tags = ["${DEFAULT_TAG}"]
}

// Default target if none specified
group "default" {
  targets = ["image-local"]
}

target "image" {
  inherits = ["docker-metadata-action"]
}

target "image-local" {
  inherits = ["image"]
  output = ["type=docker"]
}

target "image-all" {
  inherits = ["image"]
  platforms = [
    "linux/amd64",
    "linux/arm/v6",
    "linux/arm/v7",
    "linux/arm64",
    "linux/386",
    "linux/ppc64le",
    "linux/s390x"
  ]
}

docker buildx bake --metadata-file metadata.json image-local

metadata.json

{
  "metadata": {
    "build-opts": {
      "inputs": {
        "context": ".",
        "dockerfile": "Dockerfile"
      },
      "tags": [
        "7zip:local"
      ],
      "metadata-file": "./metadata.json"
    },
    "solve-opts": [
      {
        "containerimage.config.digest": "sha256:fe4387b1cf24e8b3f153b026e2843e8abddeda479b27fd6adf3ed7ad1f8eb67a",
        "containerimage.digest": "sha256:3eaf8fb892e653224be132bc26de3bfa82e4563750b5f1f6dd64ae25788c6247",
        "image.name": "docker.io/library/7zip:local"
      }
    ]
  }
}

docker buildx build --output type=docker --metadata-file ./metadata.json .

metadata.json

{
  "metadata": {
    "build-opts": {
      "inputs": {
        "context": "."
      },
      "metadata-file": "./metadata.json",
      "network-mode": "default"
    },
    "solve-opts": [
      {
        "containerimage.config.digest": "sha256:fe4387b1cf24e8b3f153b026e2843e8abddeda479b27fd6adf3ed7ad1f8eb67a",
        "containerimage.digest": "sha256:3eaf8fb892e653224be132bc26de3bfa82e4563750b5f1f6dd64ae25788c6247"
      }
    ]
  }
}

Let me know if it's the right direction.

[tonistiigi]

I thought this was just going to contain the build result metadata. Why is that part called solve-opts? The rest is somewhat interesting but a different use-case and format is an internal representation that doesn't match anything else.

For the bake, the key should be image-local, not "metadata?"

Not related to this PR code but what is containerimage.config.digest for a multi-platform build.

[crazy-max]

@tonistiigi

I thought this was just going to contain the build result metadata. Why is that part called solve-opts?

Yes I was wondering about naming. Will call it metadata and set a "per-target" representation.

The rest is somewhat interesting but a different use-case and format is an internal representation that doesn't match anything else.

I wanted to see if it could be interesting in the future indeed. We can do this in a follow-up instead.

For the bake, the key should be image-local, not "metadata?"

Ok I can make a representation per-target

Not related to this PR code but what is containerimage.config.digest for a multi-platform build.

Will be empty:

docker buildx bake --set *.tags=crazymax/7zip:test --push --metadata-file ./metadata.json image-all

{
  "metadata": {
    "build-opts": {
      "inputs": {
        "context": ".",
        "dockerfile": "Dockerfile"
      },
      "tags": [
        "crazymax/7zip:test"
      ],
      "metadata-file": "./metadata.json",
      "platform": [
        {
          "architecture": "amd64",
          "os": "linux"
        },
        {
          "architecture": "arm",
          "os": "linux",
          "variant": "v6"
        },
        {
          "architecture": "arm",
          "os": "linux",
          "variant": "v7"
        },
        {
          "architecture": "arm64",
          "os": "linux"
        },
        {
          "architecture": "386",
          "os": "linux"
        },
        {
          "architecture": "ppc64le",
          "os": "linux"
        },
        {
          "architecture": "s390x",
          "os": "linux"
        }
      ]
    },
    "solve-opts": [
      {
        "containerimage.digest": "sha256:61ba426e49e207972d9ffaf6761cffec9fb3ae63dab76414cf250c9058370643",
        "image.name": "crazymax/7zip:test"
      }
    ]
  }
}

[crazy-max]

@tonistiigi

build

docker buildx build --output type=docker --metadata-file ./metadata.json .

{
  "containerimage.config.digest": "sha256:d8b8b4f781520aeafedb5a88ff50fbb625cfebad87e392794f1e26a724a2f22a",
  "containerimage.digest": "sha256:868f04872380274dcf8528222e84dc66702394de80889e51c87a14126ea9ff6a"
}

bake

docker buildx bake --metadata-file metadata.json image-all image-local

{
  "image-all": {
    "containerimage.digest": "sha256:d9aeb67cb21c2b79da504342e3d30a6eb914b9d1b7df0c16b1947ee76587702c",
    "image.name": "crazymax/7zip:test"
  },
  "image-local": {
    "containerimage.config.digest": "sha256:d8b8b4f781520aeafedb5a88ff50fbb625cfebad87e392794f1e26a724a2f22a",
    "containerimage.digest": "sha256:868f04872380274dcf8528222e84dc66702394de80889e51c87a14126ea9ff6a",
    "image.name": "docker.io/library/7zip:local"
  }
}

[tonistiigi]

Small comments. Overall looks good 👍

[tonistiigi]

Now that we import ioutils you can remove the duplicate function in store pkg in follow-up.