buildah manifest push fails for quay.io when images were pulled from quay.io #2594

mh21 · 2020-09-08T10:44:26Z

Description

We are creating multiple single-arch container images, upload them to quay.io and later in another job try to generate multi-arch manifests for them. The manifest upload fails with "manifest invalid" for quay.io. When using docker.io, this works.

Minimal reproducer:

echo -e "FROM alpine\nRUN touch dummy.txt" > Dockerfile

export BUILDAH_FORMAT=docker
export REG=quay.io/mhofmann  # does not work
#export REG=docker.io/mh21  # works

# start clean
rm ~/.local/share/containers/cache/blob-info-cache-v1.boltdb
buildah rmi localhost/dummy:tag $REG/dummy:tag-amd64

# n jobs: create single-arch image
buildah bud -f Dockerfile -t dummy:tag-amd64 .
buildah push dummy:tag-amd64 $REG/dummy:tag-amd64

# final job: create multi-arch manifest from all single-arch images
# (1) insert skopeo line here
buildah manifest create dummy:tag docker://$REG/dummy:tag-amd64
buildah manifest push dummy:tag docker://$REG/dummy:tag --all

Error message for quay.io:

...
Writing manifest list to image destination
Uploading manifest list failed, attempted the following formats: application/vnd.docker.distribution.manifest.list.v2+json(Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid), application/vnd.oci.image.index.v1+json(Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid)
$

For docker.io, this works without error message:

...
Storing signatures
Writing manifest list to image destination
Storing list signatures
$

This might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1810768 and containers/image#733 as it can be fixed by pulling down the single-arch images via skopeo before the buildah manifest create and forcing a v2s2 manifest (without the --format v2s2 it does not work):

skopeo copy docker://$REG/dummy:tag-amd64 containers-storage:$REG/dummy:tag-amd64 --format v2s2

Versions:

$ apt list buildah
Listing... Done
buildah/unknown,now 1.15.1~1 amd64 [installed]
buildah/unknown 1.15.1~1 arm64
buildah/unknown 1.15.1~1 armhf
buildah/unknown 1.15.1~1 s390x

$ buildah version
Version:         1.15.1
Go Version:      go1.14.2
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.5.1
Git Commit:      
Built:           Thu Jan  1 01:00:00 1970
OS/Arch:         linux/amd64

$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
Linux black 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/containers/storage.conf |grep '^[^#]'
[storage]
driver = ""
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
[storage.options.overlay]
mountopt = "nodev"
[storage.options.thinpool]

The text was updated successfully, but these errors were encountered:

mh21 · 2020-09-08T10:51:42Z

Logs with --log-level debug without the skopeo line: buildah-no-skopeo.log
Logs with --log-level debug with the skopeo line without --format v2s2: buildah-skopeo.log
Logs with --log-level debug with the skopeo line with --format v2s2: buildah-skopeo-v2s2.log

QiWang19 · 2020-10-19T13:19:43Z

Saw this error with Buildah 1.15.1 but failed to reproduce this with the upstream branch. This should have been fixed. @mh21 have you tried the newer version of Buidlah?

mh21 · 2020-10-21T20:33:18Z

I still get the same error with quay.io without skopeo when using buildah in a privileged rawhide container:

Uploading manifest list failed, attempted the following formats: application/vnd.docker.distribution.manifest.list.v2+json(Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid), application/vnd.oci.image.index.v1+json(Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid)

For comparison:

docker.io works without any skopeo line
quay.io and skopeo (without --format v2s2) does not work
quay.io and skopeo --format v2s2 works

Environment:

podman command line: podman run --rm -it -v .:/code registry.fedoraproject.org/fedora:rawhide
additional installed packages: buildah runc skopeo
sed -i '/^mountopt =.*/d' /etc/containers/storage.conf

$ buildah version
Version:         1.17.0-dev
Go Version:      go1.15.2
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.6.0
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

QiWang19 · 2020-10-21T20:42:02Z

@mh21 Did you try the whole reproducer with 1.17.0-dev or you just run manifest push?

mh21 · 2020-10-21T21:11:02Z

Attached the log of the container run. I ran the script two times in the container.

It seems like the first time it actually worked.

Storing signatures
Writing manifest list to image destination
DEBU Trying to use manifest list type application/vnd.docker.distribution.manifest.list.v2+json… 
DEBU PUT https://quay.io/v2/mhofmann/dummy/manifests/tag 
Storing list signatures
DEBU shutting down the store

The second time, it fails while uploading the manifest with

Storing signatures
Writing manifest list to image destination
DEBU Trying to use manifest list type application/vnd.docker.distribution.manifest.list.v2+json… 
DEBU Manifest list has been updated               
DEBU PUT https://quay.io/v2/mhofmann/dummy/manifests/tag 
DEBU Upload of manifest list type application/vnd.docker.distribution.manifest.list.v2+json failed: Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid 
DEBU Trying to use manifest list type application/vnd.oci.image.index.v1+json… 
DEBU Manifest list has been updated               
DEBU PUT https://quay.io/v2/mhofmann/dummy/manifests/tag 
DEBU Upload of manifest list type application/vnd.oci.image.index.v1+json failed: Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid 
Uploading manifest list failed, attempted the following formats: application/vnd.docker.distribution.manifest.list.v2+json(Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid), application/vnd.oci.image.index.v1+json(Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid)
DEBU shutting down the store

buildah-1-17-0.log

QiWang19 · 2020-10-27T12:48:05Z

Reopen this issue. @nalind is this the issue you are looking at?

nalind · 2020-10-27T21:11:22Z

Thanks, it's containers/skopeo#1078 that I was looking at. I think @mtrmac guessed that it might be a symptom of containers/image#733, but I haven't chased it down well enough to be sure yet.

QiWang19 · 2020-11-18T19:49:55Z

# final job: create multi-arch manifest from all single-arch images
# (1) insert skopeo line here
buildah manifest create dummy:tag docker://$REG/dummy:tag-amd64

@mh21 The problem can be caused by the buildah manifest create in the reproducer finds the local image and use the local one to create the manifest list. I'm not sure why this leads the error, but it can work if I remove the local dummy:tag-amd64 before the manifest create. And push it with --format v2s2.

mh21 · 2020-11-19T14:46:40Z

Hi @QiWang19, thank you!

I played around with it a bit more and I don't think it is related to the local images. The error is still there, independent of whether the container images are present when manifest create is run.

But thanks to you I found an easier workaround 😄. Instead of pulling down the images with skopeo, it is enough to specify the --format v2s2 argument for buildah manifest push like

buildah manifest create dummy:tag docker://$REG/dummy:tag-amd64
buildah manifest push --format v2s2 dummy:tag docker://$REG/dummy:tag --all

For reference, this is the complete script used:

#!/bin/bash
echo -e "FROM alpine\nRUN touch dummy.txt" > Dockerfile

export BUILDAH_FORMAT=docker
export REG=quay.io/mhofmann
#export REG=docker.io/mh21

function cleanup() {
    rm ~/.local/share/containers/cache/blob-info-cache-v1.boltdb > /dev/null 2>&1
    for host in localhost "$REG"; do
        for tag in tag tag-amd64; do 
            buildah rmi $host/dummy:$tag > /dev/null 2>&1
        done
    done
}

# n jobs: create single-arch image
cleanup
buildah bud -f Dockerfile -t dummy:tag-amd64 .
buildah push dummy:tag-amd64 $REG/dummy:tag-amd64

# final job: create multi-arch manifest from all single-arch images
cleanup  # doesn't matter whether images are cleaned here
buildah manifest create dummy:tag docker://$REG/dummy:tag-amd64
buildah manifest push dummy:tag docker://$REG/dummy:tag --all  # adding --format v2s2 fixes it

QiWang19 · 2020-11-19T16:07:54Z

@mh21 Thanks for the check 🙂 . I am going to close this.

mh21 · 2020-11-19T16:34:55Z

Hi @QiWang19, this is still something that works out of the box with docker, but does not work with buildah. Wasn't the idea that buildah should be a drop-in replacement for docker?

QiWang19 · 2020-11-19T18:54:13Z

Hi @QiWang19, this is still something that works out of the box with docker, but does not work with buildah. Wasn't the idea that buildah should be a drop-in replacement for docker?

@nalind in this issue reproducer, buildah needs to specify --format v2s2 to push to quay.io, anything we can fix to be compatible with docker?

nalind · 2020-11-19T19:21:47Z

That should be something we correctly handle automatically when we do a format conversion from OCI to Docker format during the push, and I could swear that we did, at least at one point. I'm looking into it.

TomSweeneyRedHat · 2020-11-19T20:23:46Z

@nalind is this related to the issue you were chasing for the Skopeo quay images? containers/skopeo#1078

nalind · 2020-11-20T14:27:56Z

@TomSweeneyRedHat Yes, I think so.

rhatdan · 2020-11-20T16:46:27Z

The question I have is when will quay.io fully support OCI images?

mtrmac · 2020-11-25T16:05:23Z

Is it really the --format v2s2 that makes a difference? The Buildah logs above look suspiciously like containers/image#733 (upload of an individual manifest with v2s2 fails, causing a fallback to v2s1; presumably Quay then doesn’t like v2s2 manifest lists pointing to v2s1 manifests).

For containers/image#733 , what matters is a local cache between one push/pull and another push. So, it might not be the --format that makes a difference, but whether something was run as a first push from a fresh VM/container/user account, or whether it is a second command in that environment. To test for sure, find the relevant blob-info-cache-v1.boltdb and see if removing it lets the operation succeed.

mtrmac · 2020-11-25T16:06:40Z

The question I have is when will quay.io fully support OCI images?

If this is indeed something like containers/image#733 , OCI support won’t make a difference, we would still fall back to v2s1 (the only format that doesn’t carry the MIME type so it can’t be inconsistent in a way Quay would notice), and the same strict validation that rejects v2s1 images in v2s2 manifests would reject v2s1 images in OCI manifests.

mh21 · 2020-11-25T16:13:23Z

To test for sure, find the relevant blob-info-cache-v1.boltdb and see if removing it lets the operation succeed.

This is included in the scripts as

rm ~/.local/share/containers/cache/blob-info-cache-v1.boltdb

mtrmac · 2020-11-25T18:01:04Z

I’m sorry , I have somehow missed both the script and some of the logs.

Still, looking at https://github.com/containers/buildah/files/5418728/buildah-1-17-0.log , this really looks like 733: the first push of layer with DiffID 50644c29ef5a succeeds, compressing that layer on the fly and creating 0d9094d70e9c0ee00ae22533ace8595d3b1a7a24976dd1750ab6e8e62ef3a771 ; the second push of the same layer seems to reuse that cached location, and v2s2 fails.

(There’s a lot of …opportunity… to make the debug logs easier to understand, with multi-threaded pushes it’s now often not clear which log line is related to which blob.)

To test for sure, find the relevant blob-info-cache-v1.boltdb and see if removing it lets the operation succeed.

This is included in the scripts as
rm ~/.local/share/containers/cache/blob-info-cache-v1.boltdb

That’s the wrong path, at least in the log linked above:

DEBU Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb

mh21 · 2020-11-26T08:36:20Z

I added a

sudo rm /var/lib/containers/cache/blob-info-cache-v1.boltdb

to the script above. Again, without --format v2s2, this fails with

...
Storing signatures
Writing manifest list to image destination
Uploading manifest list failed, attempted the following formats: application/vnd.docker.distribution.manifest.list.v2+json(Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid), application/vnd.oci.image.index.v1+json(Error uploading manifest tag to quay.io/mhofmann/dummy: manifest invalid: manifest invalid)

For reference, the current reproducer:

#!/bin/bash
echo -e "FROM alpine\nRUN touch dummy.txt" > Dockerfile

export BUILDAH_FORMAT=docker
export REG=quay.io/mhofmann
#export REG=docker.io/mh21

function cleanup() {
    rm ~/.local/share/containers/cache/blob-info-cache-v1.boltdb > /dev/null 2>&1
    sudo rm /var/lib/containers/cache/blob-info-cache-v1.boltdb > /dev/null 2>&1
    for host in localhost "$REG"; do
        for tag in tag tag-amd64; do 
            buildah rmi $host/dummy:$tag > /dev/null 2>&1
        done
    done
}

# n jobs: create single-arch image
cleanup
buildah bud -f Dockerfile -t dummy:tag-amd64 .
buildah push dummy:tag-amd64 $REG/dummy:tag-amd64

# final job: create multi-arch manifest from all single-arch images
cleanup  # doesn't matter whether images are cleaned here
buildah manifest create dummy:tag docker://$REG/dummy:tag-amd64
buildah manifest push dummy:tag docker://$REG/dummy:tag --all  # --format v2s2 fixes it
#buildah manifest push --format v2s2 dummy:tag docker://$REG/dummy:tag --all

TomSweeneyRedHat · 2020-11-30T15:37:10Z

Based on the ongoing discussions, I'm going to reopen this one.

rhatdan · 2021-02-10T20:45:59Z

I believe this all works correctly now, I am closing, reopen if I am mistaken.

xrstf · 2021-08-17T12:35:57Z

We are seemingly experiencing this problem with Buildah 1.19.6. Though we are using images from docker.io as base images, the error is the same:

Uploading manifest list failed, attempted the following formats: application/vnd.oci.image.index.v1+json(Error uploading manifest latest to quay.io/kubermatic/user-ssh-keys-agent: manifest invalid: manifest invalid), application/vnd.docker.distribution.manifest.list.v2+json(Error uploading manifest latest to quay.io/kubermatic/user-ssh-keys-agent: manifest invalid: manifest invalid)

nalind · 2021-08-17T15:41:39Z

@xrstf does removing the blob info cache (/var/lib/containers/cache/blob-info-cache-v1.boltdb and /var/lib/containers/storage/cache/blob-info-cache-v1.boltdb for root, ~/.local/share/containers/storage/cache/blob-info-cache-v1.boltdb and ~/.local/share/containers/cache/blob-info-cache-v1.boltdb for non-root) have any effect? Are you able to test with a more recent version? 1.19.6 didn't have the benefit of containers/image#1138, but fixes applied after that may also have changed things.

rhatdan assigned QiWang19 Oct 7, 2020

rhatdan added the kind/bug Categorizes issue or PR as related to a bug. label Oct 7, 2020

rhatdan closed this as completed Oct 20, 2020

jadahl mentioned this issue Oct 27, 2020

Pushing multi arch manifest to quay.io fails with 'invalid manifest' #2724

Closed

QiWang19 reopened this Oct 27, 2020

QiWang19 removed their assignment Oct 27, 2020

QiWang19 closed this as completed Nov 19, 2020

TomSweeneyRedHat reopened this Nov 30, 2020

rhatdan closed this as completed Feb 10, 2021

github-actions bot added the locked - please file new issue/PR label Sep 4, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildah manifest push fails for quay.io when images were pulled from quay.io #2594

buildah manifest push fails for quay.io when images were pulled from quay.io #2594

mh21 commented Sep 8, 2020 •

edited

mh21 commented Sep 8, 2020

QiWang19 commented Oct 19, 2020

mh21 commented Oct 21, 2020 •

edited

QiWang19 commented Oct 21, 2020

mh21 commented Oct 21, 2020

QiWang19 commented Oct 27, 2020

nalind commented Oct 27, 2020

QiWang19 commented Nov 18, 2020

mh21 commented Nov 19, 2020

QiWang19 commented Nov 19, 2020

mh21 commented Nov 19, 2020

QiWang19 commented Nov 19, 2020

nalind commented Nov 19, 2020

TomSweeneyRedHat commented Nov 19, 2020

nalind commented Nov 20, 2020

rhatdan commented Nov 20, 2020

mtrmac commented Nov 25, 2020

mtrmac commented Nov 25, 2020

mh21 commented Nov 25, 2020

mtrmac commented Nov 25, 2020

mh21 commented Nov 26, 2020

TomSweeneyRedHat commented Nov 30, 2020

rhatdan commented Feb 10, 2021

xrstf commented Aug 17, 2021

nalind commented Aug 17, 2021

buildah manifest push fails for quay.io when images were pulled from quay.io #2594

buildah manifest push fails for quay.io when images were pulled from quay.io #2594

Comments

mh21 commented Sep 8, 2020 • edited

mh21 commented Sep 8, 2020

QiWang19 commented Oct 19, 2020

mh21 commented Oct 21, 2020 • edited

QiWang19 commented Oct 21, 2020

mh21 commented Oct 21, 2020

QiWang19 commented Oct 27, 2020

nalind commented Oct 27, 2020

QiWang19 commented Nov 18, 2020

mh21 commented Nov 19, 2020

QiWang19 commented Nov 19, 2020

mh21 commented Nov 19, 2020

QiWang19 commented Nov 19, 2020

nalind commented Nov 19, 2020

TomSweeneyRedHat commented Nov 19, 2020

nalind commented Nov 20, 2020

rhatdan commented Nov 20, 2020

mtrmac commented Nov 25, 2020

mtrmac commented Nov 25, 2020

mh21 commented Nov 25, 2020

mtrmac commented Nov 25, 2020

mh21 commented Nov 26, 2020

TomSweeneyRedHat commented Nov 30, 2020

rhatdan commented Feb 10, 2021

xrstf commented Aug 17, 2021

nalind commented Aug 17, 2021

mh21 commented Sep 8, 2020 •

edited

mh21 commented Oct 21, 2020 •

edited