isManifestUnknownError fails against Harbor registries, breaking sigstore signature upload #2203
Comments
|
Thanks for your report. As you see, the code is trying to copy signatures. This is supposed to work. AFAICS the issue here is interoperability with the specific destination registry implementation; a signed push with a sigstore signature to that registry would fail just the same. First, please make sure you are using the recently-released Skopeo 1.14.0; the behavior in this area has changed, quite possibly fixing this. If that’s not the case, we’ll need to fix that in c/image; for starters, please provide a full log from |
|
Hello, To give you some more information about the environment. Both source and destination are harbor registries - please see goharbor.io if you need more information about the product. The "test:0.1" image is a very simple image only containing an alpine image with a simple echo hello line as cmd. That image has been signed with cosign and a local self created key. The image has been uploaded to registry 1 into the project "images". Please see the following sysouts. First one copying the image into a local dir into the running container that provides skopeo version 1.14.0: ---snip--- [root@6bbacdc9b6e4 ~]# skopeo copy docker://192.168.a.b/images/test:0.1 dir:test_0-1 [root@6bbacdc9b6e4 ~]# skopeo --debug copy dir:test_0-1 docker://192.168.y.z/images/test:0.1 ---snap--- If I understand the last line correctly, instead of "signature-1" a file named sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig will be expected - is this correct? Second one showing sysout copying the image directly between harbor 1 and harbor 2: ---snip--- [root@6bbacdc9b6e4 ~]# skopeo --debug copy docker://192.168.a.b/images/test:0.1 docker://192.168.y.z/images/test:0.1 ---snap--- Second attempt showing the same result. Added the following lines to /etc/containers/registries.d/default.yaml: Hope that helps. Please let me know if any further information is needed. |
|
Thanks!
OK, the registry (ref. https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/controller/artifact/controller.go#L298) apparently uses a non-standard
No, that name refers to a tag on the destination registry (and a component of the URLs quoted above). That works as expected. Either way, we’ll want to capture the full HTTP response of the above-quoted HTTP request (with the relevant authentication headers). Is that easy to do for you in the current environment? If not, we’ll need a source code patch to capture that in Skopeo. |
mtrmac
changed the title
|
… looking more closely (and compare goharbor/harbor#19179 , not the same request but a sample of the error message):
our |
|
Hello again, If I understand you correctly, the situation is that a not expected response will be returned to prevent the signature to be delivered to harbor. Is this something that can be shipped around inside skopeo or must this be changed on harbor side? As the image copy works between two harbor registries directly, also if the source image has a signature, the functionality is available in general. |
|
Hi @mtrmac, I use the Harbor server ( 
Hope this information helps and feel free to ask me if you need me to capture additional debug information, thanks. By refer to goharbor/harbor#19179, if I understand correctly, it seems like this will be fixed in harbor |
|
@STARRY-S Thanks; that’s the request body, but not full request headers. Could you perhaps apply https://github.com/containers/image/pull/201/files (with (I don’t expect the response to contain credentials or secrets, but please double-check.) |
|
@mtrmac Hi, here's the log with captured response output: HTTP response header debug: HTTP response body: "{\"errors\":[{\"code\":\"NOT_FOUND\",\"message\":\"artifact test/alpine:sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig not found\"}]}\n" |
Skopeo is planned to be used to copy already signed images between two or more private registries.
In my situation, the images are already signed in a private source registry. This signature is also shown as existing within the source registry.
If this signed image is copied directly between the two registries - there is a graphical front end in the registry product for this - the image includes the signature appearing on the target registry.
However, this solution cannot be used due to other disadvantages. Thats the reason trying skopeo for a potential solution.
A $skopeo copy docker://192.168.x.y/images/test:0.1 dir:test_0.1 copies the image including the signature into the test_0.1 directory - into signature-1.
However, a copy directly between two registries does not work:
$skopeo copy docker://192.168.x.y/images/test:0.1 docker://192.168.y.z/images/test:0.1
Getting image source signatures
Checking if image destination supports signatures
Copying blob 96526aa774ef skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
Copying config 33b8df73a9 done
Writing manifest to image destination
Storing signatures
FATA[0000] writing signatures: reading manifest sha256-blablabla.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-blablabla.sig not found
Also, there is no positiv result trying to copy the content from inside the local directory. The same message is the result.
My question is: might it be that Skopeo cannot be used to copy images between two or more registries that are already signed in the source registry? There are no plans to re-sign the image with the private key in between again. This signature should be adopted 1:1 into the target registry - just as the registry product itself can do, although there are other reasons against using this solution.
May I ask for assistance or is this some kind of issue or enhancement?
The text was updated successfully, but these errors were encountered: