Problem with head request to docker image

[ochstobi]

What happened?

Hello,
since the last release there seems to be a problem to get information about the docker image.
I update the zigbee2mqtt container automatically via watchtower and there is an error since the last update of zigbee2mqtt.
The log says:
time="2023-01-13T23:53:08Z" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" time="2023-01-13T23:53:08Z" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"

What did you expect to happen?

No response

How to reproduce it (minimal and precise)

Run watchtower to update zigbee2mqtt

Zigbee2MQTT version

1.29.2

Adapter firmware version

20200925

Adapter

zzh!

Debug log

Log before the update:

time="2023-01-11T19:53:08Z" level=debug msg="Trying to load authentication credentials." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"
time="2023-01-11T19:53:08Z" level=debug msg="No credentials for koenkk found" config_file=/config.json
time="2023-01-11T19:53:08Z" level=debug msg="Got image name: koenkk/zigbee2mqtt:latest"
time="2023-01-11T19:53:08Z" level=debug msg="Checking if pull is needed" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"
time="2023-01-11T19:53:08Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2023-01-11T19:53:08Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2023-01-11T19:53:08Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2023-01-11T19:53:08Z" level=debug msg="Setting scope for auth token" image=koenkk/zigbee2mqtt scope="repository:koenkk/zigbee2mqtt:pull"
time="2023-01-11T19:53:08Z" level=debug msg="No credentials found."
time="2023-01-11T19:53:08Z" level=debug msg="Parsing image ref" host=index.docker.io image=koenkk/zigbee2mqtt normalized="docker.io/koenkk/zigbee2mqtt:latest" tag=latest
time="2023-01-11T19:53:08Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/koenkk/zigbee2mqtt/manifests/latest"
time="2023-01-11T19:53:09Z" level=debug msg="Found a remote digest to compare with" remote="sha256:62363f6d96ec7046a1d62212a43cb39b6efb4abbdcf16e885a1777b4004e73a6"
time="2023-01-11T19:53:09Z" level=debug msg=Comparing local="sha256:62363f6d96ec7046a1d62212a43cb39b6efb4abbdcf16e885a1777b4004e73a6" remote="sha256:62363f6d96ec7046a1d62212a43cb39b6efb4abbdcf16e885a1777b4004e73a6"
time="2023-01-11T19:53:09Z" level=debug msg="Found a match"
time="2023-01-11T19:53:09Z" level=debug msg="No pull needed. Skipping image."
time="2023-01-11T19:53:09Z" level=debug msg="No new images found for /Zigbee2MQTT"

Log after the last update:

time="2023-01-13T23:53:07Z" level=debug msg="Trying to load authentication credentials." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"
time="2023-01-13T23:53:07Z" level=debug msg="No credentials for koenkk found" config_file=/config.json
time="2023-01-13T23:53:07Z" level=debug msg="Got image name: koenkk/zigbee2mqtt:latest"
time="2023-01-13T23:53:07Z" level=debug msg="Checking if pull is needed" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"
time="2023-01-13T23:53:07Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2023-01-13T23:53:07Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2023-01-13T23:53:07Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2023-01-13T23:53:07Z" level=debug msg="Setting scope for auth token" image=koenkk/zigbee2mqtt scope="repository:koenkk/zigbee2mqtt:pull"
time="2023-01-13T23:53:07Z" level=debug msg="No credentials found."
time="2023-01-13T23:53:07Z" level=debug msg="Parsing image ref" host=index.docker.io image=koenkk/zigbee2mqtt normalized="docker.io/koenkk/zigbee2mqtt:latest" tag=latest
time="2023-01-13T23:53:07Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/koenkk/zigbee2mqtt/manifests/latest"
time="2023-01-13T23:53:08Z" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"
time="2023-01-13T23:53:08Z" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"
time="2023-01-13T23:53:08Z" level=debug msg="Pulling image" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"
time="2023-01-13T23:53:10Z" level=debug msg="No new images found for /Zigbee2MQTT"

[robertvanommen]

I have the same problem.

[keviny86]

Same here.

[olylak]

Same for me!

[Koenkk]

This looks to be a (temporary?) Docker hub issues, the image is published on Docker hub. docker pull koenkk/zigbee2mqtt:latest also works fine on my system.

[RaceAap]

A normal manual pull works good.
Only the automated update from watchtower fails with the error.

[ochstobi]

Pull is working. Watchtower does an head request du get the hash from dockerhub and compares it to the local running container. It was changed some time ago to prevent unnecessary pull request against dockerhub. Can anyone see the url that is used for the request in the logs?
I will try to debug further why this is failing

[KardinalReusen]

Same here.

[canedje]

same for me with docker under Unraid
see also issue:
#16246

[Koenkk]

I've retriggered the build, please check again in 1 hour from now.

[canedje]

For me docker under Unraid, still not available

[olylak]

Pull is working. Watchtower does an head request du get the hash from dockerhub and compares it to the local running container. It was changed some time ago to prevent unnecessary pull request against dockerhub. Can anyone see the url that is used for the request in the logs? I will try to debug further why this is failing

time="2023-01-19T22:14:35+02:00" level=debug msg="Trying to load authentication credentials." container=/zigbee2mqtt image="koenkk/zigbee2mqtt:latest"
time="2023-01-19T22:14:35+02:00" level=debug msg="No credentials for koenkk found" config_file=/config.json
time="2023-01-19T22:14:35+02:00" level=debug msg="Got image name: koenkk/zigbee2mqtt:latest"
time="2023-01-19T22:14:35+02:00" level=debug msg="Checking if pull is needed" container=/zigbee2mqtt image="koenkk/zigbee2mqtt:latest"
time="2023-01-19T22:14:35+02:00" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2023-01-19T22:14:35+02:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2023-01-19T22:14:35+02:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2023-01-19T22:14:35+02:00" level=debug msg="Setting scope for auth token" image=koenkk/zigbee2mqtt scope="repository:koenkk/zigbee2mqtt:pull"
time="2023-01-19T22:14:35+02:00" level=debug msg="No credentials found."
time="2023-01-19T22:14:36+02:00" level=debug msg="Parsing image ref" host=index.docker.io image=koenkk/zigbee2mqtt normalized="docker.io/koenkk/zigbee2mqtt:latest" tag=latest
time="2023-01-19T22:14:36+02:00" level=trace msg="Setting request token" token="Bearer ..... "
time="2023-01-19T22:14:36+02:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/koenkk/zigbee2mqtt/manifests/latest"
time="2023-01-19T22:14:36+02:00" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/zigbee2mqtt image="koenkk/zigbee2mqtt:latest"
time="2023-01-19T22:14:36+02:00" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/zigbee2mqtt image="koenkk/zigbee2mqtt:latest"

[clsferguson]

I've retriggered the build, please check again in 1 hour from now.

My watchtower has tried another 2 checks. The first check, updated the container, but still gave an error.

Could not do a head request for "docker.io/koenkk/zigbee2mqtt:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Found new docker.io/koenkk/zigbee2mqtt:latest image (d0d39a358e48)
Stopping /zigbee2mqtt (c9f978216ccd) with SIGTERM
Creating /zigbee2mqtt
Removing image 6f66c2ff40e8

After, the second check, which should do nothing, responded with this.

Could not do a head request for "docker.io/koenkk/zigbee2mqtt:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

So it's weird it still updates. It's just flooding the logs with a 404.

[ochstobi]

The question is why it can find the image by an regular pull but not by a head request. I don't think this is an issue from this software but an issue with dockerhub. We need to find out if theres are two different URLs for head and pull requests

[canedje]

maybe nothing to do with this. But since one hour also the Deconz docker is unavailable

[Ulrar]

Hi, had the same issue so I did some searching, and I found this thread : https://forums.unraid.net/topic/132686-docker-updateversion-not-available/

tl;dr is that unraid doesn't like the OCI format, and that can be solved (if it's the same issue) by passing --format=docker to the container build command.

Could that be it, and if so could the format be changed ? I have no idea what the implications of that are so just putting it there for someone else to think about :)

[olylak]

The thing is that the issue happened after last update.
No such issues with other containers so it's not related to watchtower or any versions dependencies.
I have two separate instances of watchtower looking for zigbee2mqtt docker container update and both are showing the same error.

[bigbangus]

I am using Unraid also. I have this issue also with 3 other containers: glances, SearchXNG, crowdsec.

I wonder if this is a larger systematic issue with dockerhub or just something with Unraid?

I also raised a new issue here on the Unraid forums: https://forums.unraid.net/topic/133850-lots-of-containers-have-version-not-available/

[ochstobi]

I think it is something dockhub related. Because im not using unraid and have the same problem. I try to check other images for changes in their build config the last days. Maybe there is something different

[loebelch]

I'm also thinking it's a dockhub related issue:

I'm using Nagios-Check (https://github.com/timdaman/check_docker) in order to check for updates images.
Since last update of zigbee2mqtt image the check fails with this error:

UNKNOWN: Cannot check version, couldn't retrieve digest for zigbee2mqtt while checking https://registry-1.docker.io/v2/koenkk/zigbee2mqtt/manifests/latest.

But it only happens for zigbee2mqtt image - other docker images are still fine (e.g. influxdb:latest)

[canedje]

No also with my Deconz docker not available

[spider7611]

Could not do a head request for "koenkk/zigbee2mqtt:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

[rramthun]

I do not know how koenkk builds the images, but e.g. podman creates OCI images by default. They are missing a SHA image digest in a place DockerHub expects it. I suspect DockerHub changes something in their codebase, because this has not been an issue in the past.

Adding "--format=docker" to the podman build command usually resolves this issue.

[Koenkk]

The things used to build the z2m Docker image are very standard (buildx), you can review it here:

zigbee2mqtt/.github/workflows/ci.yml

Line 37 in bb3e8f6

- name: Docker setup - Buildx

As it also happens with other images, I believe this is not something I can/should fix. It should be fixed at either Docker Hub or Watchtower.

[lafe]

No also with my Deconz docker not available

I don't think that it is a specific issue with a certain image but either a watchtower issue or an issue with the server. Personally, I have the same issue with Raspberrymatic (Homematic support) and Pihole. Raspberrymatic is also not hosted on the Docker Hub but uses GitHub (ghcr.io) to distribute its Docker images.

There is also a discussion/ticket at the Watchtower project: containrrr/watchtower#1528

[spider7611]

Hi, had the same issue so I did some searching, and I found this thread : https://forums.unraid.net/topic/132686-docker-updateversion-not-available/

tl;dr is that unraid doesn't like the OCI format, and that can be solved (if it's the same issue) by passing --format=docker to the container build command.

Could that be it, and if so could the format be changed ? I have no idea what the implications of that are so just putting it there for someone else to think about :)

--format=docker

This is not working for me. The docker just not start and give error for this: --format=docker

[Ulrar]

@spider7611 that's not an option to run the container, it's an option when building it

[wilotas]

@spider7611 that's not an option to run the container, it's an option when building it

Is possible to pass it in Portainer?

[Ulrar]

No because you're not building the container, @Koenkk does that. You just download it as is, there's nothing we can do as users for this, I believe.
Not even sure this would solve it, it just sounds like it might

[spider7611]

No because you're not building the container, @Koenkk does that. You just download it as is, there's nothing we can do as users for this, I believe. Not even sure this would solve it, it just sounds like it might

Ohh Ok. Thank You!

[u20p17]

I have also the same issue with the zigbee2mqtt docker and unraid... :(

[canedje]

I did find this in an other issue. I don't know what it means, but maybe it will help:

This actually looks to be quite an easy fix, since only adding the application/vnd.oci.image.index.v1+json mime to type the Accepted: header seems to make it work correctly.

We don't supporting parsing the response, but that is irrelevant, since the only thing we are interested in is if it's been changed since the image was pulled.

[rramthun]

The things used to build the z2m Docker image are very standard (buildx), you can review it here:

zigbee2mqtt/.github/workflows/ci.yml

Line 37 in bb3e8f6

- name: Docker setup - Buildx

Thank you for the link. This build script uses the latest version of buildx, which has a prominent warning about the problem we experience in their changelog: https://github.com/docker/buildx/releases/tag/v0.10.0

tl;dr: They are adding provenance information in manifests per default now. This creates all sort of different problems at the moment in infrastructure using buildx, c.f. buildx issue tracker.

@Koenkk I added #16297 which disables generation of these new provenance information as suggested by the buildx devs until buildx and/or all the infrastructure relying on it is fixed. This may take several additional weeks. You might consider to add this to avoid additional z2m bug reports or wait until everything fixes itself in a few weeks.

[Koenkk]

Thanks @rramthun, the fix will be included in the 1 February release.

[sylvaindd]

FYI, same issue occuring for me for zwave-js/zwave-js-ui since last update.

[bigbangus]

This has been solved for Unraid users by modifying
/usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php
on line 448 changing from:
$header = ['Accept: application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json'];
to:
$header = ['Accept: application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.index.v1+json'];

In template, Repository: koenkk/zigbee2mqtt

Reference:
https://forums.unraid.net/topic/134016-fix-to-a-small-issue-with-updating-oci-docker-images/

Note: this change may revert back when you restart your unraid server but has been included in the next version of Unraid (current version 6.11.5)

[canedje]

After latest update to 1.30. Still not avialable showing in Docker overview

[olympia]

@canedje the fix posted by bigbangus is working for unRAID. The fix has actually confirmed to be going into unRAID's next version, so expect the solution from that side, not from z2m (most likely this is not something Koenkk could fix in z2m).

[canedje]

@canedje the fix posted by bigbangus is working for unRAID. The fix has actually confirmed to be going into unRAID's next version, so expect the solution from that side, not from z2m (most likely this is not something Koenkk could fix in z2m).

@olympia Ahhh oke I did misunderstand that

[canedje]

After last update 1.30.1 the problem is fixed. And mentioned in the fixes
So it is part of the Z2M update.
Thanks for the fix