Error Could not do a head request for

[stanthewizzard]

NOTE: See #1529 for workarounds until this gets properly fixed

---

Describe the bug

Hello, since 48h hour, when auto updating I get this for every container that need an update:

Could not do a head request for "xxxx", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

thanks for help

Steps to reproduce

docker run -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once

Expected behavior

update with correct message pull

Note: The image does get updated, you only get a warning that it couldn't do so without using up the docker API rate limit -@piksel

Environment

• photon 4
• Docker version 20.10.14, build a224086

Your logs

Logs

2023-01-20T21:19:57Z [I] Watchtower 1.5.1
2023-01-20T21:19:57Z [I] Using no notifications
2023-01-20T21:19:57Z [I] Checking all containers (except explicitly disabled with label)
2023-01-20T21:19:57Z [I] Running a one time update.
2023-01-20T21:19:59Z [W] Could not do a head request for "pihole/pihole:latest", falling back to regular pull.
                         container: /pihole
                         image: pihole/pihole:latest
2023-01-20T21:19:59Z [W] Reason: registry responded to head request with "404 Not Found", auth: "not present"
                         container: /pihole
                         image: pihole/pihole:latest
2023-01-20T21:20:07Z [I] Session done
                         Failed: 0
                         Scanned: 13
                         Updated: 0
                         notify: no
2023-01-20T21:20:07Z [I] Waiting for the notification goroutine to finish
                         notify: no

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[darthShadow]

This may be due to the recent buildx change due to the Github Ubuntu Runner being upgraded:

docker/build-push-action#755

This results in images being pushed whose manifests cannot be parsed due to these issues:

moby/moby#43126

docker/buildx#1509

I didn't get time to verify via the API for a failing image (https://github.com/AnalogJ/scrutiny, specifically the collector) but the docker manifest inspect command does fail for it and docker buildx imagetools inspect shows that it's an oci image.

[darthShadow]

I have verified these images of the reports from the linked thread and all have the above issue:

koenkk/zigbee2mqtt:latest
netdata/netdata:latest
pihole/pihole:latest
crowdsecurity/crowdsec:latest

[stanthewizzard]

Yes I have it too with pihole.
So nothing to do on my side (edit:removed the ?)
thank you

[darthShadow]

They (pihole) can pin the buildx version to v0.9.1 to fix their images (Example: cilium/cilium#23206)

[stanthewizzard]

As I have some containers exhibiting this behavior it will not be easy.
I posted on pihole git

[dreary-ennui]

Confirmed, seeing it on crowdsecurity/crowdsec:latest

[MilesTEG1]

Hello 👋🏻
I too have those messages since several days.
The image impacted varies from time to to time.

Here the email notification I received this morning :

Could not do a head request for "ttionya/vaultwarden-backup:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "crowdsecurity/crowdsec:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "[ghcr.io/lissy93/dashy:latest](https://github.com/containrrr/watchtower/issues/ghcr.io/lissy93/dashy:latest)", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Found new [ghcr.io/lissy93/dashy:latest](https://github.com/containrrr/watchtower/issues/ghcr.io/lissy93/dashy:latest) image (1edbe1a164c7)
Stopping /dashy (1f1e9f26a75b) with SIGTERM
Creating /dashy
Removing image 0d5a71213f81

And the one form yesterday:

Found new gitea/gitea:latest image (86b1df821fa3)
Could not do a head request for "crowdsecurity/crowdsec:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Found new [lscr.io/linuxserver/swag:latest](https://github.com/containrrr/watchtower/issues/lscr.io/linuxserver/swag:latest) image (7f3a0574ee4e)
Could not do a head request for "ttionya/vaultwarden-backup:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Found new ttionya/vaultwarden-backup:latest image (5762f8c64fd9)
Found new adguard/adguardhome:latest image (6d0497b485db)
Stopping /adguardhome_macvlan (d8260a3b81e7) with SIGTERM
Stopping /vaultwarden_backup_ttionya (f93b66cbed33) with SIGTERM
Stopping /swag (920c755314b8) with SIGTERM
Stopping /gitea (eb454549b72f) with SIGTERM
Creating /gitea
Creating /swag
Creating /vaultwarden_backup_ttionya
Creating /adguardhome_macvlan
Removing image 3591480978a0
Removing image d9466ff4a8c3
Removing image e67bbb3e18ae
Removing image f0fd1c319729

I understand that the update are still done, right ?

And there is nothing we can do to not have those errors , correct too ?
Because if we have to fill issue for all images that are generating this head request error, it will be a pain in the…

[darthShadow]

Yeah, from my understanding, either the images need to fix it or we need to wait for the above linked issues to be resolved by docker.

In the meanwhile, you could hide these warnings by setting WATCHTOWER_WARN_ON_HEAD_FAILURE to "never" (https://containrrr.dev/watchtower/arguments/#head_failure_warnings)

[stanthewizzard]

I added in my docker compose WATCHTOWER_WARN_ON_HEAD_FAILURE: "never"

then docker-compose up -d

and I still get this:

2023-01-21T09:37:08Z [W] Could not do a head request for "pihole/pihole:latest", falling back to regular pull.
                         container: /pihole
                         image: pihole/pihole:latest
2023-01-21T09:37:08Z [W] Reason: registry responded to head request with "404 Not Found", auth: "not present"
                         container: /pihole
                         image: pihole/pihole:latest

[crazy-max]

They (pihole) can pin the buildx version to v0.9.1 to fix their images (Example: cilium/cilium#23206)

No need to pin to previous release v0.9.1, pihole repo can just disable provenance as shown in docker/buildx#1513 (comment). Also OCI images are around for a few years now so I think Watchtower should support them instead.

[stanthewizzard]

The pi-hole issue is no more
Thanks to @PromoFaux

[piksel]

Also OCI images are around for a few years now so I think Watchtower should support them instead.

Yeah, but we are using the docker client's own code, so as long as docker manifest inspect doesn't support it, we probably won't either.

> docker manifest inspect -v crowdsecurity/crowdsec:latest
no such manifest: docker.io/crowdsecurity/crowdsec:latest

[piksel]

This actually looks to be quite an easy fix, since only adding the application/vnd.oci.image.index.v1+json mime to type the Accepted: header seems to make it work correctly.

We don't supporting parsing the response, but that is irrelevant, since the only thing we are interested in is if it's been changed since the image was pulled.

[crazy-max]

Also OCI images are around for a few years now so I think Watchtower should support them instead.

Yeah, but we are using the docker client's own code, so as long as docker manifest inspect doesn't support it, we probably won't either.

> docker manifest inspect -v crowdsecurity/crowdsec:latest
no such manifest: docker.io/crowdsecurity/crowdsec:latest

Yes this issue is being tracked in moby/moby#43126 (cc @thaJeztah)

[vilmotz]

With the homarr image is the same :(
https://github.com/ajnart/homarr

[tamimology]

I have verified these images of the reports from the linked thread and all have the above issue:

koenkk/zigbee2mqtt:latest netdata/netdata:latest pihole/pihole:latest crowdsecurity/crowdsec:latest

I would like to add the following to the list

amir20/dozzle:latest
jokobsk/pi.alert:latest

[evilscientress]

esphome/esphome:latest is also affected by the issue.

There also is an explicit warning for this issue in the release notes of buildx 0.10.0.
Seams like many have updated without checking them or just use the latest version in their CI. I do the later myself. Would not expect that a breaking change like that is released until at least dockers own registry is ready for it.

[MilesTEG1]

Hello,
here some images that cause the head request with "404 Not Found", auth: "not present"
(some have already been reported)

• ghcr.io/lissy93/dashy:latest
• ttionya/vaultwarden-backup:latest
• crowdsecurity/crowdsec:latest
• ghcr.io/crazy-max/fail2ban:latest

[mytelegrambot]

here some images that cause the head request with "404 Not Found", auth: "not present"
https://github.com/wiserain/docker-flexget

[tweek11]

Was this resolved or expected to be with a upcoming update? Sorry for my confusion - but I've been reading through the post and others where it has been mentioned - But still confused. Again, my apology.

I ask because I've been getting the following for a little while now when manually running Watchtower. This was just dispayed before posting here.

time="2023-01-28T21:56:30Z" level=warning msg="Could not do a head request for "netboxcommunity/netbox:v3.4-2.4.0", falling back to regular pull." container=/netbox-docker_netbox-housekeeping_1 image="netboxcommunity/netbox:v3.4-2.4.0"
time="2023-01-28T21:56:30Z" level=warning msg="Reason: registry responded to head request with "404 Not Found", auth: "not present"" container=/netbox-docker_netbox-housekeeping_1 image="netboxcommunity/netbox:v3.4-2.4.0"
time="2023-01-28T21:56:32Z" level=warning msg="Could not do a head request for "netboxcommunity/netbox:v3.4-2.4.0", falling back to regular pull." container=/netbox-docker_netbox-worker_1 image="netboxcommunity/netbox:v3.4-2.4.0"
time="2023-01-28T21:56:32Z" level=warning msg="Reason: registry responded to head request with "404 Not Found", auth: "not present"" container=/netbox-docker_netbox-worker_1 image="netboxcommunity/netbox:v3.4-2.4.0"
time="2023-01-28T21:56:34Z" level=warning msg="Could not do a head request for "netboxcommunity/netbox:v3.4-2.4.0", falling back to regular pull." container=/netbox-docker_netbox_1 image="netboxcommunity/netbox:v3.4-2.4.0"
time="2023-01-28T21:56:34Z" level=warning msg="Reason: registry responded to head request with "404 Not Found", auth: "not present"" container=/netbox-docker_netbox_1 image="netboxcommunity/netbox:v3.4-2.4.0"
time="2023-01-28T21:56:38Z" level=warning msg="Could not do a head request for "amir20/dozzle:latest", falling back to regular pull." container=/dozzle image="amir20/dozzle:latest"
time="2023-01-28T21:56:38Z" level=warning msg="Reason: registry responded to head request with "404 Not Found", auth: "not present"" container=/dozzle image="amir20/dozzle:latest"
time="2023-01-28T21:56:42Z" level=warning msg="Could not do a head request for "ghcr.io/paperless-ngx/paperless-ngx:latest", falling back to regular pull." container=/paperless_webserver_1 image="ghcr.io/paperless-ngx/paperless-ngx:latest"
time="2023-01-28T21:56:42Z" level=warning msg="Reason: registry responded to head request with "404 Not Found", auth: "not present"" container=/paperless_webserver_1 image="ghcr.io/paperless-ngx/paperless-ngx:latest"

[tessierp]

I'm experiencing the exact same issue.

[tamimology]

I'm experiencing the exact same issue.

Add this in the env in your docker-compose for a quick and temporary fix

environment:
  WATCHTOWER_WARN_ON_HEAD_FAILURE: "never"