Could not do a head request: invalid character #1514

modem7 · 2022-12-29T05:53:19Z

Describe the bug

Issues with the crowdsec image.

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: Crowdsec

Error:

Could not do a head request for "crowdsecurity/crowdsec:latest", falling back to regular pull.
Reason: invalid character 'S' looking for beginning of value

Steps to reproduce

Watchtower config:

  watchtower:
    image: containrrr/watchtower
    container_name: Watchtower
    hostname: Watchtower
    networks:
      - isonet
    labels: 
      - autoheal=true
    environment:
      TZ: $TZ
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_REMOVE_VOLUMES: "false"
      WATCHTOWER_INCLUDE_STOPPED: "true"
      WATCHTOWER_NO_STARTUP_MESSAGE: "true"
      WATCHTOWER_TIMEOUT: 30s
      WATCHTOWER_SCHEDULE: "0 0 6 * * *" # Everyday at 04:30
      WATCHTOWER_NOTIFICATIONS: shoutrrr
      WATCHTOWER_NOTIFICATION_URL: "telegram://$TGRAM_BOT_TOKEN@telegram?channels=$TGRAM_CHAT_ID"
      WATCHTOWER_NOTIFICATIONS_LEVEL: info
      DOCKER_HOST: unix:///var/run/docker.sock
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
    logging:
      driver: "local"
      options:
        max-size: 10m
        max-file: "3"
    restart: always
    mem_limit: 250m
    mem_reservation: 60m

Expected behavior

To update.

Screenshots

Environment

Ubuntu 22.04
x86
Docker version 20.10.22, build 3a2c30b
Crowdsec

Your logs

time="2022-12-26T06:00:35Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no
time="2022-12-27T06:00:39Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no
time="2022-12-28T06:00:04Z" level=warning msg="Could not do a head request for \"crowdsecurity/crowdsec:latest\", falling back to regular pull." container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-28T06:00:04Z" level=warning msg="Reason: invalid character 'S' looking for beginning of value" container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-28T06:00:15Z" level=info msg="Found new linuxserver/bookstack:latest image (f532d54a179c)"
time="2022-12-28T06:00:23Z" level=info msg="Found new linuxserver/sonarr:latest image (10d61a66d74e)"
time="2022-12-28T06:00:51Z" level=info msg="Found new linuxserver/netbootxyz:latest image (2d07183b1077)"
time="2022-12-28T06:00:53Z" level=info msg="Stopping /Netbootxyz (14c6cda3588b) with SIGTERM"
time="2022-12-28T06:01:34Z" level=info msg="Stopping /Sonarr (d19f614e614d) with SIGTERM"
time="2022-12-28T06:01:45Z" level=info msg="Stopping /Bookstack (7a1565f59d9e) with SIGTERM"
time="2022-12-28T06:01:51Z" level=info msg="Creating /Bookstack"
time="2022-12-28T06:01:52Z" level=info msg="Creating /Sonarr"
time="2022-12-28T06:01:53Z" level=info msg="Creating /Netbootxyz"
time="2022-12-28T06:01:59Z" level=info msg="Removing image 455a86d2b778"
time="2022-12-28T06:01:59Z" level=info msg="Removing image f4e288f594a6"
time="2022-12-28T06:02:00Z" level=info msg="Removing image 4866c5508f26"
time="2022-12-28T06:02:00Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=3 notify=no

Additional context

Unsure if related to #1050

Doing a manual run does not trigger any issues.

docker run --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower \
    --debug --run-once \
    Crowdsec
time="2022-12-29T05:56:32Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2022-12-29T05:56:33Z" level=debug msg="Making sure everything is sane before starting"
time="2022-12-29T05:56:33Z" level=info msg="Watchtower 1.5.1"
time="2022-12-29T05:56:33Z" level=info msg="Using no notifications"
time="2022-12-29T05:56:33Z" level=info msg="Only checking containers which name matches \"Crowdsec\""
time="2022-12-29T05:56:33Z" level=info msg="Running a one time update."
time="2022-12-29T05:56:33Z" level=debug msg="Checking containers for updated images"
time="2022-12-29T05:56:33Z" level=debug msg="Retrieving running containers"
time="2022-12-29T05:56:35Z" level=debug msg="Trying to load authentication credentials." container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="No credentials for crowdsecurity found" config_file=/config.json
time="2022-12-29T05:56:35Z" level=debug msg="Got image name: crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="Checking if pull is needed" container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2022-12-29T05:56:35Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2022-12-29T05:56:35Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2022-12-29T05:56:35Z" level=debug msg="Setting scope for auth token" image=crowdsecurity/crowdsec scope="repository:crowdsecurity/crowdsec:pull"
time="2022-12-29T05:56:35Z" level=debug msg="No credentials found."
time="2022-12-29T05:56:36Z" level=debug msg="Parsing image ref" host=index.docker.io image=crowdsecurity/crowdsec normalized="docker.io/crowdsecurity/crowdsec:latest" tag=latest
time="2022-12-29T05:56:36Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/crowdsecurity/crowdsec/manifests/latest"
time="2022-12-29T05:56:36Z" level=debug msg="Found a remote digest to compare with" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66"
time="2022-12-29T05:56:36Z" level=debug msg=Comparing local="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66"
time="2022-12-29T05:56:36Z" level=debug msg="Found a match"
time="2022-12-29T05:56:36Z" level=debug msg="No pull needed. Skipping image."
time="2022-12-29T05:56:36Z" level=debug msg="No new images found for /Crowdsec"
time="2022-12-29T05:56:36Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=0 notify=no
time="2022-12-29T05:56:36Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

The text was updated successfully, but these errors were encountered:

piksel · 2022-12-29T06:44:58Z

Seems like there was some kind of temporary network error. During your first session, it only warned you that it could not perform a "cheap" check for updates, falling back to a normal pull.

Whatever error message the server responded with is probably now gone and it works normally.

UniverseXXX · 2022-12-30T16:49:21Z

I'm having the same issue on Raspberry Pi 4 (AARCH64). I couldn't revert back to docker-compose 1.28.2, as no binary for aarch64 existed. It looks like there is a problem with docker-compose v.2.

hoorna · 2023-01-04T08:38:08Z

I have exactly the same issue. Occasionally I get this error-message for alternating images (not the same each time). I have the impression that this message has been occurring since the last Watchtower update.

piksel · 2023-01-04T17:14:31Z

@UniverseXXX and @hoorna what part do you mean is the same?
Do you also get the invalid character 'S' looking for beginning of value reason?

I don't see how docker-compose version is related to this issue. Nor should there be anything in the last release that alters this behaviour...

hoorna · 2023-01-04T18:32:36Z

@piksel, I also get the "invalid character 'S' looking for beginning of value".

Today I got for example an email message with the following text:
"Could not do a head request for "containrrr/watchtower:latest", falling back to regular pull.
Reason: invalid character 'S' looking for beginning of value"

Before the latest Watchtower update I never got such a message.

piksel · 2023-01-04T20:43:07Z

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error.
Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

modem7 · 2023-01-04T20:48:42Z

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error.
Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

If the message is cryptic and making it difficult to troubleshoot, I wonder if it would be worthwhile investigating a way to make that particular error more verbose (but not run the entire container verbose as that might just cause noise, even more so if it's an intermittent issue (seems to be in my case))?

piksel · 2023-01-20T21:31:49Z

Since #1528 was opened for the 404 errors, let's use it for those errors, since it seems to be something different from the original issue here.

lordraiden · 2023-10-12T23:09:16Z

I still get this error

Notifications:
Container errors
Streams: Vector - Containers
--- [Event Definition] ---
Title: Container errors
Type: aggregation-v1
Timestamp: 2023-10-12T23:00:08.000Z

--- [Log] ---
Source: docker_logs
Container: Watchtower
Level: warning
Message:
Message: Could not do a head request for
Message Link: https://10.10.40.190:9000/messages/vector_7/17144332-6953-11ee-819f-0242ac170004

Container errors
Streams: Vector - Containers
--- [Event Definition] ---
Title: Container errors
Type: aggregation-v1
Timestamp: 2023-10-12T23:00:08.000Z

--- [Log] ---
Source: docker_logs
Container: Watchtower
Level: warning
Message:
Message: Reason: invalid character 'S' looking for beginning of value
Message Link: https://10.10.40.190:9000/messages/vector_7/17146a40-6953-11ee-819f-0242ac170004

piksel · 2023-10-13T06:20:09Z

@lordraiden It's not an error, but rather a warning. If you don't care about the warning you can turn it off. It means that watchtower gets an invalid response from the repository and can't perform a cheap change check and falls back to a regular pull.

This issue go hijacked a bit for the occurring 404 errors (which was due to a change in docker buildx). I'll reopen this and clean it from unrelated comments.

piksel · 2023-10-13T06:34:50Z

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error.
Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

If the message is cryptic and making it difficult to troubleshoot, I wonder if it would be worthwhile investigating a way to make that particular error more verbose (but not run the entire container verbose as that might just cause noise, even more so if it's an intermittent issue (seems to be in my case))?

The response should be logged in full to aid in further investigation. Reopening this.

modem7 added Priority: Medium Status: Available Type: Bug Unintended behavior labels Dec 29, 2022

This comment has been minimized.

Sign in to view

LaurenceJJones mentioned this issue Jan 22, 2023

Can't update image with watchtower: manifest format crowdsecurity/crowdsec#2013

Closed

This comment has been minimized.

Sign in to view

piksel closed this as completed May 18, 2023

piksel reopened this Oct 13, 2023

piksel changed the title ~~Could not do a head request~~ Could not do a head request: invalid character Oct 13, 2023

piksel mentioned this issue Oct 13, 2023

Lack of feedback from bad HEAD responses #1799

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Could not do a head request: invalid character #1514

Could not do a head request: invalid character #1514

modem7 commented Dec 29, 2022 •

edited

piksel commented Dec 29, 2022

UniverseXXX commented Dec 30, 2022 •

edited

hoorna commented Jan 4, 2023

piksel commented Jan 4, 2023

hoorna commented Jan 4, 2023 •

edited

piksel commented Jan 4, 2023

modem7 commented Jan 4, 2023 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

piksel commented Jan 20, 2023 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lordraiden commented Oct 12, 2023

piksel commented Oct 13, 2023 •

edited

piksel commented Oct 13, 2023

Could not do a head request: invalid character #1514

Could not do a head request: invalid character #1514

Comments

modem7 commented Dec 29, 2022 • edited

Describe the bug

Steps to reproduce

Expected behavior

Screenshots

Environment

Your logs

Additional context

piksel commented Dec 29, 2022

UniverseXXX commented Dec 30, 2022 • edited

hoorna commented Jan 4, 2023

piksel commented Jan 4, 2023

hoorna commented Jan 4, 2023 • edited

piksel commented Jan 4, 2023

modem7 commented Jan 4, 2023 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

piksel commented Jan 20, 2023 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lordraiden commented Oct 12, 2023

piksel commented Oct 13, 2023 • edited

piksel commented Oct 13, 2023

modem7 commented Dec 29, 2022 •

edited

UniverseXXX commented Dec 30, 2022 •

edited

hoorna commented Jan 4, 2023 •

edited

modem7 commented Jan 4, 2023 •

edited

piksel commented Jan 20, 2023 •

edited

piksel commented Oct 13, 2023 •

edited