Sidecar: reporting as ready on startup when no Prometheus process is running

[SamJamS]

Thanos, Prometheus and Golang version used:

Object Storage Provider:

What happened:

When thanos sidecar starts up and there isn't a running Prometheus behind it, sidecar reports as ready.

What you expected to happen:

When sidecar starts up, if there isn't a running Prometheus behind it then sidecar should report as not ready.

How to reproduce it (as minimally and precisely as possible):

• Ensure Prometheus is not running
• Start thanos sidecar
• curl sidecar ready endpoint
• Validate response

Full logs to relevant components:

Logs from newly started sidecar when Prometheus is not running

➜  thanos git:(main) ✗ ./build/thanos sidecar
ts=2024-04-11T17:15:47.939758Z caller=sidecar.go:139 level=info msg="no supported bucket was configured, uploads will be disabled"
ts=2024-04-11T17:15:47.939921Z caller=options.go:26 level=info protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
ts=2024-04-11T17:15:47.940187Z caller=sidecar.go:385 level=info msg="starting sidecar"
ts=2024-04-11T17:15:47.940434Z caller=reloader.go:262 level=info component=reloader msg="nothing to be watched"
ts=2024-04-11T17:15:47.940488Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
ts=2024-04-11T17:15:47.940637Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
ts=2024-04-11T17:15:47.940647Z caller=http.go:73 level=info service=http/server component=sidecar msg="listening for requests and metrics" address=0.0.0.0:10902
ts=2024-04-11T17:15:47.941Z caller=grpc.go:131 level=info service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901
ts=2024-04-11T17:15:47.941095Z caller=tls_config.go:313 level=info service=http/server component=sidecar msg="Listening on" address=[::]:10902
ts=2024-04-11T17:15:47.941138Z caller=tls_config.go:316 level=info service=http/server component=sidecar msg="TLS is disabled." http2=false address=[::]:10902
ts=2024-04-11T17:15:47.941577Z caller=sidecar.go:188 level=warn msg="failed to fetch prometheus version. Is Prometheus running? Retrying" err="perform GET request against http://localhost:9090/api/v1/status/buildinfo: Get \"http://localhost:9090/api/v1/status/buildinfo\": dial tcp [::1]:9090: connect: connection refused"
ts=2024-04-11T17:15:49.944287Z caller=sidecar.go:188 level=warn msg="failed to fetch prometheus version. Is Prometheus running? Retrying" err="perform GET request against http://localhost:9090/api/v1/status/buildinfo: Get \"http://localhost:9090/api/v1/status/buildinfo\": dial tcp [::1]:9090: connect: connection refused"
ts=2024-04-11T17:15:51.94443Z caller=sidecar.go:188 level=warn msg="failed to fetch prometheus version. Is Prometheus running? Retrying" err="perform GET request against http://localhost:9090/api/v1/status/buildinfo: Get \"http://localhost:9090/api/v1/status/buildinfo\": dial tcp [::1]:9090: connect: connection refused"

➜  ~ curl localhost:10902/-/ready
OK%

Anything else we need to know:

It seems this behaves as expected when at one point there is a running Prometheus and sidecar can enter its heartbeat loop (transitioning between ready/not ready depending on the state of Prometheus). In the case however where there is never a ready Prometheus, sidecar never enters its heartbeat loop. Instead it continuously tries to fetch the prometheus version and doesn't update its ready state, defaulting to true/ready.

Have raised a PR with a potential fix for this: #7291