Thanos Query logs warning when querying prometheus sidecar: "detecting store that does not support without replica label setting"

[maartengo]

Hello all,

What goes wrong?

The Thanos Query logs the following whenever querying one of our Prometheus Thanos sidecar (10.110.16.194:10901 is the sidecar)

level=warn ts=2023-03-31T11:33:58.855156294Z caller=proxy_heap.go:545 component=proxy request="min_time:1680262140195 max_time:1680262440195 matchers:<name:\"pod\" value:\"kafka-0\" > matchers:<name:\"__name__\" value:\"up\" > aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT without_replica_labels:\"prometheus_replica\" " msg="detecting store that does not support without replica label setting. Falling back to eager retrieval with additional sort. Make sure your storeAPI supports it to speed up your queries" store="Addr: 10.110.16.194:10901 LabelSets: {prometheus=\"prometheus/prometheus-stack-kube-prom-prometheus\", prometheus_replica=\"prometheus-prometheus-stack-kube-prom-prometheus-1\"} MinTime: 1680228000026 MaxTime: 9223372036854775807"

My setup

I run two Prometheus instances in the same cluster with Thanos sidecards. There is a Thanos Query that scrapes both sidecars and deduplicates the data. The deplucation seems to work fine since it only returns one result.

I use the kube-prometheus-stack Helm chart and the Bitnami Thanos Helm chart, both are the latest version.

I have configured the prometheus_replica replica label in the Thanos Query

My question

What causes the warning? And what can be done to fix the issue? Should I configure external labels?

[gtirloni]

Also seeing the same warning for a metric that exists in Prometheus but it's not showing in Thanos Query.

[jkotiuk]

I had the same error and it disappeared after upgrading prometheus to 2.42.0 and sidecar to 0.31.0.

[CostisC]

I concur that the upgrade to the above-mentioned releases fixed these warnings. This is the accepted answer.

[skycoop]

Upgrading the sidecar from v0.30.2 to v0.31.0 also resolved the issue for me

[prein]

I'm getting this warning in thanos qery logs with prometheus 2.51.0 and thanos sidecar 0.34.1

[atos-cosmin-gavagiuc]

I got the same error from each thanos sidecar after upgrading from 0.30.2 to 0.34.1. And it got solved by upgrading Prometheus v2.47.2 to v2.51.2.