querier cache behaviour

[shubhamsharmadvlpr]

Hi Team, I'm new to thanos + prometheus setup and working on optimising the performance by tuning different component. we are facing a interesting situation. When using prometheus and thanos sidecar as single storeapi endpoint; response from thanos querier is much faster than prometheus directly

eg
curl -i http://thanos-querier-url/api/v1/query --data-urlencode 'query=prometheus_tsdb_wal_storage_size_bytes' --> 0.107s

curl -i http://prometheus-url/api/v1/query --data-urlencode 'query=prometheus_tsdb_wal_storage_size_bytes' --> 4s

so it's a huge margin, how's querier doing that? it should be hitting the same prometheus storeapi exposed by sidecar. i'm not able to find any documentation for caching implementation in thanos querier to justify it.

querier args
--log.level=debug
--query.replica-label=replica
--store=dnssrv+prometheus-thanos-store-gateway:10901 (--> only using sidecar)
--query.auto-downsampling

[MichaHoffmann]

Do you have traces maybe? That would greatly help in debugging i think!

[MichaHoffmann]

I dont think we have it at the querier level; but i might be wrong @yeya24 is the man when it comes to caching strategies; ill defer to him!

[MichaHoffmann]

Ah nvm we do have response caches or at least we set cache control header here:

thanos/pkg/api/api.go

Line 247 in 96b8dba

func Respond(w http.ResponseWriter, data interface{}, warnings []error) {

[MichaHoffmann]

That cache applies to query frontend though, if you are querying querier directly there should not be a cached response

[shubhamsharmadvlpr]

so we don't expect querier to be faster than backend prometheus in any scenario? any optimisation/caching at sidecar?

flow:
querier --> sidecar --> prometheus

version:
thanos: v0.18.0
prometheus: 2.44

[MichaHoffmann]

I mean it depends on the query ( if you are using thanos engine ) but in this setup not generally. ( That is a pretty old version of thanos you are using there! )