Thanos Query Frontend: instant queries.

[anas-aso]

According to the documentation, Query Frontend does not process instant query API requests and simply forward them

_NOTE: Currently only range queries (/api/v1/query_range API call) are actually processed through Query Frontend. All other API calls just directly go to the downstream Querier, which means only range queries are split and cached. But we are planning to support instant queries as well.

But in of the issues, there a mention from a maintainer (@yeya24) that sharding for instant queries is supported.

Currently query sharding is supported by the query frontend to shard both instant and range queries

So I am not sure which one is correct. Can you please clarify ?

[yeya24]

Hey @anas-aso, the doc is outdated. For instant queries, we support vertical sharding to split and cache queries. But for range queries, we support both shard by day and vertical sharding.

[anas-aso]

Hi @yeya24. Thank you for the quick reply.
Honestly I was hoping for the opposite, because it would explain the slow performance I see here:

I am running 2 parallel setups:

1. LB -> Thanos Query -> Thanos Store APIs (Sidecars, StoreGateways, Rulers).
2. LB -> Thanos Query Frontend -> Thanos Query -> Thanos Store APIs (the same Sidecars, the same StoreGateways, different Rulers but the same recording/alerting rules config).
   

The reason I am looking at Query Frontend is that some of our recording rules are supper slow (e.g GCP LB metrics histogram_quantile(0.99, sum by (url_map_name,le) (rate(stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_backend_latencies_bucket[5m])))). I am trying Query Frontend to see if it will improve the performance, but as you can see in the graph above, the performance is even worst.
I am running Query Frontend with the following flags:

--log.format=json
--log.level=debug
--http-address=0.0.0.0:9090
--query-frontend.compress-responses
--query-frontend.downstream-url=http://thanos-query-frontend.experiment.svc.cluster.local
--query-range.align-range-with-step
--cache-compression-type=snappy
--labels.response-cache-config-file=/etc/config/cache.yaml
--labels.split-interval=10m
--query-frontend.downstream-tripper-config-file=/etc/config/downstream-tripper.yaml
--query-frontend.vertical-shards=2
--query-range.horizontal-shards=2
--query-range.max-retries-per-request=1
--query-range.max-split-interval=3h
--query-range.min-split-interval=30m
--query-range.response-cache-config-file=/etc/config/cache.yaml
--query-range.split-interval=0

and the cache config file is as follow:

type: IN-MEMORY
config:
  max_size: 200MB
  validity: 60s

Any idea where did I mess up ?

[yeya24]

Having query frontend doesn't mean it is always faster. In your case, recording rules are instant queries and they are not cached. Enabling vertical sharding will try to split the query by 2 shards, though each shard processes less data right now, it is not 100% faster as postings and series fetched are doubled and there is one more hop in the query path.
I recommend enabling tracing and see which component is slow. Query frontend is not the silver bullet.