multi tenancy with sidecar and layered queriers

[mzupan]

I'm trying to do something like this

central aws account with s3 bucket / storegateway
|-> leaf cluster with sidecar cluster=prod
|-> leaf cluster with sidecar cluster=staging

I have sidecar working to ship metrics after 2hr to the s3 bucket and its adding cluster=prod to the prod cluster just fine. My issue is i'm trying to see if I can do a layered querier so in my prod leaf cluster i can just do a metric graph for prometheus_build_info{} and in the central store gateway it just searches for that for cluster=prod

On the leaf clusters I've tried multiple thanos helm deploys like

This one knows about my sidecar prometheus deploys and also the shared storegateway in another aws account. i'm trying to set the selector-label there in hopes to only respond to queries with that label

fullnameOverride: thanos-root-query

query:
  replicaCount: 1
  resources:
    requests: {}
    limits: {}
  dnsDiscovery:
    sidecarsService: "prometheus-operated"
    sidecarsNamespace: "monitoring"
  replicaLabel:
    - __replica__
  stores: 
    - thanos-central-query.domain.co:10901  
  extraFlags:
    - --selector-label=cluster="prod"

Then my main thanos UI install just hits that querier like

fullnameOverride: thanos-ing-query

query:
  replicaCount: 1
  resources:
    requests: {}
    limits: {}
  dnsDiscovery:
    sidecarsService: ""
    sidecarsNamespace: ""
  replicaLabel:
    - __replica__
  extraFlags:
    - --endpoint=thanos-root-query-query-grpc:10901

I'm not sure i can do what i want to do or in leaf clusters I just have to deal with adding cluster=EnvHere to searches. I'm trying to create recording rules on the clusters for 4w out so I don't want rules with no cluster= to glob up the metrics from all leaf clusters in the SLO recording rules.