Recursive reads from object storage subfolders using single Store Gateway

[rozek1997]

Hi folks. I'm coming with ask for help, because my Store Gateway is not able to see any metrics on GCS Object Storage, which metrics are in subfolders. I will start with little description of our current architecture.

We have 5 different production grade clusters, each hosting Kube Prometheus Stack Helm Chart components. Each cluster have single Prometheus running with Thanos sidecar. Every sidecar is uploading metrics blocks to the same bucket but to differrent "subfolders" on GCS object storage(this division is made for readability purpose in Object Storage console). We have also central cluster which host Thanos: Thanos query,query frontend, compactor, store gateway. I can confirm that from Thanos Query component I'm able to reach each sidecar on production clusters as well as store gateway and they are up

Despite of everything seems to be working fine, I'm not able to see any metrics which have been uploaded to object storage. The only metrics visible are those which are coming from Thanos sidecars. When I'm disabling access to sidecars there is no data visible in Thanos Query. Moreover Store Gateway returns message successfully synchronized block metadata" duration=153.603367ms duration_ms=153 cached=0 returned=0 partial=0 which from my understanding means that it was not able to find any Proemtheus block , which is not true. The blocks are in bucket, bot not in root folder but rather in subfolders.

Is Thanos not able to find Prometheus block recursively from bucket root? There is option to specify prefix in Object Storage config but from documentation I've understood that is used for sharding between diffrent store gateway running in pararell, am I right? In that case how to configure Thanos to read metrics blocks recursively from root folder in bucket?