ALERTS and ALERTS_FOR_STATE metrics in S3

[erikeverts]

Hi,

we have set up HA Thanos, with a receiver, ruler, store, querier and alertmanager. We have set up objectstore to write to S3 and use this config in ruler, receiver and store. In querier we use two "stores" the thanos store, to get all the metrics from s3 and thanos receiver as backup to get metrics that might not have been written to S3 yet.

We noticed that any ALERTS or ALERTS_FOR_STATE metrics that are generated by thanor ruler, OR have been received by thanos receiver do not end up in S3, or at least are not found by thanos query. I can only see metrics for these two types in thanos query which come from the thanos receiver "store" which is only temporary because those metrics should get written to S3 and removed from the local store.

Is this behaviour by design, filtering out these metrics when writing to S3, or is this a bug?

[GiedriusS]

They should end up in S3 🤔 It's not quite clear to me: have you configured object storage for Thanos Ruler that points to the same S3 bucket?

[erikeverts]

yes, same S3 bucket as where thanos store is reading from and where receiver is also writing to. Also ALERTS and ALERTS_FOR_STATE metrics that are received by receiver are not stored in S3. I can only see them in thanos query for the reveiver store, not for the thanos-store

[erikeverts]

Any suggstions what it may be? They really are not stored in s3, both from thanos receiver and thanos ruler, but other metrics are stored in s3 from receiver.