Data replication in Thanos running in Kubernetes

[pankdhnd]

Hello,

I am planning to deploy Thanos in my Kubernetes cluster, but I have some queries regarding the data replication, for which I could not find a concrete answer anywhere.

I am planning to deploy receiver as a statefulset in Kubernetes and with 4 replicas. I have a Prometheus (one single) instance, which would remote write to Thanos.

Now I understand that --replication-factor flag doesn't actually takes care of data replication, instead it makes sure that the data is successfully written to the given number of replicas, and then communicates success response accordingly.

Also, with hashring algorithm, the Thanos receiver distributor makes sure that incoming load is evenly distributed to receive replicas. So lets say I have 3 timeseries as incoming data. So the distributor would find which timeseries to send to which receiver replica using hashing. The receiver replica would the store the respective time series in its PVC.

Considering functionality of replication-factor and hashring mechanism, each of the receiver replica would have a different time series. That means the data that is present in PVC of each Thanos receiver is different.

So the data replication for resiliency is not available when storing in PVC. Also when the data is sent to object storage, as each replica has different set of data, all of it combined data makes one unique set. All of this data is when stored in object storage, still does not have replication. The only way of replication available is to make backups/replication from the object storage.

This is where I am confused most. I have been reading that Thanos supports data replication, but I don't know how to achieve it. Can someone please confirm if this understanding is correct? If not, then please let me know how the replication happens between each of the Thanos Receiver?

Please let me know on this.