Thanos receiver is unable to ingest data stating msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3

[dmilind]

Hi Folks,
while playing with Thanos receive I am facing an issue where Thanos receive errors out stating: msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3
One time I got "sample not in order".
When this is happening I observed remote storage getting loaded by multiple blocks and eating storage space.
I am trying to understand why this is happening. Please help me to understand this behavior.

Giving more details here:

1. Setup Thanos cluster with receive
2. setup hpa for receive, min-3 max-15 (on 85% CPU)
3. one Prometheus instance scraping Thanos components and writing to receive (for testing purpose)

Behavior:
When there is no Prometheus running, Thanos receive is with 3 replicas and working fine with CPU utilization is (1~4%). When I deploy Prometheus instance, it starts sending blocks to remote receive, while checking on logs it shows an Error in ingesting samples that are too old or are too far into the future and within a min received shows spikes in memory with goes beyond 930%/85% and spins all replica. Also, remote storage is getting filled by multiple blocks.

after some time comes to 18%/85% and then again it shoots to >700%.
For testing, I stopped Prometheus and then hit receive with a load of Loki metrics (using avalanche) there also I am getting the same error.

Can anyone please explain what is happening, why such behaviour by receive, how can I stop this >