Thanos sidecar upload and prometheus retention period

[yliu138repo]

Hi team,

I am quite new to Thanos. we are keen on this great solution but would like to understand details RE Thanos sidecar.

If I understand correctly, we can reduce the retention period of Prometheus server after integrated with Thanos. This will help to reduce the memory utilisation on Prometheus as this is currently our most concerning issue - OOM crash. As I know, the Thanos sidecar will monitor the immutable TSDB blocks, and upload them to remote storage (typically 2-hour worth block). However, due to network latency or any other factors, it could take longer to upload blocks.

How can we make sure the retention period of Promtheus is configured long enough to make sure the blocks can be safely removed after they have been completely uploaded to object storage? Is there any event from Thanos to notify Prometheus that blocks stored can be removed safely? What would be the suggested retention period on prometheus given the fact that we aim to balance BETWEEN optimize the memory usage on Prometheus instance AND acceptable query latency (assume uploading to GCS)?

Thanks a lot

[MichaHoffmann]

Disclaimer ~ We don't use Sidecar. But we do use thanos receiver and we use it with retention of 2 days right now.

[yliu138repo]

Thanks Michal