Compact: critical error detected; halting (pre compaction overlap check: overlaps found while gathering blocks)

[thrstnh]

Hello,

we run a prometheus/grafana/thanos setup within a kubernetes cluster.
Sometimes it happens, that the thanos-compact component has an error after a new deployment of it.

level=error ts=2021-12-08T23:10:53.972947051Z caller=compact.go:434 msg="critical error detected; halting" err="compaction: group 0@1839599239947214839: pre compaction overlap check: overlaps found while gathering blocks. [mint: 1638957600560, maxt: 1638964800000, range: 1h59m59s, blocks: 2]: <ulid: 01FPE50GH7SXJBPBRFXFCS8BKJ, mint: 1638950400560, maxt: 1638979200000, range: 7h59m59s>, <ulid: 01FPD22BJNXTSTABYRKT5V4ST4, mint: 1638957600560, maxt: 1638964800000, range: 1h59m59s>"

In this case the block seems to be already integrated into the correct compacted block (above, 'compactor'), but the actual block is still there (below, 'sidecar').

| 01FPE50GH7SXJBPBRFXFCS8BKJ | 08-12-2021 08:00:00 | 08-12-2021 16:00:00 | 7h59m59.44s | 32h0m0.56s | 18,629 | 3,487,693 | 37,995 | 2 | false | job=daisy-pods | 0s | compactor |
| 01FPD22BJNXTSTABYRKT5V4ST4 | 08-12-2021 10:00:00 | 08-12-2021 12:00:00 | 1h59m59.44s | 38h0m0.56s | 9,643 | 1,157,160 | 9,643 | 1 | false | job=daisy-pods | 0s | sidecar |

In this case the compactor stops, we need to delete the block manually and after a restart of the compactor everything is fine again.
Is there a way to prevent this issue?

[yeya24]

Please take a look at https://thanos.io/tip/operating/troubleshooting.md/#overlaps

[ouhdif]

hello,

• thanos: 0.34.1

• same issue here with last image version of "repository: oci://registry-1.docker.io/bitnamicharts"

{"caller":"compact.go:527","err":"compaction: group 0@640391354374641128: pre compaction overlap check: overlaps found while gathering blocks. [mint: 1712854995645, maxt: 1712858400000, range: 56m44s, blocks: 2]: <ulid: 01HV8G12B77NVRC21BMQ5V05XE, mint: 1712851200000, maxt: 1712858400000, range: 2h0m0s>, <ulid: 01HV8CMFTWXD1WAGNCNZNH0BZK, mint: 1712854995645, maxt: 1712880000000, range: 6h56m44s>"