Why does thanos-compact exclude the recently producted block before compressing a block

[hanyuting8]

Why does thanos-compact exclude the recently producted block before compressing a block.
Its corresponding code is the plan function in planner.go
metasByMinTime = metasByMinTime[:len(metasByMinTime)-1]

[douglascamata]

If you read the comment 3 lines above, it says:

	// We do not include a recently producted block with max(minTime), so the block which was just uploaded to bucket.
	// This gives users a window of a full block size maintenance if needed.

[hanyuting8]

Thank you for your reply. I've read the comment 3 lines above, but I'm wondering in what scenario the user will maintain the recently producted block.

[douglascamata]

In the scenario mentioned in the second line of the comment: This gives users a window of a full block size maintenance if needed..

[hanyuting8]

Sorry, maybe I didn't describe it clearly. I want to know what kind of needs users might have to maintain this recently producted block size. And how to maintain the size of this block.

[douglascamata]

It still sounds unclear to me. It seems like you want to ask deeply technical questions about things you don't understand yet. What do you want to achieve? Why are you asking these questions?

That line explains why Thanos thinks it's a good idea to leave the most recent block there.

[yeya24]

@hanyuting8 I believe it is the same code as what we have in Prometheus https://github.com/prometheus/prometheus/blob/main/tsdb/compact.go#L221. And Thanos copies that algorithm directly.

// We do not include a recently created block with max(minTime), so the block which was just created from WAL.
// This gives users a window of a full block size to piece-wise backup new data without having to care about data overlap.

a full block size means a time window of 2h.

As the comment says, it excludes the latest block (2h before now) from the compaction. So that even if anything goes wrong in the latest block, operators (or Prometheus admins) have 2h of time to fix or backup data if needed. Rather than compacting the latest block into a new one, which might cause bigger impact if things went wrong.

I don't think it is 100% necessary. If you think it is safe for your own usecase, you can change the algorithm to include the latest block as Thanos code is 100% open source.

But if you want to make a change to Thanos upstream code, it probably require some discussion with Prometheus community.

Hope my answer clarifies.