Understanding compactor compaction levels and resulting disk usage

[hartfordfive]

I've recently encountered some situations where some compactor instances have substantially increased in disk usage. I've increased the compactor disk size but I've also been looking through the available flags to see if there are any changes that could be applied that may help with keeping a more stable disk usage.

I came across the following post:
Compactor: Make debug.max-compaction-level Public
which indicates that setting the --debug.max-compaction-level flag to set the compaction level to 3 which would help stabilize the disk usage. After some digging, I was able to deduce that the levels are determined by the compactionSet type here:

var (
	compactions = compactionSet{
		1 * time.Hour,
		2 * time.Hour,
		8 * time.Hour,
		2 * 24 * time.Hour,
		14 * 24 * time.Hour,
	}
)

so going on a 0-based index, lvl0=1h, lvl1=2h, lvl2=8h, and so on. Logically, I can understand that setting the max compaction level to 3 (2d) should consume less disk space as it will only download 2 days worth of historical blocks from the storage bucket. From a disk performance perspective, I can understand that having a contiguous sequence of bytes in a 2-week block (level 4) could result in better search times, especially on spinning disks. What I'd like to better understand is if setting the max compaction level to 3 (2d) may have some negative consequences or cause a non-negligible performance decrease?

It would be great if some more documentation could be added around this topic or even a blog post, especially for users that like to better understand the lower-level details as well as the reasons behind them. Also, could a short documentation section be added to the Compactor page to describe how to estimate the storage required (e.g.: include a formula that users can apply to their environment and explain each variable in the calculation). This related github issue also discusses the disk usage stabilization with the --debug.max-compaction-level although there's no clear answer as to why the flag is hidden. Is this something that might be moved over to a non-hidden flag in the future?

If any of my assumptions are incorrect or missing context, I would appreciate any clarification. Thanks for your help.

[MichaHoffmann]

I think you can have blocks with compaction level 4 spanning 2h of time ( we get that when receivers upload blocks with replicated samples that are deduplicated and so on ). I think compaction level is only loosely related to the range; its just incremented every multiple blocks get compacted into one.

[douglascamata]

@hartfordfive if you want to wonder about what can happen if you stop 2w blocks from being produced, then you have to think long term and consider downsampling.

For example, let's pick the scenario where you have tenants running queries over 3 months of data.

With only up to 2 days blocks, your Stores will have to download many small blocks and chew through data in many pieces to reply to the query. This seems more friendly to per-block sharding with many Store shards, as work might be better distributed as long as the block id hashing is behaving nicely.

If you had 2 weeks blocks, there would be much less downloads, but blocks are bigger in size (potentially mitigated by downsampling), more data will be present in each block. Here the potential for some Stores to be doing more work than others is higher, as a unit of work (2w block) is relatively big.

It's up to you to consider the tradeoffs of dealing with bigger or smaller pieces of data in your environment. The type of queries your tenants are doing will also have impact on the decision, so it's hard to make a blanket statement on it.

[hartfordfive]

In my current setup for the prometheus instance in question, both 5m and 1h downsampling is enabled for the compactor. Users can end up running queries over a longer period, but usually only to observer a trend over time. Users don't tend to zoom in to something like a 1h period of a metrics say from 4 months ago. The other part I'm currently having difficulty with is estimating the amount of disk a compactor instance might use. I know there is a guideline to help estimate the disk usage which indicates we need to have 2 times 2 weeks worth of smaller blocks (if 2 weeks is the max compaction level) although it still seems a bit unclear how to actually come up with a relatively decent estimate. I've already had to increase the compactor disk size multiple times so I'm hoping I can finally get to a number where the compaction runs successfully. If the compactor fails to run for several weeks, will each run of the compactor possibly take more and more space until all pending blocks have been compacted to their appropriate size?

Finally, which blocks should I look at (using the Compactor UI) in order to get an estimate of how much disk the instance would need in a case where we have up to level 4 compaction (14 days)?

[douglascamata]

If the estimate in the guidelines isn't being helpful, just give the Compactor a huge amount of disk and let it run. Then track it's volume usage and add some extra headroom, then cut back the volume size. That's the most effective way. Estimating in any remotely precise way that is 100% guaranteed to work and won't be a waste isn't an easy job.