Thanos Store fills up the entire hard disk at startup

[vincent-olivert-riera]

Thanos version used: built from main branch

./thanos --version
thanos, version 0.35.0-dev (branch: main, revision: 3048d991088b2de18f8f126adb7c1ee5511f52a8)
  build user:       root@486648711a5b
  build date:       20240405-02:01:59
  go version:       go1.22.2
  platform:         linux/amd64
  tags:             netgo

I create a brand new VM with a 100GB hard disk, and I install the above build of thanos.
I use only Thanos Store because I want to check something with it, and this is how I run it:

./thanos store --data-dir $PWD/store --objstore.config-file=$PWD/objstore.yml

Thanos starts loading blocks from the S3 bucket (4.67 TB) configured in the objstore.yml file, but eventually it runs out of disk space and starts throwing these errors:

ts=2024-04-05T03:16:03.10319309Z caller=bucket.go:752 level=warn msg="loading block failed" elapsed=26.240655ms id=01GHC2YDM6S
TEBMG48RKTWKMGM err="create index header reader: write index header: new index reader: get TOC from object storage of 01GHC2YD
M6STEBMG48RKTWKMGM/index: create temp dir: mkdir /home/user/store/01GHC2YDM6STEBMG48RKTWKMGM: no space left on device"

Here we can see the hard disk is full, and also the size of the Thanos' data dir:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.8G   16K  1.8G   1% /dev/shm
tmpfs           1.8G  456K  1.8G   1% /run
tmpfs           1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/vda2       100G  100G  302M 100% /
tmpfs           366M     0  366M   0% /run/user/11000

$ du -hs store/
97G     store/

---

I have read this from the documentation:

In general, an average of 6 MB of local disk space is required per TSDB block stored in the object storage bucket, but for high cardinality blocks with large label set it can even go up to 30MB and more. It is for the pre-computed index, which includes symbols and postings offsets as well as metadata JSON.

I understand that Thanos downloads blocks from the S3 bucket, generates some metadata about those blocks, and then adds that metadata to that "pre-computed index" mentioned above.

However, why doesn't Thanos delete the blocks whose metadata has already been added to the pre-computed index to avoid filling up the entire hard disk? Or does that pre-computed index actually need such amount of space?

[GiedriusS]

Is your compactor running properly? Is the compaction backlog small (thanos_compact_todo_compactions)?

[vincent-olivert-riera]

Is your compactor running properly? Is the compaction backlog small (thanos_compact_todo_compactions)?

Uhm..., from what I see, Thanos Compactor is not able to generate level-3 blocks.
Also, the metric you requested:

thanos_compact_todo_compactions 13274

It seems quite high.

So, let's see if my understanding is correct.
Because of Thanos Compactor is not able to do its job and compact the blocks, there are too many small blocks in the bucket, and that is causing the issue described in this thread?

[GiedriusS]

Yep, that's a huge backlog! It is necessary to keep that metric as low as possible. If you have lots of small blocks then the Store needs to build a lot of index headers that have in essence duplicated data due to the compaction process being stuck.