Compaction failures due to object storage timeout

[tagathi]

Hello,
We are recently facing compaction failures. It seems that the compactor fails to upload some blocks to the object storage because of timeout:

ts=2024-05-05T15:45:39.313735915Z caller=bucket_compactor.go:276 level=error component=compactor user=TDI groupKey=0@17241709254077376921-merge--1714608000000-1714694400000 minTime="2024-05-02 00:00:00 +0000 UTC" maxTime="2024-05-03 00:00:00 +0000 UTC" msg="compaction job failed" duration=4m52.053144432s duration_ms=292053 err="upload of 01HX4PNJ71RXM61D7VKBAGAP3A failed: upload index: upload file /data/compact/0@172417092540
77376921-merge--1714608000000-1714694400000/01HX4PNJ71RXM61D7VKBAGAP3A/index as 
 1HX4PNJ71RXM61D7VKBAGAP3A/index: upload object close: Timeout when reading or writing data"

We use a swift object storage. Can we somehow increase the timeout towards our object storage?

At the same time we noticed that in the swift object storage we have a lot of blocks with prefix segments, like for example:

1.0G 2024-05-03 21:44:32 application/octet-stream segments/544/4492f30314858303644464d425a4a48484b3135315244354b41444b532f696e646578c92745b7c2c
16f611ef18fe0114eb2cc8002582f71a94ccec6581d23c45cce15da39a3ee5e6b4b0d3255bfef95601890afd80709/0000000000000001

And they are filling up our object storage. They didn't exist before the compaction failures. Any idea what those are? Are they related to the compaction failures? What can we do to overcome this?

[pstibrany]

Hello. Swift configuration has request_timeout (-<prefix>.swift.request-timeout) flag which defaults to 5s, you may want to increase that. https://grafana.com/docs/mimir/latest/configure/configuration-parameters/#swift_storage_backend

I'd suggest looking at your Swift storage documentation to explain segments prefix. This can be some internal naming of the storage, Mimir itself doesn't upload objects like this.

[tagathi]

It seems that the problem is coming from Thanos-IO which Mimir uses to communicate with Swift. Raised a new issue here