Thanos store s3 backend super slow on seaweedfs

[zakiharis]

Hi,

I'm currently testing using seaweedfs for the s3 backend, unfortunately the performance is not good.

I'm not sure if thanos is not compatible with seaweedfs but seaweedfs is also a s3 compatible store.

I'm comparing minio with seaweedfs as our current production thanos is using minio.

thanos store setting

args = [
     "store",
     "--data-dir=/var/thanos/store",
     "--objstore.config-file=/etc/thanos/minio-bucket.yaml",
     "--http-address=0.0.0.0:${NOMAD_PORT_thanos_store_http}",
     "--grpc-address=0.0.0.0:${NOMAD_PORT_thanos_store_grpc}",
]

Loaded blocks same as image above for both minio and seaweedfs (rsync from minio to seaweedfs)

minio

level=info ts=2023-02-22T09:35:44.400529835Z caller=store.go:388 msg="bucket store ready" init_duration=2.004778224s

level=info ts=2023-02-23T05:11:45.356838428Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=466.271206ms duration_ms=466 cached=96 returned=64 partial=0
level=debug ts=2023-02-23T05:14:44.890790301Z caller=fetcher.go:327 component=block.BaseFetcher msg="fetching meta data" concurrency=32

seaweedfs

level=info ts=2023-02-22T09:01:53.198918808Z caller=store.go:388 msg="bucket store ready" init_duration=9m2.517874877s

level=info ts=2023-02-23T05:14:35.623977062Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=3m1.175704616s duration_ms=181175 cached=80 returned=58 partial=20
level=debug ts=2023-02-23T05:14:35.624513704Z caller=fetcher.go:327 component=block.BaseFetcher msg="fetching meta data" concurrency=32

Now here's the catch, I'm also doing a s3 benchmark using minio warp and the result show seaweedfs faster than minio

minio

❱❱❱ ./warp mixed --host minio.query.consul --duration=1m --obj.size 1024B --concurrent 4 --access-key "redacted" --secret-key "redacted"
warp: Benchmark data written to "warp-mixed-2023-02-22[152454]-b1wp.csv.zst"
Mixed operations.
Operation: DELETE, 10%, Concurrency: 4, Ran 59s.
 * Throughput: 17.01 obj/s

Operation: GET, 45%, Concurrency: 4, Ran 59s.
 * Throughput: 0.07 MiB/s, 76.39 obj/s

Operation: PUT, 15%, Concurrency: 4, Ran 59s.
 * Throughput: 0.03 MiB/s, 25.63 obj/s

Operation: STAT, 30%, Concurrency: 4, Ran 59s.
 * Throughput: 51.05 obj/s

Cluster Total: 0.10 MiB/s, 170.03 obj/s over 1m0s.
warp: Cleanup Done.%

seaweedfs

❱❱❱ ./warp mixed --host seaweedfs-s3.query.consul --duration=1m --obj.size 1024B --concurrent 4 --access-key "redacted" --secret-key "redacted"
warp: Benchmark data written to "warp-mixed-2023-02-22[152422]-E01s.csv.zst"
Mixed operations.
Operation: DELETE, 10%, Concurrency: 4, Ran 59s.
 * Throughput: 33.17 obj/s

Operation: GET, 45%, Concurrency: 4, Ran 59s.
 * Throughput: 0.15 MiB/s, 149.49 obj/s

Operation: PUT, 15%, Concurrency: 4, Ran 59s.
 * Throughput: 0.05 MiB/s, 49.99 obj/s

Operation: STAT, 30%, Concurrency: 4, Ran 59s.
 * Throughput: 99.70 obj/s

Cluster Total: 0.19 MiB/s, 332.24 obj/s over 1m0s.
warp: Cleanup Done.%

Query in grafana (Last 30 Days)

irate(node_disk_io_time_seconds_total{instance="node:9100"}[$__rate_interval])*100

minio

level=debug ts=2023-02-23T05:44:33.231856596Z caller=bucket.go:1293 msg="stats query processed" request="min_time:1674536385000 max_time:1677130200000 matchers:<name:\"instance\" value:\"node:9100\" > matchers:<name:\"__name__\" value:\"node_disk_io_time_seconds_total\" > max_resolution_window:360000 aggregates:COUNTER " stats="&{blocksQueried:20 postingsTouched:40 PostingsTouchedSizeSum:153KiB880B postingsToFetch:0 postingsFetched:20 PostingsFetchedSizeSum:148KiB912B postingsFetchCount:20 PostingsFetchDurationSum:839.830329ms cachedPostingsCompressions:20 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:148KiB912B CachedPostingsCompressedSizeSum:18KiB537B CachedPostingsCompressionTimeSum:963.377µs cachedPostingsDecompressions:20 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:227.195µs seriesTouched:1400 SeriesTouchedSizeSum:182KiB635B seriesFetched:280 SeriesFetchedSizeSum:2MiB221KiB992B seriesFetchCount:4 SeriesFetchDurationSum:803.397816ms chunksTouched:16590 ChunksTouchedSizeSum:8MiB632KiB192B chunksFetched:16590 ChunksFetchedSizeSum:81MiB772KiB794B chunksFetchCount:199 ChunksFetchDurationSum:26.763106771s GetAllDuration:60.921099ms mergedSeriesCount:140 mergedChunksCount:16590 MergeDuration:401.397492ms}" err=null

seaweedfs

level=debug ts=2023-02-23T05:27:50.490614436Z caller=bucket.go:1293 msg="stats query processed" request="min_time:1674534585000 max_time:1677128400000 matchers:<name:\"instance\" value:\"node:9100\" > matchers:<name:\"__name__\" value:\"node_disk_io_time_seconds_total\" > max_resolution_window:360000 aggregates:COUNTER " stats="&{blocksQueried:20 postingsTouched:40 PostingsTouchedSizeSum:354KiB964B postingsToFetch:0 postingsFetched:40 PostingsFetchedSizeSum:354KiB964B postingsFetchCount:40 PostingsFetchDurationSum:34m33.431925741s cachedPostingsCompressions:40 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:354KiB964B CachedPostingsCompressedSizeSum:20KiB535B CachedPostingsCompressionTimeSum:3.645035ms cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:0s seriesTouched:592 SeriesTouchedSizeSum:39KiB615B seriesFetched:592 SeriesFetchedSizeSum:2MiB313KiB224B seriesFetchCount:16 SeriesFetchDurationSum:15m51.735955654s chunksTouched:0 ChunksTouchedSizeSum:0B chunksFetched:0 ChunksFetchedSizeSum:0B chunksFetchCount:0 ChunksFetchDurationSum:1h21m43.326906509s GetAllDuration:1m11.347856931s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Unknown desc = receive series from 01GSXEWBR7M8AASQ5PJ3NZS2QA: load chunks: get range reader: Get \"http://seaweedfs-s3.query.consul/thanos/01GSXEWBR7M8AASQ5PJ3NZS2QA/chunks/000001\": context canceled"

any help or advice is appreciated