Compactor is stuck, S3 storage constantly growing #7176

ggabijaa · 2024-03-04T08:49:50Z

ggabijaa
Mar 4, 2024

Hello there,

I'm encountering an issue with the Thanos compactor that appears to be stuck, leading to continuous growth in S3 storage.

Bit of context:
Several months ago, we did a migration from one S3 bucket to another. This was achieved using Thanos tools (bucket tools replicate) . During this time the compactor was turned off, and turned on after fully switching buckets.

After the migration, we noticed that the compactor wasn't completing its compactions and failed to delete blocks marked for deletion. To address this, we took action, which seemed to alleviate the issue temporarily (storage decreased by 40 TB, returning to previous levels, and the todo compactions metric began to shrink, eventually reaching 0).
What we did was :

increasing compact.concurrency from 1 to 2
increasing k8s memory limits from 128Gi to 256Gi

However, we're now facing the same challenges once more. The todo compaction/downsampling metrics aren't decreasing, and S3 storage continues to grow. Compactor is not halted.

Here's some of our compactor configuration (Running in K8s):

containerArgs: [
  "compact",
  "--compact.concurrency=2",
  "--log.level=debug",
  "--log.format=json",
  "--wait",
  "--data-dir=/data",
  "--objstore.config-file=/config/bucket.yml",
  "--retention.resolution-raw=31d",
  "--retention.resolution-5m=31d",
  "--retention.resolution-1h=31d",
  "--consistency-delay=40m",
  "--hash-func=SHA256",
  "--selector.relabel-config-file=/config/relabel-1mo.yml",
  "--delete-delay=12h"
]

resources:
  limits:
    cpu: 20
    memory: 256Gi

version: v0.32.5

Any insights or assistance you can provide would be greatly appreciated. Thanks in advance!

yeya24 · 2024-03-04T09:01:25Z

yeya24
Mar 4, 2024
Maintainer

Can you check the compactor log or metric and see if there it is hanging?

1 reply

ggabijaa Mar 4, 2024
Author

Compactor is not halted: thanos_compact_halted is 0

Logs sometimes show these errors, but don't think they are related:
{"caller":"block.go:68","err":"opening file: open /data/compact/0@3321434660821436344/01HQHD0D8BDAFBJT4HGWSNTKAC/chunks/000216: no such file or directory","group":"0@{monitor=\"prtapp\", region=\"dk\", replica=\"vmname\", retention=\"1mo\", tier=\"pr\"}","groupKey":"0@3321434660821436344","level":"info","msg":"failed to calculate hash when downloading; re-downloading","relPath":"chunks/000216","ts":"2024-03-04T04:03:19.124689854Z"}

{"caller":"compact.go:502","err":"compaction: 2 errors: group 0@3321434660821436344: download block 01HQGN3KC5HHY5CFR03JX1AVZY: context canceled; group 0@18324289402901941230: download block 01HQBK6HVPPEA6N2M6NCFTNSGC: context canceled","level":"error","msg":"retriable error","ts":"2024-03-04T02:49:41.226809712Z"}

Any other metric i can check?

GiedriusS · 2024-03-05T07:54:47Z

GiedriusS
Mar 5, 2024
Maintainer

Maybe you could check through /debug/pprof and the goroutine stack trace dumps whether it is stuck or if the Compactor is doing something?

6 replies

GiedriusS Mar 5, 2024
Maintainer

Maybe you could upload the whole output? The most important parts are missing from the screenshot

ggabijaa Mar 5, 2024
Author

Yes, sure. here it is:

goroutine 61160 [running]:
runtime/pprof.writeGoroutineStacks({0x2c5fbe0, 0xc0002388c0})
	/usr/local/go/src/runtime/pprof/pprof.go:703 +0x6a
runtime/pprof.writeGoroutine({0x2c5fbe0?, 0xc0002388c0?}, 0xc001400000?)
	/usr/local/go/src/runtime/pprof/pprof.go:692 +0x25
runtime/pprof.(*Profile).WriteTo(0x23f5ae0?, {0x2c5fbe0?, 0xc0002388c0?}, 0xc?)
	/usr/local/go/src/runtime/pprof/pprof.go:329 +0x146
net/http/pprof.handler.ServeHTTP({0xc00340c011, 0x9}, {0x2c73090, 0xc0002388c0}, 0x2c5e500?)
	/usr/local/go/src/net/http/pprof/pprof.go:267 +0x4a8
net/http/pprof.Index({0x2c73090?, 0xc0002388c0}, 0xc001b0ac00?)
	/usr/local/go/src/net/http/pprof/pprof.go:384 +0xe5
net/http.HandlerFunc.ServeHTTP(0x446540?, {0x2c73090?, 0xc0002388c0?}, 0x799f9a?)
	/usr/local/go/src/net/http/server.go:2136 +0x29
net/http.(*ServeMux).ServeHTTP(0x42a32c0?, {0x2c73090, 0xc0002388c0}, 0xc001b0ac00)
	/usr/local/go/src/net/http/server.go:2514 +0x142
net/http.serverHandler.ServeHTTP({0xc00370c030?}, {0x2c73090?, 0xc0002388c0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc005504090, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 1 [chan receive, 31 minutes]:
github.com/oklog/run.(*Group).Run(0xc000920d20)
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0x155
main.main()
	/app/cmd/thanos/main.go:159 +0x1878

goroutine 6 [select, 31 minutes]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
	/go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:375 +0xa8
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger in goroutine 1
	/go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:368 +0x125

goroutine 34 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0006d4600)
	/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 325 [chan send, 9 minutes]:
golang.org/x/sync/errgroup.(*Group).Go(0xc005514a80, 0xc00568c2c0)
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:68 +0x32
github.com/thanos-io/thanos/pkg/compact.(*Group).compact.func2(...)
	/app/pkg/compact/compact.go:1081
github.com/thanos-io/thanos/pkg/compact.(*Group).compact(0xc0055f5180, {0x2c7db70, 0xc004ff57d0}, {0xc004fc96e0, 0x23}, {0x2c5e700, 0xc000c3c4b0}, {0x2c724f8, 0xc000c19da0}, {0x2c5f220, ...}, ...)
	/app/pkg/compact/compact.go:1119 +0xc5d
github.com/thanos-io/thanos/pkg/compact.(*Group).Compact.func2({0x2c7db70, 0xc004ff57d0?})
	/app/pkg/compact/compact.go:830 +0x9d
github.com/thanos-io/thanos/pkg/tracing.DoInSpanWithErr({0x2c7dba8?, 0xc000825e50?}, {0x2676ccb?, 0x9?}, 0xc000811e10, {0xc004fd1470?, 0xc0005ac000?, 0xc000d00b40?})
	/app/pkg/tracing/tracing.go:82 +0xbf
github.com/thanos-io/thanos/pkg/compact.(*Group).Compact(0xc0055f5180, {0x2c7dba8, 0xc000825e50}, {0xc0006d9140, 0xd}, {0x2c5e700, 0xc000c3c4b0}, {0x2c724f8, 0xc000c19da0}, {0x2c5f220, ...}, ...)
	/app/pkg/compact/compact.go:829 +0x3ae
github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact.func2()
	/app/pkg/compact/compact.go:1373 +0x179
created by github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact in goroutine 298
	/app/pkg/compact/compact.go:1370 +0x8d6

goroutine 98 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730d77720, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000cc2000?, 0x5?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000cc2000)
	/usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc000cc2000)
	/usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000ca8020)
	/usr/local/go/src/net/tcpsock_posix.go:152 +0x1e
net.(*TCPListener).Accept(0xc000ca8020)
	/usr/local/go/src/net/tcpsock.go:315 +0x30
net/http.(*Server).Serve(0xc000c16690, {0x2c73bd0, 0xc000ca8020})
	/usr/local/go/src/net/http/server.go:3056 +0x364
github.com/prometheus/exporter-toolkit/web.Serve({0x2c73bd0, 0xc000ca8020}, 0xc000c16690, 0xc000ca4048, {0x2c5ec40?, 0xc00013b4f0})
	/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:278 +0x34e
github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1()
	/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:232 +0x2b
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 297
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x96

goroutine 296 [chan receive, 31 minutes]:
main.main.func2()
	/app/cmd/thanos/main.go:119 +0x2d
github.com/oklog/run.(*Group).Run.func1({0xc000920d38?, 0xc0002cd1a0?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67

goroutine 297 [semacquire, 31 minutes]:
sync.runtime_Semacquire(0xc000ca40c0?)
	/usr/local/go/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x248f640?)
	/usr/local/go/src/sync/waitgroup.go:116 +0x48
golang.org/x/sync/errgroup.(*Group).Wait(0xc000ca21c0)
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:53 +0x25
github.com/prometheus/exporter-toolkit/web.ServeMultiple({0xc000c98240, 0x1, 0x266fb05?}, 0xc000c16690, 0xc000ca4048, {0x2c5ec40?, 0xc00013b4f0})
	/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:235 +0x145
github.com/prometheus/exporter-toolkit/web.ListenAndServe(0x0?, 0xc000ca4048, {0x2c5ec40, 0xc00013b4f0})
	/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:268 +0x456
github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe(0xc000678a10)
	/app/pkg/server/http/http.go:85 +0x22d
main.runCompact.func1()
	/app/cmd/thanos/compact.go:195 +0x2f
github.com/oklog/run.(*Group).Run.func1({0xc0002c6960?, 0xc0002c6980?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67

goroutine 298 [select, 30 minutes]:
github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact(0xc000979a40, {0x2c7db70?, 0xc000c3c120})
	/app/pkg/compact/compact.go:1452 +0xc85
main.runCompact.func7()
	/app/cmd/thanos/compact.go:426 +0xce
main.runCompact.func8.1()
	/app/cmd/thanos/compact.go:481 +0x4a
github.com/thanos-io/thanos/pkg/runutil.Repeat(0xc000509ec8?, 0xc0006cca20, 0xc000815f40)
	/app/pkg/runutil/runutil.go:74 +0x82
main.runCompact.func8()
	/app/cmd/thanos/compact.go:480 +0x18d
github.com/oklog/run.(*Group).Run.func1({0xc000c86320?, 0xc00057f2e0?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67

goroutine 299 [select]:
github.com/thanos-io/thanos/pkg/runutil.Repeat(0xc000c78a80?, 0xc0006cca20, 0xc00564af40)
	/app/pkg/runutil/runutil.go:77 +0xcb
main.runCompact.func12()
	/app/cmd/thanos/compact.go:545 +0x10c
github.com/oklog/run.(*Group).Run.func1({0xc000c78b00?, 0xc000c74d30?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67

goroutine 300 [select, 1 minutes]:
github.com/thanos-io/thanos/pkg/runutil.Repeat(0xc000109278?, 0xc0006cca20, 0xc001612f50)
	/app/pkg/runutil/runutil.go:77 +0xcb
main.runCompact.func14()
	/app/cmd/thanos/compact.go:563 +0xaf
github.com/oklog/run.(*Group).Run.func1({0xc000c78b40?, 0xc000c74d40?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67

goroutine 301 [select]:
github.com/thanos-io/thanos/pkg/runutil.Repeat(0x0?, 0xc0006cca20, 0xc0009d7f18)
	/app/pkg/runutil/runutil.go:77 +0xcb
main.runCompact.func16()
	/app/cmd/thanos/compact.go:591 +0x2e5
github.com/oklog/run.(*Group).Run.func1({0xc000c63380?, 0xc000c74d50?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67

goroutine 302 [select, 31 minutes]:
main.interrupt({0x2c5ec80, 0xc000acd280}, 0xc000c725a0)
	/app/cmd/thanos/main.go:170 +0x105
main.main.func4()
	/app/cmd/thanos/main.go:143 +0x1f
github.com/oklog/run.(*Group).Run.func1({0xc000c76480?, 0xc000c74f20?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67

goroutine 303 [select, 31 minutes]:
main.reload({0x2c5ec80, 0xc000acd280}, 0xc000c72600, 0x0?)
	/app/cmd/thanos/main.go:183 +0xf2
main.main.func6()
	/app/cmd/thanos/main.go:153 +0x25
github.com/oklog/run.(*Group).Run.func1({0xc000c7c7b0?, 0xc000c74f30?})
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
	/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x67

goroutine 305 [syscall, 31 minutes]:
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 303
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 326 [chan send, 30 minutes]:
golang.org/x/sync/errgroup.(*Group).Go(0xc005427000, 0xc005427080)
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:68 +0x32
github.com/thanos-io/thanos/pkg/compact.(*Group).compact.func2(...)
	/app/pkg/compact/compact.go:1081
github.com/thanos-io/thanos/pkg/compact.(*Group).compact(0xc0055f4640, {0x2c7db70, 0xc005420db0}, {0xc0049364e0, 0x24}, {0x2c5e700, 0xc000c3c4b0}, {0x2c724f8, 0xc000c19da0}, {0x2c5f220, ...}, ...)
	/app/pkg/compact/compact.go:1119 +0xc5d
github.com/thanos-io/thanos/pkg/compact.(*Group).Compact.func2({0x2c7db70, 0xc005420db0?})
	/app/pkg/compact/compact.go:830 +0x9d
github.com/thanos-io/thanos/pkg/tracing.DoInSpanWithErr({0x2c7dba8?, 0xc000825e50?}, {0x2676ccb?, 0x9?}, 0xc00269de10, {0xc005478370?, 0xc0005ac000?, 0xc000d00c00?})
	/app/pkg/tracing/tracing.go:82 +0xbf
github.com/thanos-io/thanos/pkg/compact.(*Group).Compact(0xc0055f4640, {0x2c7dba8, 0xc000825e50}, {0xc0006d9140, 0xd}, {0x2c5e700, 0xc000c3c4b0}, {0x2c724f8, 0xc000c19da0}, {0x2c5f220, ...}, ...)
	/app/pkg/compact/compact.go:829 +0x3ae
github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact.func2()
	/app/pkg/compact/compact.go:1373 +0x179
created by github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact in goroutine 298
	/app/pkg/compact/compact.go:1370 +0x8d6

goroutine 46341 [IO wait]:
internal/poll.runtime_pollWait(0x7fd7308973c8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000821c00?, 0xc001073000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000821c00, {0xc001073000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000821c00, {0xc001073000?, 0x7b3220?, 0xc000901540?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc003b62080, {0xc001073000?, 0x0?, 0xc001c34b60?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*persistConn).Read(0xc000968360, {0xc001073000?, 0xc000ce0180?, 0xc002139d38?})
	/usr/local/go/src/net/http/transport.go:1954 +0x4a
bufio.(*Reader).fill(0xc0051c0900)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc0051c0900, 0x1)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*persistConn).readLoop(0xc000968360)
	/usr/local/go/src/net/http/transport.go:2118 +0x1b9
created by net/http.(*Transport).dialConn in goroutine 46336
	/usr/local/go/src/net/http/transport.go:1776 +0x169f

goroutine 63040 [IO wait]:
internal/poll.runtime_pollWait(0x7fd7308975b8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000cc2200?, 0xc002f36000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000cc2200, {0xc002f36000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000cc2200, {0xc002f36000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc001538008, {0xc002f36000?, 0x0?, 0xc004a8c3f8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc004a8c3f0, {0xc002f36000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc004476120)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc004476120, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc005504000, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 63193 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730d76a88, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000c3ec80?, 0xc000945000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c3ec80, {0xc000945000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000c3ec80, {0xc000945000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc001aa2018, {0xc000945000?, 0x0?, 0xc0006382a8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc0006382a0, {0xc000945000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc004a7a000)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc004a7a000, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc0046ba000, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 63016 [IO wait]:
internal/poll.runtime_pollWait(0x7fd7308971d8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000c3e900?, 0xc004d7a000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c3e900, {0xc004d7a000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000c3e900, {0xc004d7a000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc002c98020, {0xc004d7a000?, 0x0?, 0xc005e002a8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc005e002a0, {0xc004d7a000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc00439a420)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc00439a420, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc001552b40, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 62823 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730d77628, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc00025c980?, 0xc000ab9000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00025c980, {0xc000ab9000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc00025c980, {0xc000ab9000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0008a4008, {0xc000ab9000?, 0x0?, 0xc004a8c158?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc004a8c150, {0xc000ab9000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc00439a180)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc00439a180, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc0015525a0, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 1006 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0c058, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000820080?, 0xc0011b6000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000820080, {0xc0011b6000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000820080, {0xc0011b6000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000cc0020, {0xc0011b6000?, 0x0?, 0xc0006380f8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc0006380f0, {0xc0011b6000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc001d88360)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc001d88360, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc001c40000, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 10578 [runnable]:
github.com/prometheus/prometheus/tsdb/encoding.(*Decbuf).Uvarint64(0xc00062d2d0)
	/go/pkg/mod/github.com/prometheus/prometheus@v0.46.1-0.20230818184859-4d8e380269da/tsdb/encoding/encoding.go:238 +0xe5
github.com/prometheus/prometheus/tsdb/encoding.(*Decbuf).Uvarint(...)
	/go/pkg/mod/github.com/prometheus/prometheus@v0.46.1-0.20230818184859-4d8e380269da/tsdb/encoding/encoding.go:181
github.com/prometheus/prometheus/tsdb/index.(*Decoder).Series(0xc001aa2000, {0x7fd31ebb5a72?, 0x10be14a70?, 0x280?}, 0xc00062d668, 0xc00062d650)
	/go/pkg/mod/github.com/prometheus/prometheus@v0.46.1-0.20230818184859-4d8e380269da/tsdb/index/index.go:1844 +0xd0
github.com/prometheus/prometheus/tsdb/index.(*Reader).Series(0xc004e42cc0, 0x10be14a7, 0x20?, 0xc00358af00?)
	/go/pkg/mod/github.com/prometheus/prometheus@v0.46.1-0.20230818184859-4d8e380269da/tsdb/index/index.go:1605 +0x125
github.com/thanos-io/thanos/pkg/block.GatherIndexHealthStats({_, _}, {_, _}, _, _)
	/app/pkg/block/index.go:274 +0x5f6
github.com/thanos-io/thanos/pkg/compact.(*Group).compact.(*Group).compact.func2.func7.2({0x2c7dba8?, 0xc005414b90?})
	/app/pkg/compact/compact.go:1094 +0xdd
github.com/thanos-io/thanos/pkg/tracing.DoInSpanWithErr({0x2c7dba8?, 0xc005414b90?}, {0x269cff9?, 0x8?}, 0xc00062df38, {0xc00521da20?, 0x0?, 0x4107e5?})
	/app/pkg/tracing/tracing.go:82 +0xbf
github.com/thanos-io/thanos/pkg/compact.(*Group).compact.(*Group).compact.func2.func7()
	/app/pkg/compact/compact.go:1093 +0x5cb
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 326
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x96

goroutine 62913 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0a918, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc00025ca80?, 0xc00301f000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00025ca80, {0xc00301f000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc00025ca80, {0xc00301f000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc005244018, {0xc00301f000?, 0x0?, 0xc00385e308?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc00385e300, {0xc00301f000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc004476000)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc004476000, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc0001c7c20, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 1373 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0bf60, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc001b98100?, 0xc001513000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001b98100, {0xc001513000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc001b98100, {0xc001513000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000cc0058, {0xc001513000?, 0x0?, 0xc002554ab8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc002554ab0, {0xc001513000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc00144f320)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc00144f320, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc00064c3f0, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 8057 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0b990, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc001e9c000?, 0xc003e31000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001e9c000, {0xc003e31000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc001e9c000, {0xc003e31000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc002c98048, {0xc003e31000?, 0x0?, 0xc001f80c38?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc001f80c30, {0xc003e31000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc005a0eae0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc005a0eae0, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc001552360, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 62859 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0a728, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000cc2500?, 0xc004713000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000cc2500, {0xc004713000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000cc2500, {0xc004713000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc003b62040, {0xc004713000?, 0x0?, 0xc001c986f8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc001c986f0, {0xc004713000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc00439b260)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc00439b260, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc005504a20, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 46342 [select]:
net/http.(*persistConn).writeLoop(0xc000968360)
	/usr/local/go/src/net/http/transport.go:2421 +0xe5
created by net/http.(*Transport).dialConn in goroutine 46336
	/usr/local/go/src/net/http/transport.go:1777 +0x16f1

goroutine 63149 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0b4b8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000c3ea00?, 0xc001ab7000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c3ea00, {0xc001ab7000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000c3ea00, {0xc001ab7000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc001538040, {0xc001ab7000?, 0x0?, 0xc0052151d8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc0052151d0, {0xc001ab7000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc000c40a80)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc000c40a80, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc005748480, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 63367 [runnable]:
syscall.Syscall(0x4e4edf?, 0xc000302738?, 0x4e5107?, 0x7ffff800000?)
	/usr/local/go/src/syscall/syscall_linux.go:69 +0x25
syscall.read(0xc00543c680?, {0xc005908000?, 0x0?, 0x0?})
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:721 +0x38
syscall.Read(...)
	/usr/local/go/src/syscall/syscall_unix.go:181
internal/poll.ignoringEINTRIO(...)
	/usr/local/go/src/internal/poll/fd_unix.go:736
internal/poll.(*FD).Read(0xc00543c680, {0xc005908000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:160 +0x2ae
net.(*netFD).Read(0xc00543c680, {0xc005908000?, 0x0?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc003982000, {0xc005908000?, 0x28?, 0xc00073a412?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*persistConn).Read(0xc0014b38c0, {0xc005908000?, 0x0?, 0x0?})
	/usr/local/go/src/net/http/transport.go:1954 +0x4a
bufio.(*Reader).Read(0xc005194000, {0xc005908000, 0x8000, 0x0?})
	/usr/local/go/src/bufio/bufio.go:230 +0xe3
io.(*LimitedReader).Read(0xc00516c000, {0xc005908000?, 0xc0003029e0?, 0x417d27?})
	/usr/local/go/src/io/io.go:480 +0x42
net/http.(*body).readLocked(0xc00189c000, {0xc005908000?, 0xc000303e40?, 0xc002e051c0?})
	/usr/local/go/src/net/http/transfer.go:839 +0x3b
net/http.(*body).Read(0xc0000735d0?, {0xc005908000?, 0xc000680000?, 0x43e2ae?})
	/usr/local/go/src/net/http/transfer.go:831 +0x118
net/http.(*bodyEOFSignal).Read(0xc00189c040, {0xc005908000, 0x8000, 0x8000})
	/usr/local/go/src/net/http/transport.go:2810 +0x13f
github.com/minio/minio-go/v7.glob..func2({0x2c5f800, 0xc00189c040}, {0xc005908000, 0x8000, 0x8000})
	/go/pkg/mod/github.com/minio/minio-go/v7@v7.0.61/utils.go:400 +0xbb
github.com/minio/minio-go/v7.(*Client).GetObject.func1()
	/go/pkg/mod/github.com/minio/minio-go/v7@v7.0.61/api-get-object.go:208 +0xda3
created by github.com/minio/minio-go/v7.(*Client).GetObject in goroutine 63366
	/go/pkg/mod/github.com/minio/minio-go/v7@v7.0.61/api-get-object.go:62 +0x349

goroutine 63366 [chan receive]:
github.com/minio/minio-go/v7.(*Object).doGetRequest(_, {{0xc005908000, 0x8000, 0x8000}, 0x12230000, 0x0, 0x1, 0x0, 0x1, 0x0, ...})
	/go/pkg/mod/github.com/minio/minio-go/v7@v7.0.61/api-get-object.go:310 +0x12c
github.com/minio/minio-go/v7.(*Object).Read(0xc00569e200, {0xc005908000?, 0xc000c40000?, 0xc005908000?})
	/go/pkg/mod/github.com/minio/minio-go/v7@v7.0.61/api-get-object.go:384 +0x245
github.com/thanos-io/objstore.(*timingReadCloser).Read(0xc005504120, {0xc005908000?, 0x4f9be5?, 0xc000cc0008?})
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/objstore.go:705 +0x2c
github.com/thanos-io/objstore/tracing/opentracing.(*tracingReadCloser).Read(0xc00189c080, {0xc005908000?, 0xc000cc0008?, 0xc002e05ad0?})
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/tracing/opentracing/opentracing.go:165 +0x2c
io.copyBuffer({0x2c5eae0, 0xc000cc0008}, {0x7fd730b58948, 0xc00189c080}, {0x0, 0x0, 0x0})
	/usr/local/go/src/io/io.go:430 +0x1a6
io.Copy(...)
	/usr/local/go/src/io/io.go:389
os.genericReadFrom(0xc000cc0008?, {0x7fd730b58948?, 0xc00189c080?})
	/usr/local/go/src/os/file.go:161 +0x35
os.(*File).ReadFrom(0xc000cc0008, {0x7fd730b58948, 0xc00189c080})
	/usr/local/go/src/os/file.go:155 +0x9c
io.copyBuffer({0x2c5e1c0, 0xc000cc0008}, {0x7fd730b58948, 0xc00189c080}, {0x0, 0x0, 0x0})
	/usr/local/go/src/io/io.go:416 +0x147
io.Copy(...)
	/usr/local/go/src/io/io.go:389
github.com/thanos-io/objstore.DownloadFile({0x2c7dba8, 0xc005e1c370}, {0x2c5ec40?, 0xc005414780?}, {0x7fd730b58898, 0xc00057e490}, {0xc004e1cf60, 0x28}, {0xc00569a0a0, 0x4c})
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/objstore.go:329 +0x448
github.com/thanos-io/objstore.DownloadDir.func1.1()
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/objstore.go:368 +0x639
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 46776
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x96

goroutine 63364 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0acf8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000cc2700?, 0xc001c23000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000cc2700, {0xc001c23000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000cc2700, {0xc001c23000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc003acc010, {0xc001c23000?, 0x0?, 0xc004c56338?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc004c56330, {0xc001c23000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc00439a0c0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc00439a0c0, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc0015526c0, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 57443 [select]:
net/http.(*persistConn).writeLoop(0xc0014b38c0)
	/usr/local/go/src/net/http/transport.go:2421 +0xe5
created by net/http.(*Transport).dialConn in goroutine 57724
	/usr/local/go/src/net/http/transport.go:1777 +0x16f1

goroutine 56974 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0bb80, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000916180?, 0xc001281000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000916180, {0xc001281000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000916180, {0xc001281000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc005244000, {0xc001281000?, 0x0?, 0xc00560c0f8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc00560c0f0, {0xc001281000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc0001287e0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc0001287e0, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc004ab6000, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 61155 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0ba88, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc00025cc00?, 0xc002114000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00025cc00, {0xc002114000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc00025cc00, {0xc002114000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc003acc008, {0xc002114000?, 0x0?, 0xc00385e3f8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc00385e3f0, {0xc002114000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc00439a1e0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc00439a1e0, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc005504240, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 57442 [select]:
net/http.(*persistConn).readLoop(0xc0014b38c0)
	/usr/local/go/src/net/http/transport.go:2238 +0xd25
created by net/http.(*Transport).dialConn in goroutine 57724
	/usr/local/go/src/net/http/transport.go:1776 +0x169f

goroutine 46777 [select]:
github.com/minio/minio-go/v7.(*Client).listObjects.func2(0xc0051c0a80)
	/go/pkg/mod/github.com/minio/minio-go/v7@v7.0.61/api-list.go:340 +0x528
created by github.com/minio/minio-go/v7.(*Client).listObjects in goroutine 46776
	/go/pkg/mod/github.com/minio/minio-go/v7@v7.0.61/api-list.go:314 +0x307

goroutine 46333 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc001f0d5c8?)
	/usr/local/go/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x0?)
	/usr/local/go/src/sync/waitgroup.go:116 +0x48
golang.org/x/sync/errgroup.(*Group).Wait(0xc0027f9b80)
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:53 +0x25
github.com/thanos-io/objstore.DownloadDir({0x2c7db70, 0xc0059e8240}, {0x2c5ec40?, 0xc005414780}, {0x7fd730b58898?, 0xc00057e490}, {0xc00374d120, 0x1a}, {0xc00374d140, 0x1a}, ...)
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/objstore.go:381 +0x3f6
github.com/thanos-io/thanos/pkg/block.Download({0x2c7db70, 0xc0059e8240}, {0x2c5ec40, 0xc005414780}, {0x2c939b8?, 0xc00057e490}, {0x1, 0x8d, 0xe6, 0xf2, ...}, ...)
	/app/pkg/block/block.go:77 +0x969
github.com/thanos-io/thanos/pkg/compact.(*Group).compact.(*Group).compact.func2.func7.1({0x2c7db70, 0xc0059e8240})
	/app/pkg/compact/compact.go:1084 +0x109
github.com/thanos-io/thanos/pkg/tracing.DoInSpanWithErr({0x2c7dba8?, 0xc004fcf860?}, {0x2690d23?, 0x8?}, 0xc006390ee0, {0xc000927030?, 0x426e680?, 0x0?})
	/app/pkg/tracing/tracing.go:82 +0xbf
github.com/thanos-io/thanos/pkg/compact.(*Group).compact.(*Group).compact.func2.func7()
	/app/pkg/compact/compact.go:1083 +0x1e9
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 325
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x96

goroutine 62652 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730d76c78, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0006d4800?, 0xc0015fd000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0006d4800, {0xc0015fd000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0006d4800, {0xc0015fd000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0008a4000, {0xc0015fd000?, 0x0?, 0xc00385e128?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc00385e120, {0xc0015fd000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc004c740c0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc004c740c0, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc005538240, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 62908 [IO wait]:
internal/poll.runtime_pollWait(0x7fd7308974c0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000cc2300?, 0xc003e0b000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000cc2300, {0xc003e0b000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000cc2300, {0xc003e0b000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc002c98000, {0xc003e0b000?, 0x0?, 0xc004a8c248?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc004a8c240, {0xc003e0b000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc004c74120)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc004c74120, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc001552120, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 62962 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0ab08, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000c3e580?, 0xc0029a5000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c3e580, {0xc0029a5000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000c3e580, {0xc0029a5000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc002c98008, {0xc0029a5000?, 0x0?, 0xc004a8c2a8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc004a8c2a0, {0xc0029a5000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc0044760c0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc0044760c0, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc001552900, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 63377 [IO wait]:
internal/poll.runtime_pollWait(0x7fd7308977a8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000c3ea80?, 0xc00370c041?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c3ea80, {0xc00370c041, 0x1, 0x1})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000c3ea80, {0xc00370c041?, 0x12000001?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000cc00c0, {0xc00370c041?, 0xc00439a900?, 0xc003b85020?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).backgroundRead(0xc00370c030)
	/usr/local/go/src/net/http/server.go:683 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 61160
	/usr/local/go/src/net/http/server.go:679 +0xba

goroutine 62874 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730d77530, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000cc2680?, 0xc000375000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000cc2680, {0xc000375000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000cc2680, {0xc000375000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc003990000, {0xc000375000?, 0x0?, 0xc0044400f8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc0044400f0, {0xc000375000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc004c74180)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc004c74180, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc001552750, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 62555 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0a440, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000c3eb00?, 0xc000b87000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c3eb00, {0xc000b87000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000c3eb00, {0xc000b87000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc001aa2010, {0xc000b87000?, 0x0?, 0xc0053a5e98?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc0053a5e90, {0xc000b87000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc004476300)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc004476300, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc00064c090, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 46776 [chan send]:
golang.org/x/sync/errgroup.(*Group).Go(0xc002ddc3c0, 0xc0002bfc30)
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:68 +0x32
github.com/thanos-io/objstore.DownloadDir.func1({0xc004e1cf90, 0x28})
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/objstore.go:351 +0x2f5
github.com/thanos-io/objstore/providers/s3.(*Bucket).Iter(0xc000c18900, {0x2c7dba8, 0xc005e1c370}, {0xc0037de060?, 0x22?}, 0xc000718790, {0x0, 0x0, 0x0?})
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/providers/s3/s3.go:413 +0x2a5
github.com/thanos-io/objstore.(*metricBucket).Iter(0xc00013b810, {0x2c7dba8, 0xc005e1c370}, {0xc0037de060, 0x22}, 0x4122b5?, {0x0, 0x0, 0x0})
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/objstore.go:507 +0xf7
github.com/thanos-io/objstore/tracing/opentracing.(*TracingBucket).Iter.TracingBucket.Iter.func1({0x2c7dba8, 0xc005e1c370}, {0x2c964e0, 0x42a32c0})
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/tracing/opentracing/opentracing.go:50 +0x158
github.com/thanos-io/objstore/tracing/opentracing.doWithSpan({0x2c7dba8?, 0xc005e1c370?}, {0x266b83e?, 0x4105c5?}, 0xc0021f3c20, {0x0?, 0xa8?, 0x259d900?})
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/tracing/opentracing/opentracing.go:215 +0xa4
github.com/thanos-io/objstore/tracing/opentracing.TracingBucket.Iter(...)
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/tracing/opentracing/opentracing.go:48
github.com/thanos-io/objstore.DownloadDir({0x2c7dba8, 0xc005e1cfa0}, {0x2c5ec40?, 0xc005414780}, {0x7fd730b58898?, 0xc00057e490}, {0xc00374d120, 0x1a}, {0xc0037de060, 0x22}, ...)
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/objstore.go:350 +0x3e4
github.com/thanos-io/objstore.DownloadDir.func1.1()
	/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20230921130928-63a603e651ed/objstore.go:354 +0x25b
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 46333
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x96

goroutine 63014 [IO wait]:
internal/poll.runtime_pollWait(0x7fd730b0b898, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000c3e800?, 0xc0030aa000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c3e800, {0xc0030aa000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000c3e800, {0xc0030aa000?, 0x4e56e5?, 0x0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc002c98010, {0xc0030aa000?, 0x0?, 0xc005e000f8?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc005e000f0, {0xc0030aa000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:791 +0x14b
bufio.(*Reader).fill(0xc00439a2a0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc00439a2a0, 0x4)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*conn).serve(0xc001552630, {0x2c7db70, 0xc000ca6480})
	/usr/local/go/src/net/http/server.go:2044 +0x75c
created by net/http.(*Server).Serve in goroutine 98
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

GiedriusS Mar 6, 2024
Maintainer

Seems like it is not stuck as it is busy downloading files and decoding data. Maybe bumping --block-files-concurrency would help you to download/upload files quicker and make the compaction process faster?

ggabijaa Mar 6, 2024
Author

I can try doing that, are there any recommendations for setting it ?
Also metrics shows compactor failing to complete its iteration for more than a month, would it be related to block-files-concurrency as well?

ggabijaa Mar 7, 2024
Author

tried bumping block-files-concurrency to 4, which lead to data storage filling up too quickly, hence making the pod crash (we are using 10TB PVC for /data so decreased it to 2 and compact.concurrency to 1

Another thing noticed is that logs are showing
{"caller":"blocks_cleaner.go:44","level":"info","msg":"started cleaning of blocks marked for deletion","ts":"2024-03-07T14:04:01.695731517Z"} {"caller":"blocks_cleaner.go:58","level":"info","msg":"cleaning of blocks marked for deletion done","ts":"2024-03-07T14:04:01.695759158Z"}

Issue is that blocks are never deleted. What could be the reason for that? S3 has around 150 blocks marked for deletion, that seems to stay there indefinitely

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compactor is stuck, S3 storage constantly growing #7176

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Compactor is stuck, S3 storage constantly growing #7176

ggabijaa Mar 4, 2024

Replies: 2 comments · 7 replies

yeya24 Mar 4, 2024 Maintainer

ggabijaa Mar 4, 2024 Author

GiedriusS Mar 5, 2024 Maintainer

GiedriusS Mar 5, 2024 Maintainer

ggabijaa Mar 5, 2024 Author

GiedriusS Mar 6, 2024 Maintainer

ggabijaa Mar 6, 2024 Author

ggabijaa Mar 7, 2024 Author

ggabijaa
Mar 4, 2024

Replies: 2 comments 7 replies

yeya24
Mar 4, 2024
Maintainer

ggabijaa Mar 4, 2024
Author

GiedriusS
Mar 5, 2024
Maintainer

GiedriusS Mar 5, 2024
Maintainer

ggabijaa Mar 5, 2024
Author

GiedriusS Mar 6, 2024
Maintainer

ggabijaa Mar 6, 2024
Author

ggabijaa Mar 7, 2024
Author