Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thanos-store: Store Pod is not able to load certain blocks from object storage #320

Open
epetry30 opened this issue Mar 28, 2024 · 1 comment

Comments

@epetry30
Copy link

Thanos, Prometheus and Golang version used:
quay.io/thanos/thanos:v0.31.0

Object Storage Provider: own Object Storage with S3 endpoint

What happened:
At some point after a restart of a Thanos-store pod several blocks are no able to be loaded from our object storage , it is not a huge number but it still results in data loss
What you expected to happen:
The Thanos-store pod should be able to load all of the blocks.

Full logs to relevant components:
level=warn ts=2024-03-27T13:56:45.548274508Z caller=bucket.go:652 msg="loading block failed" elapsed=3.549132ms id=01GNH44X841X3PA2YF67AE4CVB err="create index header reader: write index header: new index reader: get object attributes of 01GNH44X841X3PA2YF67AE4CVB/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548551713Z caller=bucket.go:652 msg="loading block failed" elapsed=3.520884ms id=01HG5MJHV6536D5NKFMN6Z3HD7 err="create index header reader: write index header: new index reader: get object attributes of 01HG5MJHV6536D5NKFMN6Z3HD7/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548600296Z caller=bucket.go:652 msg="loading block failed" elapsed=2.923082ms id=01HG695QEDVAGJ9MCPYEFM1B1E err="create index header reader: write index header: new index reader: get object attributes of 01HG695QEDVAGJ9MCPYEFM1B1E/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548964003Z caller=bucket.go:652 msg="loading block failed" elapsed=3.079476ms id=01GTJ8VHZJ70EXBHDF7R9A1MZ2 err="create index header reader: write index header: new index reader: get object attributes of 01GTJ8VHZJ70EXBHDF7R9A1MZ2/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.549556296Z caller=bucket.go:652 msg="loading block failed" elapsed=3.047629ms id=01HFV43YJX0SY19SSJNJ4Q05WE err="create index header reader: write index header: new index reader: get object attributes of 01HFV43YJX0SY19SSJNJ4Q05WE/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.55066445Z caller=bucket.go:652 msg="loading block failed" elapsed=3.244963ms id=01H7J41MEM1EZGEAY9Y695CM1T err="create index header reader: write index header: new index reader: get object attributes of 01H7J41MEM1EZGEAY9Y695CM1T/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.566415912Z caller=bucket.go:652 msg="loading block failed" elapsed=21.854809ms id=01H5V67MFXA5PZG4VCF1SW5REM err="create index header reader: write index header: new index reader: get object attributes of 01H5V67MFXA5PZG4VCF1SW5REM/index: The specified key does not exist."

@epetry30
Copy link
Author

epetry30 commented Apr 2, 2024

Missing blocks are in /var/thanos/store/meta-syncer/ directory but not in /var/thanos/store so they have no index reference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant