Does Thanos Store support Amazon EKS Pod Identity Agent for service accounts?

[jonsbun]

I am trying to switch Thanos Store AWS S3 object storage authentication from the old fashion "IAM roles for service accounts (IRSA)", to the new approach via "EKS Pod Identities".

However, I always getting bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied:

ts=2024-02-21T14:03:45.266560504Z caller=factory.go:53 level=info msg="loading bucket configuration"
ts=2024-02-21T14:03:45.267298663Z caller=inmemory.go:180 level=info msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
ts=2024-02-21T14:03:45.267976041Z caller=options.go:26 level=info protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
ts=2024-02-21T14:03:45.269722033Z caller=store.go:536 level=info msg="starting store node"
ts=2024-02-21T14:03:45.269870318Z caller=store.go:434 level=info msg="initializing bucket store"
ts=2024-02-21T14:03:45.270728954Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
ts=2024-02-21T14:03:45.270767121Z caller=http.go:73 level=info service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
ts=2024-02-21T14:03:45.270989108Z caller=tls_config.go:274 level=info service=http/server component=store msg="Listening on" address=[::]:10902
ts=2024-02-21T14:03:45.271011338Z caller=tls_config.go:277 level=info service=http/server component=store msg="TLS is disabled." http2=false address=[::]:10902
ts=2024-02-21T14:04:15.293069777Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-21T14:04:15.293112197Z caller=http.go:91 level=info service=http/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-21T14:04:15.293590864Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
ts=2024-02-21T14:04:15.293925214Z caller=grpc.go:131 level=info service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
ts=2024-02-21T14:04:15.297745728Z caller=http.go:110 level=info service=http/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-21T14:04:15.298711525Z caller=intrumentation.go:81 level=info msg="changing probe status" status=not-healthy reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-21T14:04:15.299454109Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-21T14:04:15.300840683Z caller=grpc.go:138 level=info service=gRPC/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-21T14:04:15.30126242Z caller=grpc.go:151 level=info service=gRPC/server component=store msg="gracefully stopping internal server"
ts=2024-02-21T14:04:15.301696892Z caller=grpc.go:164 level=info service=gRPC/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-21T14:04:15.304565941Z caller=main.go:161 level=error err="Access Denied\nBaseFetcher: iter bucket\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetchMetadata\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:453\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch.func2\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:526\ngithub.com/golang/groupcache/singleflight.(*Group).Do\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/pkg/mod/github.com/golang/groupcache@v0.0.0-20210331224755-41bb18bfe9da/singleflight/singleflight.go:56\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:524\ngithub.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:584\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).SyncBlocks\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/store/bucket.go:630\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/store/bucket.go:699\nmain.runStore.func5.1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:443\ngithub.com/thanos-io/thanos/pkg/runutil.RetryWithLog\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:97\ngithub.com/thanos-io/thanos/pkg/runutil.Retry\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:87\nmain.runStore.func5\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:442\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\nsync block\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/store/bucket.go:700\nmain.runStore.func5.1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:443\ngithub.com/thanos-io/thanos/pkg/runutil.RetryWithLog\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:97\ngithub.com/thanos-io/thanos/pkg/runutil.Retry\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:87\nmain.runStore.func5\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:442\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\nbucket store initial sync\nmain.runStore.func5\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:448\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\nstore command failed\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/main.go:161\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650"

Object storage config:

  type: S3
  config:
    bucket: bucket-name
    endpoint: s3.eu-west-2.amazonaws.com
    signature_version2: false
    aws_sdk_auth: true

My main question is does Thanos Store support authentication via EKS Pod Identity or only old way via Web Identity works? I can access S3 Thanos storage bucket via Web Identity and IAM role without problems.

However, no luck via EKS Pod Identity. I am sure that EKS Pod Identity solution is working because I can access S3 bucket using Thanos store Statefulset initContainer and amazon/aws-cli image and the same service account thanos-storegateway.

Any ideas?