[tempo-distributed]: IRSA Authentication Broken with Latest tempo-distributed Helm Chart

[asonnleitner]

Description

The latest version of the tempo-distributed Helm chart seems to have a broken IRSA authentication for S3. The issue is observed without explicitly upgrading the tempo.image.tag.

Environment

• Helm Chart: tempo-distributed
• Chart Version: 1.6.3
• Kubernetes Version: v1.27.4-eks-2d98532
• Cloud Provider: AWS

Steps to Reproduce

1. Install or update to the latest tempo-distributed Helm chart.
2. Deploy Tempo.
3. Observe logs for error messages.

Expected Behavior

IRSA authentication should work, and S3 should be accessible.

Actual Behavior

Pods fail with an "Access Denied" error when trying to list objects in our production S3 bucket.

level=error ts=2023-09-05T08:47:45.318564352Z caller=main.go:111 msg="error running Tempo" err="failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on <REDACTED_BUCKET_NAME>: Access Denied"

Additional Information

No issues observed when the Helm chart used grafana/tempo:2.2.1.

Is it possible this broke IRSA?

Related to PR #2743. Is it possible this PR broke IRSA?

Workaround

Explicitly set tempo.image.tag=2.2.1 in the Helm chart values.

[asonnleitner]

This issue seems related to grafana/tempo#2888.