error running Tempo

[LH8PPL]

Describe the bug
we are deploying Tempo on-premise with the distributed helm chart and connect to AWS S3 via PrivateLink.
im having the following error:
level=info ts=2023-06-20T12:38:44.891213365Z caller=main.go:102 msg="Starting Tempo" version="(version=2.1.1, branch=HEAD, revision=4157d7620)"
level=error ts=2023-06-20T12:40:14.89436955Z caller=main.go:105 msg="error running Tempo" err="failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on bucket1: Get "http://bucket1.s3.dualstack.eu-central-1.amazonaws.com/?delimiter=%2F&encoding-type=url&prefix=\": dial tcp xxx.xxx.xxx.xxx:80: i/o timeout"

To Reproduce
Steps to reproduce the behavior:

1. Start Tempo (Helm Version: 1.3.0, App Version: 2.1.1)
2. Perform Operations (Read/Write/Others)

Expected behavior

connect to:
bucket1.s3.eu-central-1.amazonaws.com

not connect to:
bucket1.s3.dualstack.eu-central-1.amazonaws.com

the option to connect to S3 without dualstack and/or disable the IPv6

Environment:

• Infrastructure: OpenShift 4.10
• Deployment tool: helm

Additional Context

storage:
trace:
block:
version: vParquet
backend: s3
s3:
access_key: 'q1w2e3r4t5y6u7i8'
secret_key: 'q1w2e3r4t5y6u7i8o9p0zaxscdvfbgnhmj,k.l/;''
bucket: 'bucket1'
endpoint: "s3.eu-central-1.amazonaws.com"
insecure: true
forcepathstyle: false
region: "eu-central-1"
admin:
backend: s3

[joe-elliott]

That does seem odd. I'm surprised it's not using the endpoint specified in the config. These values are directly pushed into the minio client here:

tempo/tempodb/backend/s3/s3.go

Line 411 in 2edbdfa

return minio.NewCore(cfg.Endpoint, opts)

Perhaps drop the region option? Then maybe it will honor the endpoint provided?

[LH8PPL]

thanks for the replay,

tried that, and other variances but didn't work.

[joe-elliott]

So tracking this into minio some it looks like it rewrites your hostname here:

https://github.com/minio/minio-go/blob/691ae870325d3cd7c3fc426cbf8396342cd58907/api.go#L906

The logic is skipped if it doesn't look like an amazon endpoint here:

https://github.com/minio/minio-go/blob/691ae870325d3cd7c3fc426cbf8396342cd58907/api.go#L909

and there's a second check to see if it looks like an "amazon private link endpoint" here:

https://github.com/minio/minio-go/blob/691ae870325d3cd7c3fc426cbf8396342cd58907/api.go#L922

Here is the regex it uses to determine if a hostname is a "private link endpoint":

https://github.com/minio/minio-go/blob/691ae870325d3cd7c3fc426cbf8396342cd58907/pkg/s3utils/utils.go#L114

I don't know much about this. Should your hostname match this regex?

[LH8PPL]

i don't think so... i could be wrong...

[antonio-tolentino]

Hi guys,

I am facing a similar problem here but I suspect that is related to the aws sdk and the region you are accessing.
I have exactly the same version and configuration of tempo accessing S3 bucket in 2 different regions "eu-central-1" and "eu-central-2", but it works only in "eu-central-1".

Details of configuration:

storage:
  trace:
    block:
      version: vParquet
    backend: s3
    s3:
      bucket: tempo-storage
      endpoint: s3.eu-central-2.amazonaws.com
      region: eu-central-2
      insecure: false
      forcepathstyle: true

Error :

kube logs tempo-compactor-794d7d8b8c-zj98h
level=info ts=2023-06-22T14:48:51.58320735Z caller=main.go:215 msg="initialising OpenTracing tracer"
level=info ts=2023-06-22T14:48:51.600076966Z caller=main.go:102 msg="Starting Tempo" version="(version=2.1.1, branch=HEAD, revision=4157d7620)"
level=error ts=2023-06-22T14:48:52.182694591Z caller=main.go:105 msg="error running Tempo" err="failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on tempo-storage: The eu-central-2 location constraint is incompatible for the region specific endpoint this request was sent to."

[joe-elliott]

Reopening. Accidentally closed this instead of #2565.

[antonio-tolentino]

Hi @joe-elliott,

I have successfully tested the image tag main-c633387.
It seems that the parameter "tolerate_failed_blocks" from queryFrontend is no longer available and I had to remove it from helm chart config template to install this new image.
Is there any defined date for a new tempo release?

Thanks a lot for your help.

[joe-elliott]

I have successfully tested the image tag main-c633387.

Are you saying main-c633387 fixed your issue? Likely an update to the minio s3 client.

Is there any defined date for a new tempo release?

There is not. We tend to release every 3 months or so. On that cadence our next release would be late july/early august.

[antonio-tolentino]

Hi Joe,
Thank you so much for the explanations.
If you need anything from my side that helps you to improve Tempo let me know.

[LH8PPL]

Is there anything we can do about my issue?

[joe-elliott]

Is there anything we can do about my issue?

If the latest Tempo release does not fix the issue then I'd file an issue with the minio go client to get some insight into this behavior.

[LH8PPL]

I'm using tempo image version 2.1.1

[zalegrala]

I believe the change that fixes this is in minio/minio-go#1726, which has been updated in main, but we don't have a release for it yet.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.