Unable to Override Region for Timestream ListDatabase Call

[bbernays]

Describe the bug

When you specify a region that is different than the region in the config via the request options the call always fails. The internal endpoint discovery mechanism always uses the region from the sdk client rather than the region specified in the request options

Expected Behavior

The call should not fail with an error saying the credentials are invalid. The endpoint discovery should use the region that is passed in to determine the endpoint rather than the region set in the sdk client

Current Behavior

The call returns an error:

2023/06/27 12:35:25 failed to list tables, operation error Timestream Write: ListDatabases, https response error StatusCode: 403, RequestID: ZQQKMDGYKKG4R47F5S2B22IYZM, api error InvalidSignatureException: Credential should be scoped to a valid region. , region was: us-east-2

Reproduction Steps

func main() {
	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-east-1"))
	if err != nil {
		log.Fatalf("unable to load SDK config, %v", err)
	}
	// Using the Config value, create the TimestreamWrite client
	svc := timestreamwrite.NewFromConfig(cfg)
	for _, region := range []string{"us-east-1", "us-east-2"} {
		_, err := svc.ListDatabases(context.TODO(), &timestreamwrite.ListDatabasesInput{}, func(o *timestreamwrite.Options) {
			o.Region = region
		})
		if err != nil {
			log.Fatalf("failed to list tables, %v, region was: %s", err, region)
		}
		log.Printf("success, region was: %s", region)
	}
}

With the output being:

% go run main.go
2023/06/27 12:35:25 success, region was: us-east-1
2023/06/27 12:35:25 failed to list tables, operation error Timestream Write: ListDatabases, https response error StatusCode: 403, RequestID: ZQQKMDGYKKG4R47F5S2B22IYZM, api error InvalidSignatureException: Credential should be scoped to a valid region. , region was: us-east-2
exit status 1

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

	github.com/aws/aws-sdk-go-v2/config v1.18.27
	github.com/aws/aws-sdk-go-v2/service/timestreamwrite v1.17.2

Compiler and Version used

go version go1.20.5 darwin/arm64

Operating System and version

MacOS

[lucix-aws]

I can confirm this is an issue. You have noted correctly that services using endpoint discovery will always make the "describe endpoints" call with the initially configured region, ignoring any other regions sideloaded through per-invocation options.

Note this occurs whether the client region came from shared config as your example demonstrates, OR through a region manually set via Options, e.g.

svc := timestreamwrite.NewFromConfig(cfg, func(o *timestreamwrite.Options) {
    o.Region = "us-east-2"
})

The actual failure you're seeing appears to come from signing, since we're effectively (incorrectly) always operating in a timestream "cell" in us-east-1 once discovery has taken place, but the SDK correctly signs the request with the per-operation region which has now differed.

There are two changes we need to make to correct this behavior:

• initial endpoint discovery needs to respect the per-operation region value in the first place
• the client needs to scope each discovered cell to and only to its original region, and re-discover accordingly as needed (as region changes).

As of this writing, the endpoint discovery feature is used only by timestream (both timestreamwrite and timestreamquery).

While it's not an ideal workaround, you can circumvent this issue by creating a separate client for each region in which you're operating.

[RanVaknin]

@lucix-aws @bbernays ,

I see you merged a fix? if this is fixed can we close this issue?

Thanks,
Ran~

[bbernays]

@RanVaknin - The fix that we put in place is purely a workaround that we intend to remove once the SDK is fixed

[lucix-aws]

Yes-- please leave this issue open for now. There is still a bug in the SDK itself.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.