SDK client overrides region settings for custom s3api endpoint

[ornokrean]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
When using custom api endpoint, with a set of none-AWS regions, SDK client overrides this regions, blocking us from making the request.

Version of AWS SDK for Go?
v1.38.3

Version of Go (go version)?
go1. 16

To Reproduce (observed behavior)

aws-sdk-go/service/s3/bucket_location.go

Line 97 in bba6aba

if r.ParamsFilled() && aws.StringValue(r.Config.Region) != "us-east-1" {

func getS3Client(creds *credentials) (*s3.S3, error) {
        sess, err := getAwsClient(creds)
        if err != nil {
                return nil, fmt.Errorf("failed to create an AWS session: %s", err)
        }
        return s3.New(sess), nil
}
func getAwsClient(creds *credentials) (*session.Session, error) {
        return session.NewSession(&aws.Config{
                Region:           aws.String("my-custom-region"),
                Credentials:      awsCredentials.NewStaticCredentials(creds.awsAccess, creds.awsSecret, ""),
                Endpoint:         aws.String("https://my-endpoint"),
                HTTPClient:       creds.httpClient,
                S3ForcePathStyle: aws.Bool(true),
        })
}
func do() error {
        svcS3, err := getS3Client(creds)
        if err != nil {
                return err
        }
        bucketName := "my-bucket-name"
        // ensure bucket is created
        _, err = svcS3.CreateBucket(&s3.CreateBucketInput{
                Bucket: &bucketName,
        })
        if err != nil {
                return fmt.Errorf("failed to create %q bucket: %v", bucketName, err)
        }
}

error message:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidLocationConstraint
  </Code>
  <Message>The specified location-constraint is not valid
  </Message>
  <LocationConstraint>my-custom-region
  </LocationConstraint>
  <RequestId>A8PASP7TRCNWJ7DJ
  </RequestId>
  <HostId>***masked***
  </HostId>
</Error>

Expected behavior
We want the client not to change our region configuration, meaning he will pass it on without modifying it.

Additional context
As a workaround, we used this code:
if false && r.ParamsFilled() && aws.StringValue(r.Config.Region) != "us-east-1" {
which keeps the client in the same region as it configured.

we opened an issue #3906 and got this answer:

Try setting the CreateBucketConfiguration member of CreateBucketInput.

        _, err = svcS3.CreateBucket(&s3.CreateBucketInput{
                Bucket: &bucketName,
                CreateBucketConfiguration: &CreateBucketConfiguration{
                          Location: aws.String("my-custom-region"), 
                },

        })

and then the issue was closed, but we did try using this solution, even before we got the answer, and it did'nt work, gave us this error:
Error: failed to create "--bucketname--" bucket: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'

[vudh1]

Hi, is this bug still persisting with the latest version of SDK?

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[kayrus]

Was it fixed?

[kayrus]

Our workaround for the issue:

        req, _ := svc.CreateBucketRequest(&s3.CreateBucketInput{
                Bucket:                    &bucketName,
                CreateBucketConfiguration: bucketConf,
        })
        // removes "populateLocationConstraint" handler from aws/aws-sdk-go/service/s3/bucket_location.go
        req.Handlers.Validate.RemoveByName("__anonymous")

[vudh1]

Can you confirm if this is still persisting with the latest version of SDK?

[kayrus]

@vudh1 the bug still persists

[jasdel]

Thanks for the update @kayrus. I'm attempting to reproduce the issue described where setting a custom CreateBucketConfiguration isn't being used, but not able to.

In addition removal of the populateLocationConstraint handler shouldn't be needed, because it checks if CreateBucketConfiguration is populated or not. Only setting the default value if nil. Removal of __anonymous is also pretty dangerous as it could apply to any anonymous handler, not the populateLocationConstraint one specifically.

Error: failed to create "--bucketname--" bucket: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'

With regard to this error message you're seeing when explicitly setting CreateBucketConfiguration could you provide more detail, and enable signing logging for this? From the error message it sounds like the request is being sent to us-east-1, but is the LocationConstraint set to us-west-2? The AWS SDK for Go requires all request made to the correct region. If a bucket is to be created in us-west-2 region, the request request must also be signed and made to that region as well.

The following is the code that I'm using to try to reproduce the issue, but I'm seeing the configured LocationConstraint specified being used in the request body.

package main

import (
	"fmt"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	sess, err := session.NewSession(&aws.Config{
		Region:           aws.String("my-custom-region"),
		Endpoint:         aws.String("https://my-endpoint"),
		S3ForcePathStyle: aws.Bool(true),
		LogLevel:         aws.LogLevel(aws.LogDebugWithHTTPBody | aws.LogDebugWithSigning),
		MaxRetries:       aws.Int(0),
	})
	if err != nil {
		panic(err)
	}

	client := s3.New(sess)

	result, err := client.CreateBucket(&s3.CreateBucketInput{
		Bucket: aws.String("my-bucket"),
		CreateBucketConfiguration: &s3.CreateBucketConfiguration{
			LocationConstraint: aws.String("us-west-2"),
		},
	})
	fmt.Println(result, err)
}

Example output.

2022/06/27 10:06:29 DEBUG: Request Signature:
---[ CANONICAL STRING  ]-----------------------------
PUT
/my-bucket

content-length:153
host:my-endpoint
x-amz-content-sha256:a4a3d263179bfcbaf88755b14ea9db3048e4179c03bbc94ccf0f13c8c2d3df6a
x-amz-date:20220627T170629Z
x-amz-security-token:<token>

content-length;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
a4a3d263179bfcbaf88755b14ea9db3048e4179c03bbc94ccf0f13c8c2d3df6a
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20220627T170629Z
20220627/my-custom-region/s3/aws4_request
3adf1ff7ae8933a1df796585a400d062fdf0ca67f6d50791884152a8e23745bc
-----------------------------------------------------
2022/06/27 10:06:29 DEBUG: Request s3/CreateBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /my-bucket HTTP/1.1
Host: my-endpoint
User-Agent: aws-sdk-go/1.44.42 (go1.18.3; darwin; amd64)
Content-Length: 153
Authorization: AWS4-HMAC-SHA256 Credential=<AccessKey>/20220627/my-custom-region/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=4342803553d2278750a45c63775614cd6901aa36990ca48f6f0f916a85917fc1
X-Amz-Content-Sha256: a4a3d263179bfcbaf88755b14ea9db3048e4179c03bbc94ccf0f13c8c2d3df6a
X-Amz-Date: 20220627T170629Z
X-Amz-Security-Token: <token>
Accept-Encoding: gzip

<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>us-west-2</LocationConstraint></CreateBucketConfiguration>

[kayrus]

In the original message you can see:

When using custom api endpoint, with a set of none-AWS regions

it also visible in the code:

aws-sdk-go/service/s3/bucket_location.go

Lines 97 to 106 in 32d9e06

	if r.ParamsFilled() && aws.StringValue(r.Config.Region) != "us-east-1" {
	in := r.Params.(*CreateBucketInput)
	if in.CreateBucketConfiguration == nil {
	r.Params = awsutil.CopyOf(r.Params)
	in = r.Params.(*CreateBucketInput)
	in.CreateBucketConfiguration = &CreateBucketConfiguration{
	LocationConstraint: r.Config.Region,
	}
	}
	}

If the region doesn't correspond to us-east-1 and CreateBucketConfiguration is not set (we don't need it to be set), sdk forces this CreateBucketConfiguration. If we set an empty (not nil) CreateBucketConfiguration, then SDK will send empty constraints, which are not expected on the receiving side and it complains.

[skmcgrail]

Given that there isn't a reasonably or infallible way to determine what is or isn't a custom API endpoint there isn't really a specific action we can do here without creating a breaking change. If you are using the SDK to talk to a third-party S3 API implementation then you can could remove this request handler on the request like so:

req.Handlers.Unmarshal.Remove(s3.NormalizeBucketLocationHandler)

[kayrus]

@skmcgrail you're mixing up two things: normalize handler (which actually has a handler name) and populateLocationConstraint handler. The last is added as an anonymous handler and removing it from the request is not reliable, because there could be other anonymous handlers. In order to keep the backward compatibility you can adjust AWS SDK and add the populateLocationConstraint handler to a request using a request.NamedHandler struct.