S3: False Request and 'Object Not Found' Error When Bucket Name Matches Object Key Prefix and usePathStyle is False

[SamuelTJackson]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

We have a bucket named 'test' containing multiple objects under the path 'test/test/dir1/'. While listing the objects within this directory poses no issues, attempting to retrieve them results in an incorrect request and an Object Not Found error. This occurs when the usePathStyle parameter is set to false.

Expected Behavior

Retrieving objects from an S3 bucket where the bucket name matches the prefix of the object keys should succeed without errors.

Current Behavior

Attempting to retrieve an object results in the following error message: Failed to get object from bucket: operation error S3: GetObject, HTTPS response error with StatusCode: 404.

Reproduction Steps

Bucket name = test
Full path to object = test/test/dir1/object1
usePathStyle = false

GetObject(context.Background, &s3.GetObjectInput{
		Bucket: aws.String("test"),
		Key:    aws.STring("/test/dir1/object1"),
	})

Possible Solution

No response

Additional Information/Context

service/s3/internal/customizations/update_endpoint.go:238

	u.Host = bucket + "." + u.Host
	removeBucketFromPath(u, bucket)

	if strings.HasPrefix(u.Path, "/"+bucket) {
		// modify url path
		u.Path = strings.Replace(u.Path, "/"+bucket, "", 1)

		// modify url raw path
		u.RawPath = strings.Replace(u.RawPath, "/"+httpbinding.EscapePath(bucket, true), "", 1)
	}

removeBucketFromPath is removing the prefix of our object key which results in a request like 'https://test.[aws domain]/dir1/object1' instead of 'https://test.[aws domain]/test/dir1/object1'

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2/service/s3 v1.48.1

Compiler and Version used

go version go1.21.4 linux/amd64

Operating System and version

6.8.4-arch1-1

[RanVaknin]

Hi @SamuelTJackson ,

I'm not able to reproduce this behavior. usePathStyle should only affect buckets that are created as path style buckets (usually self hosted buckets, custom domain buckets or old legacy S3 buckets).

Here is my example using a standard (non legacy) S3 bucket:

uploading sample object to my bucket using the CLI:

$ echo "This is a test file" > testfile.txt
$ aws s3 cp testfile.txt s3://testbucket-3650/test/test/dir1/object1

upload: ./testfile.txt to s3://testbucket-3650/test/test/dir1/object1

Get the object from my bucket without using usePathStyle:

package main

import (
	"context"
	"fmt"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"io"
	"strings"
)

func main() {
	cfg, err := config.LoadDefaultConfig(context.TODO())
	if err != nil {
		panic(err)
	}

	s3Client := s3.NewFromConfig(cfg)

	result, err := s3Client.GetObject(context.Background(), &s3.GetObjectInput{
		Bucket: aws.String("testbucket-3650"),
		Key:    aws.String("test/test/dir1/object1"),
	})
	if err != nil {
		panic(err)
	}

	buf := new(strings.Builder)
	_, err = io.Copy(buf, result.Body)
	if err != nil {
		panic(err)
	}
	fmt.Println(buf.String())

}
// prints: This is a test file

Is your bucket actually named test? I assumed not if you are using the actual S3 server since that name would be taken. Are you using an S3 clone like Digital Ocean Spaces / minIO?
If your bucket is not named test, does the name contain any unique characters that might need encoding (dot, slashes hyphens)?

Any additional info will help.
Thanks
Ran~

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.