service/s3: Metadata returns uppercase character

[mattes]

In the console:

When I do:

        result, err := c.HeadObject(&s3.HeadObjectInput{
            Bucket: aws.String("xxx"),
            Key:    aws.String("xxx"),
        })
        fmt.Printf("%#v\n", result)

I get:

...
{
  AcceptRanges: "bytes",
  ContentLength: 0,
  ContentType: "binary/octet-stream",
  ETag: "\"d41d8cd98f00b204e9800998ecf8427e\"",
  LastModified: 2015-11-18 19:00:42 +0000 UTC,
  Metadata: {
    Lp2: "87adaa3c-0bab-4808-a05b-cfe79ba23714"
  }
}
...

Please note Lp2. I would assume it returns lp2. Is this expected?

[mattes]

Using these versions:

github.com/aws/aws-sdk-go ce51895e994693d65ab997ae48032bf13a9290b7
github.com/go-ini/ini 060d7da055ba6ec5ea7a31f116332fe5efa04ce0
github.com/jmespath/go-jmespath 3433f3ea46d9f8019119e7dd41274e112a2359a9

[jasdel]

Thanks for reporting this issue @mattes. It looks like this issue is attributed to the REST header map unmarshaller. We can test it, but I think this method can be updated to extract the original header casing/format instead of always returning the canonical header key form.

Will need to also verify that setting the metadata headers via the SDK don't mutate the casing also.

You could workaround this issue in the interim by manually extracting the metadata headers from the Request.HTTPResponse.Header field.

[jasdel]

@mattes The SDK will automatically case the metadata fields to Go net/http's standard header canonical formating. The issue we're seeing here is the service's implementation details of the metadata fields being transmitted via headers bleeding into how the SDK returns them.

This issue could be addressed and maintain backwards compatibility by introducing an option to ensure the metadata parameters maintain their original casing. Instead of using Go's canonical header key format.

[gwatts]

To add a datapoint to this issue; I tripped over the same thing while retrieving metadata i'd set as lowercase keys and finding that i needed to fetch them using canonical header keys.

I think even adding an accessor method to GetObjectOutput such as GetMetadataValue(k string) string which applies the key normalization would help avoid this kind of confusion down the line.

[MasterCarl]

Please note that this issue still persists more than two years after it was originally opened. I would consider this a bug rather than a Feature Request - different casing is unexpected behavior and requires additional code as a workaround.

[NeoyeElf]

the problem still not solved, using version: v1.16.26 😕

[pwmcintyre]

still an issue with v1.20.2 :(

[kerma]

This is definitely a bug, not a feature-request. Still there in 1.29.21.

What makes it even more confusing is that if you set the metadata for example via PutObjectInput the keys will always be set as all lowercase.

[kntajus]

Tripped up on this myself today. Worth noting that the official docs explicitly state "Amazon S3 stores user-defined metadata keys in lowercase".

[graywolf-at-work]

Adding obligatory #MeToo . It is definitely not expected behaviour...

[bkda]

Hi, guys. Actually, you can use LowerCaseHeaderMaps to do this trick now.
From the aws/config.go:

// Set this to `true` to enable the SDK to unmarshal API response header maps to
// normalized lower case map keys.
//
// For example S3's X-Amz-Meta prefixed header will be unmarshaled to lower case
// Metadata member's map keys. The value of the header in the map is unaffected.
LowerCaseHeaderMaps *bool

But there was a bug in this feature, and the PR #3671 fixed this.

Amazon metadata for S3 is case insensitive on the keys, but the Golang SDK is weird. The unmarshalHeaderMap method always returns the canonical header key form. So enable LowerCaseHeaderMaps config will return the lower case key form. You can check this test private/protocol/rest/rest_test.go to see how it works.

Cheers.

[github-actions]

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

[graywolf-at-work]

Ping.

[jasdel]

With #3033 and #3671 Config.LowerCaseHeaderMaps feature released in the SDK, I think we can close this issue. With this option you can opt-in to lower casing all S3 metadata header key values for consistent usage round tripping request and responses between operation calls.

The v2 SDK, aws-sdk-go-v2, uses lower case header maps by default.

@graywolf-at-work are there additional issues that you're running into that this option does not address?

[graywolf-at-work]

Especially with v2 using the lower case by default, I think there might not be anything left to do. But this was not stated here anywhere (or I was not able to understand it), except for now in your comment. With that comment in place I think this can be closed.

[jasdel]

Thanks for the feedback. I created Pr #4297 documenting this difference.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.