[Bug]: Data Source aws_s3_object - Unable to retrieve User Defined metadata

[andremachado94]

Terraform Core Version

1.5.7

AWS Provider Version

5.17.0,5.18.0,5.18.1

Affected Resource(s)

• aws_s3_object

Expected Behavior

We are using the data source aws_s3_object to retrieve a User Defined metadata key-value pair:

data "aws_s3_object" "package" {
  bucket = var.s3_bucket
  key    = var.s3_key
}

Key name: Hash (in the object metadata is set as x-amz-meta-hash)

Expected behaviour:

Calling data.aws_s3_object.package.metadata.Hash should return the value of the key x-amz-meta-hash

Actual Behavior

Fails with the following exception:

╷
│ Error: Missing map element
│ 
│   on .terraform/modules/utils_sfn.common_lambdas/main.tf line 18, in resource "aws_lambda_function" "lambda":
│   18:   source_code_hash = data.aws_s3_object.package.metadata.Hash
│     ├────────────────
│     │ data.aws_s3_object.package.metadata is map of string with 1 element
│ 
│ This map does not have an element with the key "Hash".
╵

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

data "aws_s3_object" "package" {
  bucket = var.s3_bucket
  key    = var.s3_key
}

resource "aws_lambda_function" "lambda" {
  # ...
  s3_bucket        = var.s3_bucket
  s3_key           = var.s3_key
  source_code_hash = data.aws_s3_object.package.metadata.Hash
  # ...
}

Steps to Reproduce

Using provider >= v5.17.0:

1. Create an s3 object wit the metadata tag x-amz-meta-hash
2. Create a tf module that uses the data source aws_s3_object on that s3 object and with an output block that returns data.aws_s3_object.<whatever_name_you_gave_it>.metadata.Hash

Debug Output

No response

Panic Output

No response

Important Factoids

This was working on v5.16.2. The breaking change occurred on the release v5.17.0. Maybe from #33358 ?

References

https://github.com/hashicorp/terraform-provider-aws/releases/tag/v5.17.0
#33358

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ewbankkit]

@andremachado94 Thanks for raising this issue 👏.
I assume that the corresponding S3 object was created/modified outside of Terraform as the aws_s3_object resource allows only lowercase metadata keys?

[andremachado94]

Hi @ewbankkit !

Yes, it's done outside of terraform (when creating a new release with GH actions). Like this:

aws s3 cp $(ZIP_NAME) s3://$(BUCKET_NAME)/$(ZIP_NAME) --metadata hash="$(shell openssl dgst -binary -sha256 $(ZIP_NAME) | openssl base64)"

[ewbankkit]

The cause is that the AWS SDK for Go v2 HeadObject response normalizes all keys to lowercase.
As this is a breaking change I will see if there's a way to change this.

From the S3 User Guide:

User-defined metadata is a set of key-value pairs. Amazon S3 stores user-defined metadata keys in lowercase.

Relates aws/aws-sdk-go#445.

[andremachado94]

Ok, thanks for the help!

[github-actions]

This functionality has been released in v5.19.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.