Unmarshal Response s3/GetObjectTagging failed, attempt 11/25, error SerializationError: failed to decode REST XML response

[mattduguid]

Describe the bug

We have an odd issue which is blocking us from moving forward, when we attempt to upload a "wpad.dat" (proxy config file) to a cloudflare r2 bucket (https://www.cloudflare.com/en-gb/products/r2/) which uses an "S3-compatible API".

In the terraform debug and in the cloudflare webUI the http post of the "wpad.dat" file to the cloudflare r2 bucket succeeds, but it looks like the http response to the terraform provider contains all or part of the content of the "wpad.dat" file containing a character which breaks the response. It attempts 25 times before we either have to cancel the pipeline or it times out.

If we use a another test file with only the text "helloworld" in it this works perfectly every time.

Expected Behavior

Process to complete for "wpdat.dat" file just like it does with our "helloworld" file.

Current Behavior

This is a portion of the debug showing the break, I cannot share the "wpad.dat" publicly due to sensitive contents but the error does relate to portions of its contents eg: & (no semicolon), etc.,

2023-01-30T21:12:09.4704619Z 2023-01-30T21:12:09.467Z [DEBUG] provider.terraform-provider-aws_v4.20.1_x5: [aws-sdk-go] DEBUG: Unmarshal Response s3/GetObjectTagging failed, attempt 11/25, error SerializationError: failed to decode REST XML response
2023-01-30T21:12:09.4705833Z 	status code: 200, request id: 
2023-01-30T21:12:09.4706541Z caused by: XML syntax error on line 21: invalid character entity & (no semicolon): timestamp=2023-01-30T21:12:09.467Z
2023-01-30T21:13:18.2283423Z ##[error]The operation was canceled.
2023-01-30T21:13:18.2306387Z ##[section]Finishing: Terraform configuration

Reproduction Steps

This is the terraform code,

resource "aws_s3_bucket" "cf" {
  bucket = var.cloudflare_r2_bucket_name
}

resource "aws_s3_object" "cf" {
  bucket = var.cloudflare_r2_bucket_name
  key = var.cloudflare_r2_filename
  source = var.cloudflare_r2_filename
  etag = filemd5(var.cloudflare_r2_filename)
}

Possible Solution

Maybe dont include contents of the file in the http response?

Additional Information/Context

Similar outcomes in this issue with the http response #4577 but different things leading to it

SDK version used

aws_v4.20.1_x5

Environment details (Version of Go (go version)? OS name and version, etc.)

terraform v1.3.7

[RanVaknin]

Hi @mattduguid ,

Maybe dont include contents of the file in the http response?

The response shape is based on the service API and should look like this:

            <?xml version="1.0" encoding="UTF-8"?>
            <Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
               <TagSet>
                 <Tag>
                   <Key>tag1</Key>
                   <Value>val1</Value>
                </Tag>
                <Tag>
                    <Key>tag2</Key>
                    <Value>val2</Value>
                 </Tag>
              </TagSet>
            </Tagging> 

Since you didnt share any code, and are using two separate third party tools, I can only make gusses as to what is actually happening. As for "S3-compatible API" goes, we don't know that for sure.

You can enable logging and inspect the raw http response yourself and see why the xml is malformed by doing the following:

s3Client := s3.New(sess, aws.NewConfig().WithLogLevel(aws.LogDebugWithHTTPBody))

My guess is something like the following is happening:

            <?xml version="1.0" encoding="UTF-8"?>
            <Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
               <TagSet>
                 <Tag>
                   <Key>1&&&&&&</Key>  error occurs here
                   <Value>val1</Value>
                </Tag>
                <Tag>
                    <Key>tag2</Key>
                    <Value>val2</Value>
                 </Tag>
              </TagSet>
            </Tagging> 

Please let us know what you find.
Thanks,
Ran~

[mattduguid]

Hi @RanVaknin we also think that will be the fix eg: "to not include the file contents in the http response body"

We are using the public aws terraform provider so this is not our code, im wondering if i have logged this in the wrong channel and might be better placed here https://github.com/hashicorp/terraform-provider-aws/issues will spin up issue there and close this one, thanks

[mattduguid]

closing and moving to hashicorp/terraform-provider-aws#29207

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[RanVaknin]

@mattduguid ,

I see what you are saying 👍
Since closed github issues are not monitored, if you need assistance please open a new one.

Thanks,
Ran~