S3 Put object request returned has "Etag" in metadata, which does not work with aws-sdk-java putObject API

[minchen07]

In invoking AWS S3 java API against Swift, we failed with the following NPE
java.lang.NullPointerException
at com.amazonaws.util.BinaryUtils.fromHex(BinaryUtils.java:69)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1140)
at com.cloud.utils.S3Utils.putObject(S3Utils.java:178)
After investigation, we realized that Amazon S3 expects that returned metadata should have "ETag" as the metadata key, but the metadata returned from Swift3 has "Etag" as the key, thus causing this NPE. Here is the returned metadata from Swift3:

{Content-Length=0, Etag="5e714348185ffe355a76b754f79176d6", X-Trans-Id=tx1b475494fc1e41049a8be-00530fc274, Content-Type=text/html; charset=UTF-8, Connection=keep-alive}

[asramakanth]

Anyone successfully use Swift3 with Amazon S3 Java SDK?

[bloodeagle40234]

This is caused from a problem of eventlet which is a WSGI library of python.
Eventlet always converts the key (name) of HTTP headers into the formats like as 'Etag' with the rules as follows:

• Converting the first character of the key into a upper case letter
• Converting the first characters leading to '-' into a upper case letters
• Converting all other characters into lower case letters

i.e. currently, we have no way to prevent that eventlet converts "ETag" into "Etag".
Does anyone have good idea for solving it?

[asramakanth]

Opened a issue on Eventlet project : eventlet/eventlet#80

[temoto]

Hello. Please, see this resolution on behalf of Eventlet. With great respect.

---

Hello, @asramakanth . Thank you for your message.

As per HTTP standard, header names are case-insensitive. That means, with Eventlet or any other software, you could receive any of "etag", "ETAG" or "eTAg" and they all MUST work the same way.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

... Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive. The field value MAY be preceded by any amount ...

Consider there may be other users who also ignore HTTP standard and expect current header name case, "Etag". Suppose we fix the issue in your favor, then it would break some other code.

Clean solution is to ignore header name case. Usually people do this by converting all header names to lower case ("etag") and searching for those strings.

[hrchu]

Implementation of AWS S3 violates many HTTP specs (even violate S3 spec itself!!), and many S3 clients follow it. If what you want is to claim that your product is fully compatible with AWS S3, you should violate spec, and hacking current HTTP framework/library, too.

[temoto]

Okay, could anyone please explain software actors involved in this problem, where and when Eventlet comes into play and changes (if?) headers?

[asramakanth]

@temoto, Software actors are AWS S3 Java SDK, Swift3, and Eventlet.
@bloodeagle40234, can you explain where Eventlet comes into play?

[bloodeagle40234]

I'm not sure but I've confirmed the behavior at Eventlet 0.9.17.
As discussed above, it should be solved at AWS S3 Java SDK, I think, because it violates RFC.

[temoto]

I mean what are the steps to reproduce the problem? I am not familiar with Swift, AWS SDK and only briefly know that OpenStack is API compatible open source version of AWS.

[bloodeagle40234]

There are 2 ways (perhaps and more) to fix.

One way is ignoring case when getting metadata as below:
https://github.com/aws/aws-sdk-java/blob/master/src/main/java/com/amazonaws/services/s3/model/ObjectMetadata.java#L582-L584
I would like to recommend this way because I know one of the other libraries called "boto", python library, works as same as this.

The other is making eventlet allow any case headers such as "ETag" and then adding "ETag" instead of "Etag" on swift3.

Does it make you sense?

[temoto]

Did not expect AWS Java SDK to be open source and even on github. It makes much more sense to patch it, then. The only problem then is frozen package versions in distributions.

And in parallel, we could allow eventlet.wsgi.server to accept special argument to disable header case normalization. I will reopen eventlet issue and look into this.

[temoto]

Please, try this eventlet/eventlet@0433029

[minchen07]

@temoto, what are the steps to apply your fix in an existing swift3 setup? Directly patching the file?

[temoto]

Yes, you could patch the file or use virtualenv and pip install https://github.com/eventlet/eventlet/archive/0433029cc6106a8e05319436ee0220c29c37abe4.zip

[temoto]

Also you need to add capitalize_response_headers=False argument to eventlet.wsgi.server() call.

[temoto]

Here's a nice reminder. Please, try fixed eventlet version above and report if the issue is fixed.

[kazum]

Hi @temoto,

I've confirmed that your patch fixes the problem. Thanks a lot!

For swift3 users, please note that we also need a patch for Swift
to disable header case normalization. kazum/swift@b437b13

[temoto]

Thanks for reply. The fix is merged into eventlet master eventlet/eventlet@e4dedf7 and will be included in the next release (v0.15).

[minchen07]

What swift3 official release will have this fix? So that our QA can pick up that release to test.

Thanks
-min

From: Sergey Shepelev <notifications@github.commailto:notifications@github.com>
Reply-To: fujita/swift3 <reply@reply.github.commailto:reply@reply.github.com>
Date: Tuesday, April 22, 2014 7:42 AM
To: fujita/swift3 <swift3@noreply.github.commailto:swift3@noreply.github.com>
Cc: Min Chen <min.chen@citrix.commailto:min.chen@citrix.com>
Subject: Re: [swift3] S3 Put object request returned has "Etag" in metadata, which does not work with aws-sdk-java putObject API (#67)

Thanks for reply. The fix is merged into eventlet master eventlet/eventlet@e4dedf7eventlet/eventlet@e4dedf7 and will be included in next release (v0.15).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/67#issuecomment-41048009.

[Kuruvi]

Has this fix in swift been merged into swift trunk? Or any official release support this.

Thank you
Kuruvi

[bloodeagle40234]

@Kuruvi
It has been fixed in current mainline. (https://github.com/stackforge/swift3)
It needs also swift 2.1.0 or later and eventlet 0.15 or later.

[Kuruvi]

@bloodeagle40234
Thank you. I can see the changes. Sorry I should have checked carefully before asking.

[bloodeagle40234]

@Kuruvi
No worries :)