org.apache.httpcomponents:httpclient:4.5.7 breaks fetching S3 objects with consecutive slashes in the key

[akhaku]

org.apache.httpcomponents:httpclient:4.5.7 made a change[1] to url rewriting (used when executing a request) where it normalizes the provided urls by removing empty path segments (effectively collapsing consecutive slashes into a single slash).

S3 supports objects with consecutive slashes, but because of this behaviour change in httpclient, you can no longer fetch those objects with the AWS SDK, since the url httpclient makes a request to is different. On a side note, the manifestation of this issue is a signature mismatch, since the signature calculated by the SDK uses double slashes but when httpclient hits the path without double slashes, the calculated signature is different.

It looks like there's some debate on it now and they're actually working on making normalization optional[2], so once that's released we can update the RequestConfig[3] to disable normalization and revert to the previous behaviour[3].

Example stack trace showing the code path to slash collapsing:

rewriteURI:158, URIUtils (org.apache.http.client.utils)
rewriteURIForRoute:231, URIUtils (org.apache.http.client.utils)
rewriteRequestURI:95, ProtocolExec (org.apache.http.impl.execchain)
execute:132, ProtocolExec (org.apache.http.impl.execchain)
doExecute:185, InternalHttpClient (org.apache.http.impl.client)
execute:83, CloseableHttpClient (org.apache.http.impl.client)
execute:56, CloseableHttpClient (org.apache.http.impl.client)
execute:72, SdkHttpClient (com.amazonaws.http.apache.client.impl)
executeOneRequest:1285, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
executeHelper:1101, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
doExecute:758, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
executeWithTimer:732, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
execute:714, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
access$500:674, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
execute:656, AmazonHttpClient$RequestExecutionBuilderImpl (com.amazonaws.http)
execute:520, AmazonHttpClient (com.amazonaws.http)
invoke:4705, AmazonS3Client (com.amazonaws.services.s3)
invoke:4652, AmazonS3Client (com.amazonaws.services.s3)
getObject:1457, AmazonS3Client (com.amazonaws.services.s3)
getObject:1316, AmazonS3Client (com.amazonaws.services.s3)

1: https://issues.apache.org/jira/browse/HTTPCLIENT-1960, apache/httpcomponents-client@8c04c6a
2: https://issues.apache.org/jira/browse/HTTPCLIENT-1968, apache/httpcomponents-client@4093a30
3: https://github.com/aws/aws-sdk-java/blob/1.11.502/aws-java-sdk-core/src/main/java/com/amazonaws/http/AmazonHttpClient.java#L1282

[debora-ito]

@akhaku Thank you for the detailed report!

Using org.apache.httpcomponents:httpclient:4.5.7 indeed removed one of the two consecutive slashes in my tests.

Based on the resolution of [2] https://issues.apache.org/jira/browse/HTTPCLIENT-1968, we'll keep an eye on the next httpclient release to see what is the new behaviour of the fix. I'll keep this issue open in the meantime.

[valters]

I am seeing this issue ("AmazonS3Exception: The request signature we calculated does not match the signature you provided.") on S3 key that contains "+" character (that seems the only thing out of ordinary) - like "2019/a+b/item". Could it be related? The regression is caused by upgrading to Spring Boot 2.1.3 which brings in (version manages) httpclient:jar:4.5.7 (from 4.5.6).

[debora-ito]

@valters yes, it looks like keys containing special characters like + were also affected by 4.5.7. We are investigating the whole impact.

[CatalinaMoisuc]

The problem persists with org.apache.httpcomponents:httpclient#4.5.8

[akhaku]

Indeed - #1966 will fix it, just waiting for them to take a look.

[swetavkamal]

This error also comes up while we we call getBootstrapBrokers on AWS MSK(managed streaming kafka).
This error comes up with httpclient 5.5.7 and above while works fine for 5.5.6 or lower.

[debora-ito]

Status update:

We opened a ticket with Apache asking to change the normalization default value to false to reduce the impact on the SDK, but they won't make the change.

Then, while working on the solution, @varunnvs92 found a bug in httpclient 4.5.8 in the URIBuilder#isPathEmpty method.

This bug was fixed and merged, but now the solution on the SDK side is blocked by the next release of httpclient.

We are actively working to fix this. Thank you for your patience.

[ursNiemi]

What is the last version of AWS SDK that does not have this issue? I have a setup, where upgrading Apache httpclient is not possible, but downgrading AWS SDK is.

[debora-ito]

@ursNiemi What version of httpclient are you using? If you're using httpclient-4.5.9 or newer, it will work with SDK version 1.11.596 or newer.

And just to be clear, the issue was not in SDK but in the changed behavior in Apache httpclient-4.5.7.