Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with native compilation using S3 sdk #2086

Closed
cmoine opened this issue Oct 13, 2020 · 2 comments
Closed

Issues with native compilation using S3 sdk #2086

cmoine opened this issue Oct 13, 2020 · 2 comments
Labels
guidance Question that needs advice or information. needs-triage This issue or PR still needs to be triaged.

Comments

@cmoine
Copy link

cmoine commented Oct 13, 2020

This ticket is a subset of #1852 (Make SDK easily compileable to GraalVM native image)

Describe the issue

The AWS S3 Client has an awkward behavior with some characters in the URL (character ' ', '@' for instance) with native compilation using GraalVM

Steps to Reproduce

Use the following snippet:

public class Main {

    public static void main(String[] args) {
        S3Client client = S3Client.create();
        // This one works natively
        System.out.println("object size: " + client.headObject(HeadObjectRequest.builder().bucket("mybucket").key("destDir/testProductDto.json").build()).contentLength());
        // This one fails (/!\ only with native compilation)
        System.out.println("object size: " + client.headObject(HeadObjectRequest.builder().bucket("mybucket").key("destDir/commodule-9@2.bin").build()).contentLength());
    }
}

Current Behavior

The second invocation raises this exception:

Exception in thread "main" software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 403, Request ID: null, Extended Request ID: jyQ8wJCeXzMYD68kQq9Wb9Mml1jvThAaVGWvR7S7CRdstWtflbY6tj1pDfOg8rzUOAJ3PK1K8U8
=)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94)
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$4(BaseClientHandler.java:214)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:101)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:4080)
        at Main.main(Main.java:160)

Our investigation so far regarding this issue:

  • We have compared Aws4Signer behavior with JVM and native compilation, both seem to behave the same
  • We have compared both URL (there was a warning during native execution (NoSuchMethodException was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See org.apache.httpcomponents:httpclient:4.5.7 breaks fetching S3 objects with consecutive slashes in the key aws-sdk-java#1919 for more information)", so we added "org/apache/http/client/config/RequestConfig$Builder" to the reflect config, and the warn disappeared, but no différence)

Your Environment

  • AWS Java SDK version used:
    v2.13.47
  • JDK version used:
    OpenJDK 64-Bit Server VM GraalVM 20.1.1-dev (build 25.252-b09-jvmci-20.1-b02, mixed mode)
  • Operating System and version:
    Linux 64bits
@cmoine cmoine added guidance Question that needs advice or information. needs-triage This issue or PR still needs to be triaged. labels Oct 13, 2020
@cmoine
Copy link
Author

cmoine commented Oct 13, 2020

After 3 days of investigation, I finally found the problem: S3 SDK use execution interceptors. The execution interceptor is loaded dynamically, and thus requires to add this in the resources-config.json file:

{"pattern":"\\Qsoftware/amazon/awssdk/services/s3/execution.interceptors\\E"}

After that modification, the classes listed in that file required to be accessible reflectively in "reflect-config.json" file:

{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.AddContentMd5HeaderInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
},
{
  "name":"software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor",
  "methods":[{"name":"<init>","parameterTypes":[] }]
}

If this can helps other people, I'll be glad about that!

As a conclusion, I have been punished for not using the native agent as described here:
https://www.graalvm.org/reference-manual/native-image/Configuration/#assisted-configuration-of-native-image-builds

@cmoine cmoine closed this as completed Oct 13, 2020
aws-sdk-java-automation pushed a commit that referenced this issue Jul 7, 2022
Removing CW integration test that fails and is no longer relevant
@jmahonin
Copy link

jmahonin commented Feb 2, 2023

Just adding a comment here for anyone who has also fallen down this rabbit-hole. The execution.interceptors functionality was removed in AWS SDK 2.17.46
https://github.com/aws/aws-sdk-java-v2/blob/f941b3f7ae1a35e289c10be940757525feed8290/.changes/2.17.46.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
guidance Question that needs advice or information. needs-triage This issue or PR still needs to be triaged.
Projects
None yet
Development

No branches or pull requests

2 participants