Difficulties while upgrading the AWS JAVA SDK from 1.x.x to 2.x.x

[pjonnlagadda]

Describe the bug

As i am trying to migrate aws-sdk-java from V1 to V2,
to make start i have replaced the artifacts in pom.xml accordingly for S3, STS, SQS and IAM etc.. . Later when i am trying to build, getting few of the compile errors saying that, classes like AmazonSQS, SQSActions are missing and many other compile errors for few other classes. Seems the package "com.amazonaws.auth.policy" is missing.
Ex:-
constructor S3BucketResource in class com.amazonaws.auth.policy.resources.S3BucketResource cannot be applied to given types;
required: java.lang.String
found: java.lang.String,java.lang.String

Expected Behavior

Is there any smooth way to replace V2 from V1?

Current Behavior

getting few of the compile errors saying that, classes like AmazonSQS, SQSActions are missing

Reproduction Steps

Replace the artifacts in pom.xml accordingly for S3, STS, SQS and IAM etc.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

AWS SDK for Java 2.0

JDK version used

java 1.8

Operating System and version

Windows 11 Enterprise

[yasminetalby]

Hello @pjonnlagadda ,

Thank you very much for your submission.
Could you please clarify the following:
Is the only step you have took to migrate modifying the pom.xml file with v2 artifact?

There exists multiple changes between the two versions. You would need to update your code appropriately while migrating from v1 to v2.

The Developer Guide describes the main changes to be aware of when converting an application from using the AWS SDK for Java version 1.x to version 2.x.

The 1.11 to 2.0 Changelog is also a good resource to see changes between the two versions.

If your application depends on libraries that have not yet been released in version 2.x, see Using both SDKs side-by-side to learn how to configure your pom.xml to use both 1.x and 2.x.

Best,

Yasmine

[pjonnlagadda]

Hi @yasminetalby ,
Thank you for the response.
regarding your question "Is the only step you have took to migrate modifying the pom.xml file with v2 artifact?"
Yes, The very first step i started with replacing the pom.xml entries only.
Not modified any code changes before changing form V1 to V2

[yasminetalby]

Hello @pjonnlagadda ,

Thank you very much for the information.

You will need to update your code based on the changes between the two version of the SDK.
You can find most documentation on the resources I have shared above!

Another good resource to look at for an overview of the use of different services in AWS Java SDK v2 is in aws-docs/aws-doc-sdk-examples/javav2/example_code.

This does not seems like a bug but more a documentation question so I will change the label on the issue.
Let me know if you have any question or if there is any information that is not provided by the documentation that I can help with!

Sincerely,

Yasmine

[pjonnlagadda]

Hi @yasminetalby,
Thank you very much for the valuable info.

So the final conclusion you mean,
Step 1:- Start using side – by side
Step 2:- Start making the code changes for migration of the services which we are using from V1 to V2(Ex:- to S3, SQS, IAM, STS etc..)
Step 3:- In future Once all the changes are migrated from V1 to V2(long process), we can completely remove the V1 entries from Pom.xml if everything is supported in V2

As I we have policy related classed in V1 com.amazonaws.auth.policy are found in the V2 test folder. But not all. Any idea where I can found them all or any examples like how can we start using them in the V2?
or the usage of Statement, Principal, Condition, Policy etc..

Thanks in Advance.

[yasminetalby]

Hello @pjonnlagadda ,

Yes exactly!

This is the V1 package com.amazonaws.auth.policy

Here is the V2 package software.amazon.awssdk.core.auth.policy

If you need example for a specific use case you can also submit a documentation request to the aws-docs team to ask for new documentation of a specific use case here under feature-request.

I hope these resources help!

Is there any way we can improve the documentation provided on the migration guide to facilitate migration from AWS Java SDK V1 to V2 on your end? We would love to have any feedback on your experience and what you would find helpful/valuable to support you through this process.

Sincerely,

Yasmine

[pjonnlagadda]

Hi @yasminetalby,
Once again thank you for the information.
Still i see that few of the enums are missing or not able to identify in the V2 when compare to V1. Any clue on where could i found them?
com.amazonaws.auth.policy.actions.S3Actions;
com.amazonaws.auth.policy.actions.SQSActions;
com.amazonaws.auth.policy.actions.SecurityTokenServiceActions;

Thanks in Afvance

[yasminetalby]

Hello @pjonnlagadda,

Of course. Could you please provide the code sample you are trying to migrate to v2?

Best,

Yasmine

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[pjonnlagadda]

Hi Yasminetalby,
In the process of migration, i have started using them side by side and trying to create an S3 client with the DefaultCredentialsProvider, where i am not able to find Regions.getCurrentRegion() in the V2 support to find out the current region as i need to have few validations in this?
This is required for the application is running in, when running in EC2.
In the V1 i use to find in using the Region region= Regions.getCurrentRegion();

S3Client s3 = S3Client.builder()
.region(region)
.credentialsProvider(DefaultCredentialsProvider.create()).build();
Any help on getting this current region in V2?

Thanks in advance.

[yasminetalby]

Hello @pjonnlagadda ,

Regions.getCurrentRegion() is not supported in v2 see: Region class name changes in the migration guide.
To retrieve the current region of the running EC2 instance you can use EC2MetadataUtils.getEC2InstanceRegion.

Let me know if you have any further questions!

Best,

Yasmine

[pjonnlagadda]

Hi Yasmine,
Thank you for the response and i am clear with the S3Client part.
Another problem in the area of SqsClient where as i am trying to migrate from V1 to V2 and in the part of sqsClient.deleteMessageBatch().
Here i am trying to create an objects for DeleteMessageBatchRequestEntry and DeleteMessageBatchRequest to make an batch delete from the queue.
Using the below code snippet i am trying to create an objects and not able to understand how to pass the BuilderImpl to the constructor there i am getting compile error and not allowing further. Can you please help on clearing the same??
software.amazon.awssdk.services.sqs.model.DeleteMessageBatchRequestEntry entry = new software.amazon.awssdk.services.sqs.model.DeleteMessageBatchRequestEntry**()**.builder() .id(String.format("%d", counter)) .receiptHandle(message.receiptHandle()) .build();

software.amazon.awssdk.services.sqs.model.DeleteMessageBatchRequest deleteRequest = new software.amazon.awssdk.services.sqs.model.DeleteMessageBatchRequest**()**.builder() .queueUrl(sqsQueueURL).entries(deleteEntries).build();

[yasminetalby]

Hello @pjonnlagadda ,
Of course! Glad to hear the S3Client part is all clear.

Here is the documentation covering DeleteMessageBatchRequestRequest.Builder and DeleteMessageBatchEntry.Builder

Here is also all the AWS Java SDK v2 documentation for usage of SQS. It doesn't go over DeleteMessageBatchRequestEntry.builder however the example on Send a batch of message to queue should give you enough guidance on how to use the DeleteMessageBatchRequest.builder and DeleteMessageBatchRequest.builder.

Documentation on full example of the usage of SQS in AWS Java SDK v2 can also be found in the aws-docs repository here. This documentation can be helpful to further your knowledge on how to use the sdk v2 for SQS.

Let me know if you have any further questions!

Best,

Yasmine

[pjonnlagadda]

Hi Yasmine,
Thanks you for the given clarification on S3Client.
Another area where as i am trying to migrate the KinesisClient from V1 to V2 and here in the V1 we are set setting up the EndpointConfiguration to the KinesisClientBuilder where as am not able to make it in the V2 and i dont fund any information in the code examples or in the catalog. Any idea on make it this happen?
In the V1 i am using like
AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard(); clientBuilder.setEndpointConfiguration(new EndpointConfiguration(<endpoint>, <region>));

In V2 migration to the below class i am not able to find any thing related to end point configuration except the "endpointOverride" where we cannot set the region and for .statndared also bydefault support?
KinesisClientBuilder kinesisClientBuilder = KinesisClient.builder();

[yasminetalby]

Hello @pjonnlagadda ,

Of course!

This is documented in the1.11 to 2.0 Changelog provided earlier.

You can find this information under : AWS Client Configuration: Custom Regions, Credentials and Endpoints

The example provided is for DynamoDBClient but this documentation is relevant for general endpoint configuration.

See the v1.11.x endpoint configuration syntax:

AmazonDynamoDBClientBuilder.standard() .withEndpointConfiguration(new EndpointConfiguration(endpoint, signingRegion)) .build();

And the v2.0 endpoint configuration syntax:

DynamoDbClient.builder() .region(signingRegion) .endpointOverride(endpoint) .build()

Looking at the documentation for the KinesisClientBuilder in v2 you can find both region and endpointOverride method used in the syntax above.

Best,

Yasmine

[pjonnlagadda]

Hi @yasminetalby,
Thank you for response. and using the ..endpointOverride(endpoint) .build(), i have configured the KinesisClient and make a call to finding out the stream status where as i am ending up with some error. here is the below code snippet i am using on V1 vs V2.
i am using the local NettyJaxrsServer to make an api call
In the V1 my code is 100% working as expected

`BasicAWSCredentials awsCreds = new BasicAWSCredentials(kiStreamDef.getAccessKeyId(), secertKey);
AWSStaticCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(awsCreds);
AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard();
clientBuilder.setEndpointConfiguration(new EndpointConfiguration(endpoint, region));
clientBuilder.setCredentials(credentialsProvider);
KinesisClient kinesisClient = clientBuilder.build();

DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();
describeStreamRequest.setStreamName(streamName);

DescribeStreamResponse response = kinesisClient.describeStream(describeStreamRequest);`

What i see in the V1 log
Sending Request: POST http://localhost:59628/stream / Headers: (amz-sdk-invocation-id: 6acc15ee-c89c-132a-267e-279842498c99, Content-Length: 41, Content-Type: application/x-amz-json-1.1, User-Agent: aws-sdk-java/1.12.308 Windows_10/10.0 OpenJDK_64-Bit_Server_VM/25.322-b06 java/1.8.0_322 vendor/Amazon.com_Inc. cfg/retry-mode/legacy, X-Amz-Target: Kinesis_20131202.DescribeStream, )

In V2
Following is the code I have replaced with V2 from V1 where as i am getting error "49 - RESTEASY002010: Failed to execute
javax.ws.rs.NotSupportedException: RESTEASY003065: Cannot consume content type"

`_

AwsBasicCredentials awsCreds = AwsBasicCredentials.create(accessKey, secertKey);
StaticCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(awsCreds);
KinesisClient kinesisClient = KinesisClient.builder()
.region(Region.of(region))
.endpointOverride(URI.create(endpoint))
.credentialsProvider(credentialsProvider)
.build();

DescribeStreamRequest describeStreamRequest = DescribeStreamRequest.builder().streamName(streamName).build();
DescribeStreamResponse response = kinesisClient.describeStream(describeStreamRequest);

_`

What i see in the V2 log
2022-10-13 17:55:11.239 [main ] DEBUG software.amazon.awssdk.request : 85 - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=http, host=localhost, port=61417, encodedPath=/stream/, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent, X-Amz-Target], queryParameters=[]) 2022-10-13 17:55:11.463 [main ] DEBUG s.a.a.c.internal.io.SdkLengthAwareInputStream : 85 - Specified InputStream length of 39 has been reached. Returning EOF. 2022-10-13 17:55:11.590 [main ] DEBUG software.amazon.awssdk.auth.signer.Aws4Signer : 85 - AWS4 String to sign: XXXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 2022-10-13 17:55:14.074 [pool-3-thread-4] DEBUG org.jboss.resteasy.resteasy_jaxrs.i18n : 49 - RESTEASY002315: PathInfo: /stream/ 2022-10-13 17:55:14.084 [pool-3-thread-4] ERROR org.jboss.resteasy.resteasy_jaxrs.i18n : 49 - RESTEASY002010: Failed to execute javax.ws.rs.NotSupportedException: RESTEASY003065: Cannot consume content type at org.jboss.resteasy.core.registry.SegmentNode.match(SegmentNode.java:460) at org.jboss.resteasy.core.registry.SegmentNode.match(SegmentNode.java:141)

Not understanding what went wrong, is headers part something missing in the V2 where as compare to V1?, Can you please help on make this further?.

[yasminetalby]

Hello @pjonnlagadda ,

I hope that you are doing well. I was wondering if you would be ok for me to transfer this GitHub Issue to General Discussion.
I think this could be a great resource for other users who are attempting to migrate from v1 to v2.

Let me know if that is ok with you!

Best,

Yasmine

[pjonnlagadda]

I am OK to make it general. But i want to continue this thread for my further migration activities also.

[yasminetalby]

Hello @pjonnlagadda ,

Thank you very much for your response.
Of course! This thread and previous discussion will not be lost, It will just be transferred under General Discussion and will stay open for any further questions or guidance request you might have throughout the migration process.

Sincerely,

Yasmine

[pjonnlagadda]

Hi @yasminetalby,
Good Day. continuing on the discussion on my migration part, i have seen there is a VersionInfo.java in sdk-core-2.17.286.jar. Ideally jar should contain only class file. Any reason for Why this java file is included into the base level of the jar file?

When i unzip the jar file it has been observed that below structure

META-INF
software
mime.types
VersionInfo.java

Thanks in advance.

Regards,
Praveen

[yasminetalby]

Hello Praveen,

I can confirm that is the structure for the sdk-core-2.17.286.jar file. Is this causing any limitation or issue on your part?

Best,

Yasmine

[pjonnlagadda]

Hi @yasminetalby
We have restriction that no external jar should contain ".java" file and thinking on how to handle this.
And is there any documentation in the AWS SDK JAVA V2 on shading of external jars, i mean all the dependency (third-party) libraries derived through AWS SDK should start the package name with "software.amazon.awssdk"
for example if i take the utils-2.17.286.jar, the package structure is software.amazon.awssdk.utils

[yasminetalby]

Hello @pjonnlagadda ,

Unfortunately we do not have documentation in the AWS SDK Java V2 on shading external jars as this is a general question. We provide guidance documentation regarding SDK specific information.
There are a couple blog post mentioning shading for both v1 and v2:

1. https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-2-17-removes-its-external-dependency-on-jackson/
2. https://aws.amazon.com/blogs/developer/java-sdk-bundle/
   You can also check Maven shade plugin documentation.

Sincerely,

Yasmine

[pjonnlagadda]

Hi @yasminetalby,
I would like to know where i can see the exact release notes on ASW-SDK_JAVA-V2 along with what improvements has been on each version officially
Currently as i am referring https://github.com/aws/aws-sdk-java-v2
Regards,
Praveen

[yasminetalby]

Hello @pjonnlagadda ,

Thank you very much for your question.
This information can be found in the CHANGELOG.md
Best,

Yasmine

[pjonnlagadda]

Hi @yasminetalby,
Good Day.
Now i am migrating the S3 services from V1 to V2 and here i am facing the problem with v1 AmazonS3URI.java, where i am not able to find the replacement in V2. We are using this in the V1 for finding out the bucket name and key parsed from the s3url.
i found this issue related to #272 still opened
Is there any other alternative to get the bucket and key name from the S3url in java V2?

Thanks in advance.
Regards,
Praveen

[yasminetalby]

Hello @pjonnlagadda ,

Thank you very much for reaching out!

Unfortunately, this feature is not yet available in v2. #272 is the right place to look for update regarding the issue release. Per the feature-request prompt, the current work around is to use the v1 SDK on the side until this feature is released.

Best regards,

Yasmine

[pjonnlagadda]

Hi @yasminetalby,
Thank you for the information.
Another issue i am facing while upgrading V2
Regarding IamClient where i am trying to list all the accessKeys through IamClient.listAccessKeys(listAccessKeysRequest))
Before that as i need to create listAccessKeysRequest object using ListAccessKeysRequest.builder().......build();_, i don't see the method to provide the 'awsCredentialsProvider' where as in the V1 we are providing like (Ex:- new ListAccessKeysRequest()
.withRequestCredentialsProvider(awsCredentialsProvider);).
Can you please advice is it vanished?? if yes, how does it work?

Thanks in advance.
Regards,
Praveen

[debora-ito]

For setting a custom credential provider at request level, check the overrideConfiguration method -
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/iam/model/ListAccessKeysRequest.Builder.html

ListAccessKeysRequest request = ListAccessKeysRequest.builder()
                    .overrideConfiguration(c->c.credentialsProvider(myCredentialProvider))
                    .build();

The AwsRequestOverrideConfiguration class offers several other configurations, I recommend you take a look at.

[pjonnlagadda]

Thank you @debora-ito.
Another conflict where as compared from V1 ClientConfiguration
I want to set to Client configuration options such as proxy settings, user agent string, max retry attempts, etc to my s3Client or SQSClient in V2.
In V1 we know we are using ClientConfiguration setter methods to set the required values. But in V2 it is observed that, the client ClientConfiguration is split into separate configuration classes.

Can you have any code examples to set the Client configuration options in S3Client, especially i want to set the The maximum number of open connections (ClientConfiguration.withMaxConnections(xxx))?

i am following this for my synchronized threads and referring the Client Configuration section in https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-whats-different.html

Regards,
Praveen

[debora-ito]

Yes, we do have examples in our Developer Guide, in the Configuring HTTP clients section.

You can check the example for the ApacheHttpClient in the Configuring the Apache-based HTTP client page. Examples for the other HTTP clients can also be found in the respective pages in the guide.

The list of available settings will depend on the HTTP Client you are using, you can check the HTTP client Javadoc for more information - here's the link to the ApacheHttpClient Javadoc:
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/apache/ApacheHttpClient.Builder.html

As a reminder, you can find this info in the 1.11 to 2.0 Changelog readme, just search for "ClientConfiguration" (or any term you are currently using in v1) and you'll probably find the equivalent in v2 -
https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md

[pjonnlagadda]

Hi @debora-ito,
As part of the V2 migration, my EC2MetadataUtils.getInstanceInfo().getAccountId() is not working with V2 and getting "Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to contact EC2 metadata service.".

But V1 for the same code it is working as expected. Is that the calling of "EC2MetadataUtils.getInstanceInfo().getAccountId()" results the same awsAccountId irrespective of the internal implementation between the V1 & V2??
Because it working with V1s com.amazonaws.util.EC2MetadataUtils.getInstanceInfo().getAccountId()
but not with V2s
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getInstanceInfo().getAccountId()

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to contact EC2 metadata service. at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:403) at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:414) at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:414) at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:355) at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:351) at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getInstanceInfo(EC2MetadataUtils.java:301)

Thanks in advance
Regards,
Praveen

[debora-ito]

I can't really say why you are seeing the error in V2 without more details of your use case and a deeper investigation of the logs, but I want to point out that V2 EC2MetadataUtils is an internal API and we don't recommend your code depend on it, because it is subject to change - this is mentioned in the javadoc.

Instead, we recommend the use of the recently released EC2InstanceMetadata client - more info in the blog post:
https://aws.amazon.com/blogs/developer/introducing-a-new-client-in-the-aws-sdk-for-java-2-x-for-retrieving-ec2-instance-metadata/

Please check it out and let us know of any questions.

[pjonnlagadda]

HI @debora-ito,
Thank you for the replay.
I have another problem while creating the SqsClient in v2, where it is recommended to use ApacheHttpClient.Builder().maxConnections(100) to set the client configurations.
My question is if i dont set SqsClient.builder().httpClientBuilder(httpClientBuilder).build(); will this SqsClient.builder() works with any default client configuration automatically for 'maxConnections', 'connectionTimeout' 'requestTimeout' like as we have in the V1's ClientConfiguration??

Right now i am trying with below code, But i dont see any default values for maxConnections=50 for the sqs client configuration??
ApacheHttpClient.Builder httpClientBuilder = ApacheHttpClient.builder(); SqsClient.builder().httpClientBuilder(httpClientBuilder).build();

Thanks in Advance
Regards,
Praveen

[debora-ito]

My question is if i dont set SqsClient.builder().httpClientBuilder(httpClientBuilder).build(); will this SqsClient.builder() works with any default client configuration automatically for 'maxConnections', 'connectionTimeout' 'requestTimeout' like as we have in the V1's ClientConfiguration

Yes. The default values can be found here: https://github.com/aws/aws-sdk-java-v2/blob/02c678c17704901e03f89c78d82ed5642d76aec1/http-client-spi/src/main/java/software/amazon/awssdk/http/SdkHttpConfigurationOption.java

Note that different HTTP Clients (Apache, Netty, CRT, etc) have different settings exposed, so not all configuration options listed in SdkHttpConfigurationOption are applicable to all HTTP Clients.

[pjonnlagadda]

Hi @debora-ito,
Thank you for the quick response.
So you mean, if i simply say
SqsClient client = SqsClient.builder().build();
OR
SqsClient client = SqsClient.builder().httpClientBuilder(ApacheHttpClient.builder()).build();
with empty ApacheHttpClient.builder() works with the default values for maxConnections=50 etc.. from SdkHttpConfigurationOption ??

Because i see some thing else about ApacheHttpClient.builder().buildWithDefaults() and confusing for me and though of using this to set the defaults??
Regards,
Praveen

[pjonnlagadda]

Hi @debora-ito, @yasminetalby , @scheerer , @leepa
Any update on my above comment please?

[pjonnlagadda]

Hi @debora-ito / @yasminetalby
Good day.
Question #1: I am having problem with creating the software.amazon.awssdk.core.auth.policy.Statement in aws java sdk v2. I am getting the compile error in the line ".withResources(new S3ObjectResource(partition, bucketName, mountPolicyPath),"

I don't see any S3ObjectResource from the package software.amazon.awssdk.core.auth.policy to create the resource with S3ObjectResource. Can you help me on completing the creation of Statement withResources provided parameters?

Ex:-
Statement s1 = new Statement(Effect.Allow)
.withId(id)
.withActions(new Action("s3:PutObject"))
.withResources(new S3ObjectResource(partitionName, bucketName, keyPattern1),
new S3ObjectResource(partitionName, bucketName, keyPattern2),
new S3ObjectResource(partitionName, bucketName, keyPattern3),
new S3ObjectResource(partitionName, bucketName, keyPattern4),
new S3ObjectResource(partitionName, bucketName, keyPattern5);

Question #2: What are replacements of V1 Enums like SQSActions, IAMAction, S3Actions etc in V2??
Thanks in advance.
Regards,
Praveen