java.lang.NoSuchMethodError: 'void com.amazonaws.util.IOUtils.closeQuietly

[umesh2611]

Describe the bug

Describe the bug
Hi,

I am following the link : https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html to send a kafka message to the MSK cluster ( provisioned / IAM )

When i am sending the message i get the following error :

2023-01-10T14:05:04.035+05:30

Copy
2023-01-10 08:35:04,033 - ERROR - kafka-producer-network-thread | producer-1 - org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-producer-network-thread | producer-1':

2023-01-10T14:05:04.035+05:30 java.lang.NoSuchMethodError: 'void com.amazonaws.util.IOUtils.closeQuietly(java.io.Closeable, org.apache.commons.logging.Log)'
at software.amazon.msk.auth.iam.internals.UserAgentUtils.getLibraryVersion(UserAgentUtils.java:73)
at software.amazon.msk.auth.iam.internals.UserAgentUtils.getLibraryVersion(UserAgentUtils.java:73)
at software.amazon.msk.auth.iam.internals.UserAgentUtils.(UserAgentUtils.java:43)
at software.amazon.msk.auth.iam.internals.UserAgentUtils.(UserAgentUtils.java:43)

complete log :

https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Fecs$252FTmf628NormalizationUPALIWAL/log-events/ecs$252Ftmf628NormalizationContainer2$252F798843bb106f4e5a8fe41e423673e0a5

What i analyzed, the IOUtils pacakged in com.amazonaws.aws-java-sdk-bundle-1.12.262.jar contains a method with the following signature :

public static void com.amazonaws.util.IOUtils.closeQuietly(java.io.Closeable,com.amazonaws.thirdparty.apache.logging.Log)

but "aws-msk-iam-auth" is expecting a method with the following signature :
'void com.amazonaws.util.IOUtils.closeQuietly(java.io.Closeable, org.apache.commons.logging.Log)'

on analysis, I found that there are 2 IOUtils class from core and bundle jar.
the signature on the core jar is crrect but the bundle contains public static void com.amazonaws.util.IOUtils.closeQuietly(java.io.Closeable,com.amazonaws.thirdparty.apache.logging.Log)

Can you pease help me on moving ahead ?

/Umesh Paliwal

Expected Behavior

NA

Current Behavior

NA

Reproduction Steps

NA

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.17.295

JDK version used

11

Operating System and version

linux

[debora-ito]

That's because the aws-java-sdk-bundle includes all AWS services, their dependent jars and third-party libraries in one standalone jar. For this to work, dependencies had to be shaded, that's why you see the Log class in a different package name.

Instead of using the bundled jar, we recommend to add as a dependency only the services you are going to use in you project. For more info see - https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-project-maven.html#configuring-maven

[umesh2611]

Thanks for your reply.

Just to give more insight, we are using AWS SDK Java 2 .

But when we add AWS Hadoop library with the version 3.3.4, the SDK Bundle of aws sdk java 1 gets added.

(dependency tree )

[info] | +-org.apache.hadoop:hadoop-aws:3.3.4
[info] | | +-com.amazonaws:aws-java-sdk-bundle:1.12.262
[info] | | +-org.wildfly.openssl:wildfly-openssl:1.0.7.Final

Can you help us on this

[debora-ito]

Since you can't change the libs in your dependency list, you need to make sure that the version of the Java SDK dependencies match the version expected by the version of aws-msk-iam-auth you are using, and that the not-bundled IOUtils class is in your classpath at runtime.

[sangeethsasidharan]

@umesh2611 have you able to make any resolution here , I am facing the same issue when trying to connect to spark application to msk cluster

[safwank]

I managed to resolve this by

• excluding com.amazonaws:aws-java-sdk-bundle from org.apache.hadoop:hadoop-aws
• explicitly adding com.amazonaws:aws-java-sdk-s3 which is all that's required by hadoop-aws apparently, at least in my case

[debora-ito]

@umesh2611 I'm going to move this to the Discussions tab, since it's a usage question and not really a bug.