Replies: 1 comment
-
Too bad I haven't gotten any feedback yet. Can anyone answer my questions? Or are further details required? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We are running a hazelcast (5.2.1) cluster in AWS ECS. This cluster consists of 3 tasks, each running a hazelcast instance. According to the documentation (https://docs.hazelcast.com/hazelcast/latest/deploy/deploying-on-aws), we have assigned the following IAM Task Roles:
In addition, we have enabled the AWS config for hazelcast accordingly. When the cluster is started up, it is successfully formed via the AWS discovery strategy and is ready for use after a few seconds.
After a short time, however, the following logs can be found. This log is output again every x seconds and are therefore very annoying.
Couldn't connect to the service, [3] retrying in 3 seconds…
This error message is triggered by the describeNetworkInterfaces(...) method (AwsEc2Api.java:170).
Stacktrace:
com.hazelcast.spi.exception.RestClientException: Failure in executing REST call com.hazelcast.spi.utils.RestClient.call(RestClient.java:167) com.hazelcast.spi.utils.RestClient.lambda$callWithRetries$0(RestClient.java:134) com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:65) com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:51) com.hazelcast.spi.utils.RestClient.callWithRetries(RestClient.java:134) com.hazelcast.spi.utils.RestClient.get(RestClient.java:126) com.hazelcast.aws.AwsEc2Api.callAwsService(AwsEc2Api.java:247) com.hazelcast.aws.AwsEc2Api.describeNetworkInterfaces(AwsEc2Api.java:177) com.hazelcast.aws.AwsEcsClient.getAddresses(AwsEcsClient.java:58) com.hazelcast.aws.AwsDiscoveryStrategy.discoverNodes(AwsDiscoveryStrategy.java:183) com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:72) com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:72) com.hazelcast.internal.cluster.impl.TcpIpJoiner.searchForOtherClusters(TcpIpJoiner.java:477) com.hazelcast.internal.cluster.impl.SplitBrainHandler.searchForOtherClusters(SplitBrainHandler.java:75) com.hazelcast.internal.cluster.impl.SplitBrainHandler.run(SplitBrainHandler.java:42) com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:77) com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:829) com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: java.net.SocketTimeoutException: connect timed out java.base/java.net.PlainSocketImpl.socketConnect(Native Method) java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) java.base/java.net.Socket.connect(Socket.java:609) java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305) java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:507) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:602) java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266) java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193) java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592) java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) com.hazelcast.spi.utils.RestClient.checkResponseCode(RestClient.java:218) com.hazelcast.spi.utils.RestClient.call(RestClient.java:164) \t... 21 common frames omitted
The reason for this is that the tasks do not have rights to access the EC2 Describe Network Interfaces API.
This right was specially not assigned, because the tasks have no public IP! Contrary to the documentation where this right is specified as optional, this does not seem to be optional (see error message + stack trace).
Is there a way to disable the call to the EC2 Describe Network Interfaces API or is the documentation incorrect at this point and the ec2:DescribeNetworkInterfaces right is required?
Furthermore, describeNetworkInterfaces(() is also called when DiscoveryMode = Member. Is this correct? In another place AwsEc2Client.java:59-61 the call getEcsAddresses(...) and thus describeNetworkInterfaces(...) is only called if DiscoveryMode = Client.
Improvement/bugfix?
If no public ip is used, the call to the EC2 Describe Network Interfaces API should not be made. This avoids unnecessary and annoying log messages.
In addition, the call could perhaps be disabled via a setting, this would prevent the unnecessary requests that are executed every x seconds.
Additional context
Hazelcast version: 5.2.1
Beta Was this translation helpful? Give feedback.
All reactions