JedisCluster fails to execute command hincrbyfloat inside ClusterCommandExecutor#executeCommand(CommandObject<T> commandObject).

[sodrulamin]

I am getting exception while executing hincrbyfloat command using JedisCluster. This is not a regular case, like If I try to execute the command 1000 times it happens once.

redis.clients.jedis.exceptions.JedisClusterOperationException: Cluster retry deadline exceeded.
	at redis.clients.jedis.executors.ClusterCommandExecutor.executeCommand(ClusterCommandExecutor.java:88) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.UnifiedJedis.executeCommand(UnifiedJedis.java:167) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.UnifiedJedis.hincrByFloat(UnifiedJedis.java:1318) ~[jedis-4.3.1.jar:?]

I have set max try count to 1 while passing this value in JedisCluster constructor.
The strange thing is that, even though it throws the exception, the command was executed successfully in redis. As I can see from redis directly.

If I set max try count to 3. Then I see, it prints one Connection Timeout exception then try to execute the command again. So, what happens here, the first and second both attempts were successful, as a result the increment was happening twice which was very much unexpected for my case.

Full exception for both of these cases is below

2023-02-04 12:40:51,951 DEBUG ClusterCommandExecutor:64 - Failed connecting to Redis: Connection{DefaultJedisSocketFactory{IP:PORT}}
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
	at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:208) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:46) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Protocol.process(Protocol.java:126) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Protocol.read(Protocol.java:192) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:316) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Connection.getOne(Connection.java:298) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Connection.executeCommand(Connection.java:123) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.executors.ClusterCommandExecutor.execute(ClusterCommandExecutor.java:103) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.executors.ClusterCommandExecutor.executeCommand(ClusterCommandExecutor.java:57) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.UnifiedJedis.executeCommand(UnifiedJedis.java:167) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.UnifiedJedis.hincrByFloat(UnifiedJedis.java:1318) ~[jedis-4.3.1.jar:?]
	
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_352]
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_352]
	at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_352]
	at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_352]
	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:464) ~[?:1.8.0_352]
	at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) ~[?:1.8.0_352]
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1346) ~[?:1.8.0_352]
	at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73) ~[?:1.8.0_352]
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:962) ~[?:1.8.0_352]
	at java.io.InputStream.read(InputStream.java:101) ~[?:1.8.0_352]
	at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:202) ~[jedis-4.3.1.jar:?]
	... 16 more

I am keeping one JedisCluster instance for long time and re-using for multiple times to avoid creating new connection for each command.

Can you please suggest what should I do to avoid this situation ? I am thinking to close the instance of JedisCluster after certain times. Will it solve the problem ?

[sazzad16]

If "the command was executed successfully in redis" then it is most likely that the reply from Redis didn't reach to Jedis in time. This could be network issue, in which case we can't help you.

What is your (socket) timeout value? If it small, it'd be a strong recommendation to increase it.

[sodrulamin]

This could be network issue

Yes. I think so. But network issue is only in one way. Response is not coming. But command is getting sent.

What is your (socket) timeout value?

Currently the socket timeout value is 10 seconds. But I have tried multiple variations like 20 seconds, 30 seconds 60 seconds and so on. Still this happens for all of these cases.