Replicated mode with AWSElastiCache not detecting DNS change when Route53 record updated #4437

rajuraog · 2022-07-21T13:30:21Z

Redisson version: 3.17.3
Config Type: replicatedServersConfig
dnsMonitoringInterval: 5000

Application not detecting when there is a DNS change. We registered AWSElastiCache Primary, Reader endpoints under Route53 and application connects to Route53 records. Everyday, we recreate new ElastiCache and swap the Route53 records with new ones and remove the previous elasticache. Expectation is that, application detects when route53 gets updated and reconnects successfully. But we don't see any logs such as "DNSMonitor - Detected DNS change" which comes in case of SingleServerConfig.

But later, application is reconnecting to the new cache when old cache gets deleted but it's not consistent either. We see below logs in case of successful reconnection.
master master-xxx.aws.env.local/10.xx.xx.254:6379 has changed to master-xxx.aws.env.local/10.xx.xx.144:6379
24 connections initialized for master-xxx.aws.env.local/10.xx.xx.144:6379
1 connections initialized for replica-xxx.aws.env.local/10.xx.xx.179:6379
slave: replica-xxx.aws.env.local/10.xx.xx.179:6379 added
slave: [addr=rediss://replica-xxx.aws.env.local:6379] is down
24 connections initialized for replica-xxx.aws.env.local/10.xx.xx.179:6379

When it fails, it won't reconnect but will throw below errors over and over by pointing to the deleted cache until application restart.
RedisTimeoutException - Command execution timeout for command: (INFO REPLICATION), params: [], Redis client: [addr=rediss://replica-xxx.aws.env.local:6379]
MasterSlaveEntry - master master-xxx.aws.env.local/10.xx.xx.168:6379 used as slave
ReplicatedConnectionManager - slave: [addr=rediss://replica-xxx.aws.env.local:6379] is down
LoadBalancerManager - Unable to unfreeze entry: [freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:51:queue:0, freeConnectionsAmount=0, freeConnectionsCounter=value:64:queue:0, freezeReason=SYSTEM, client=[addr=rediss://master-xxx.aws.env.local:6379], nodeType=MASTER, firstFail=0]
RedisConnectionException - Unable to connect to Redis server: master-xxx.aws.env.local/10.xx.xx.168:6379

Happy to provide additional details. Please advice on this.

mrniko · 2022-07-21T15:58:30Z

You need to check your DNS. If it works properly.

rajuraog · 2022-07-21T16:52:48Z

@mrniko In case of SingleServerConfig, DNS detection works correctly and we see the log "DNSMonitor - Detected DNS change" immediately after Route53 update and will be reconnected.

It's not detecting only in case of replicatedServersConfig config.

rajuraog · 2022-08-02T11:20:26Z

@mrniko Could you please comment on above issue we are facing with replicatedServersConfig

mrniko · 2022-08-02T11:59:21Z

Can you share logs in trace logging level for org.redisson package?

rajuraog · 2022-08-08T16:40:05Z

2022-08-06T02:26:39.915+01:00 2022-08-06 01:26:39,915 | DEBUG | redisson-netty-5-20 | o.r.c.ClientConnectionsEntry - new connection created: RedisConnection@1174406187 [redisClient=[addr=rediss://qa4-xxx.aws.qa.local:6379], channel=[id: 0x85f0b508, L:/10.xx.x.97:40170 - R:qa4-xxx.aws.qa.local/10.xx.x.128:6379], currentCommand=CommandData [promise=java.util.concurrent.CompletableFuture@7e118b2f[Completed normally], command=(PING), params=[], codec=null], usage=0]

2022-08-06T02:26:39.915+01:00 2022-08-06 01:26:39,915 | INFO | redisson-netty-5-20 | o.r.c.ReplicatedConnectionManager - slave: qa4-xxx.aws.qa.local/10.xx.x.128:6379 added

2022-08-06T02:26:39.916+01:00 2022-08-06 01:26:39,916 | INFO | redisson-netty-5-20 | o.r.c.ReplicatedConnectionManager - slave: [addr=rediss://qa4-xxx.aws.qa.local:6379] is down

2022-08-06 01:26:39,916 | INFO | redisson-netty-5-20 | o.r.c.pool.SlaveConnectionPool - 24 connections initialized for qa4-xxx.aws.qa.local/10.xx.x.128:6379
2022-08-06 01:26:39,916 | INFO | redisson-netty-5-20 | o.r.c.pool.SlaveConnectionPool - 24 connections initialized for qa4-xxx.aws.qa.local/10.xx.x.128:6379

2022-08-06T02:26:39.916+01:00 2022-08-06 01:26:39,916 | TRACE | redisson-netty-5-2 | o.r.c.handler.CommandDecoder - reply: +PONG

2022-08-06T02:26:39.916+01:00 , channel: [id: 0xb3b15220, L:/10.xx.x.97:40172 - R:qa4-xxx.aws.qa.local/10.xx.x.128:6379], command: CommandData [promise=java.util.concurrent.CompletableFuture@1450225[Not completed, 1 dependents], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]

2022-08-06T02:26:39.916+01:00 2022-08-06 01:26:39,916 | TRACE | redisson-netty-5-1 | o.r.c.handler.CommandDecoder - reply: +PONG

2022-08-06T02:26:39.916+01:00 , channel: [id: 0x85f0b508, L:/10.xx.x.97:40170 - R:qa4-xxx.aws.qa.local/10.xx.x.128:6379], command: CommandData [promise=java.util.concurrent.CompletableFuture@372318dd[Not completed, 1 dependents], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]

2022-08-06T02:26:39.936+01:00 2022-08-06 01:26:39,935 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@428339238 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x90379e53, L:/10.xx.x.97:45840 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.936+01:00 2022-08-06 01:26:39,936 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@846894071 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xe3c29324, L:/10.xx.x.97:45716 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.936+01:00 2022-08-06 01:26:39,936 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@2092865978 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xadea4a46, L:/10.xx.x.97:45498 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.937+01:00 2022-08-06 01:26:39,937 | DEBUG | redisson-netty-5-22 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.937+01:00 2022-08-06 01:26:39,937 | DEBUG | redisson-netty-5-23 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.938+01:00 2022-08-06 01:26:39,938 | DEBUG | redisson-netty-5-24 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.938+01:00 2022-08-06 01:26:39,937 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@269637534 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x0f6f4a64, L:/10.xx.x.97:48974 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.938+01:00 2022-08-06 01:26:39,938 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@239267496 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6096897a, L:/10.xx.x.97:45842 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.938+01:00 2022-08-06 01:26:39,938 | DEBUG | redisson-netty-5-25 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.939+01:00 2022-08-06 01:26:39,939 | DEBUG | redisson-netty-5-26 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.939+01:00 2022-08-06 01:26:39,938 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1811188570 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xf9479402, L:/10.xx.x.97:45574 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.939+01:00 2022-08-06 01:26:39,939 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@2009264056 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x0d2a0729, L:/10.xx.x.97:45668 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.940+01:00 2022-08-06 01:26:39,939 | DEBUG | redisson-netty-5-27 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.940+01:00 2022-08-06 01:26:39,940 | DEBUG | redisson-netty-5-28 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.940+01:00 2022-08-06 01:26:39,940 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@533933130 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xb9510516, L:/10.xx.x.97:45670 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.940+01:00 2022-08-06 01:26:39,940 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@774502167 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xd28dd5d6, L:/10.xx.x.97:45442 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.941+01:00 2022-08-06 01:26:39,941 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1958654679 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xcc7e26a6, L:/10.xx.x.97:45780 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.941+01:00 2022-08-06 01:26:39,941 | DEBUG | redisson-netty-5-29 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.941+01:00 2022-08-06 01:26:39,941 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@740536893 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x22367fc2, L:/10.xx.x.97:45576 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.941+01:00 2022-08-06 01:26:39,941 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1549865868 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x4320ce9f, L:/10.xx.x.97:45558 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.941+01:00 2022-08-06 01:26:39,941 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1286570900 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xf3032d90, L:/10.xx.x.97:45658 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.942+01:00 2022-08-06 01:26:39,942 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1761581434 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6afef39f, L:/10.xx.x.97:45554 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.942+01:00 2022-08-06 01:26:39,942 | DEBUG | redisson-netty-5-30 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.942+01:00 2022-08-06 01:26:39,942 | DEBUG | redisson-netty-5-5 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.942+01:00 2022-08-06 01:26:39,942 | DEBUG | redisson-netty-5-3 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.943+01:00 2022-08-06 01:26:39,942 | DEBUG | redisson-netty-5-4 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.943+01:00 2022-08-06 01:26:39,943 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@665882685 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x1e2c027d, L:/10.xx.x.97:45654 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.943+01:00 2022-08-06 01:26:39,943 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@320748709 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6563f27f, L:/10.xx.x.97:45552 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.943+01:00 2022-08-06 01:26:39,943 | DEBUG | redisson-netty-5-6 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.943+01:00 2022-08-06 01:26:39,943 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1579777745 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x25e43c76, L:/10.xx.x.97:45714 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:39.944+01:00 2022-08-06 01:26:39,944 | DEBUG | redisson-netty-5-7 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.944+01:00 2022-08-06 01:26:39,944 | DEBUG | redisson-netty-5-31 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.945+01:00 2022-08-06 01:26:39,945 | DEBUG | redisson-netty-5-2 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:39.945+01:00 2022-08-06 01:26:39,945 | DEBUG | redisson-netty-5-1 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.036+01:00 2022-08-06 01:26:40,036 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@759484136 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x4074ec3b, L:/10.xx.x.97:45626 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.036+01:00 2022-08-06 01:26:40,036 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1271159722 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x7865eb0d, L:/10.xx.x.97:45500 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.037+01:00 2022-08-06 01:26:40,037 | DEBUG | redisson-netty-5-32 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.037+01:00 2022-08-06 01:26:40,037 | DEBUG | redisson-netty-5-8 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.051+01:00 2022-08-06 01:26:40,051 | TRACE | redisson-netty-2-3 | o.r.c.handler.CommandEncoder - channel: [id: 0xd32b1fbc, L:/10.xx.x.97:50818 - R:qa4-util-cache.datalake.aws.qa.local/10.70.38.93:6379] message: *2

2022-08-06T02:26:40.051+01:00 $4

2022-08-06T02:26:40.051+01:00 INFO

2022-08-06T02:26:40.051+01:00 $11

2022-08-06T02:26:40.051+01:00 REPLICATION

2022-08-06T02:26:40.051+01:00 2022-08-06 01:26:40,051 | TRACE | redisson-netty-2-6 | o.r.c.handler.CommandEncoder - channel: [id: 0xbf75dc8e, L:/10.xx.x.97:46270 - R:master-qa4-util-cache.datalake.aws.qa.local/10.70.34.92:6379] message: *2

2022-08-06T02:26:40.051+01:00 $4

2022-08-06T02:26:40.051+01:00 INFO

2022-08-06T02:26:40.051+01:00 $11

2022-08-06T02:26:40.051+01:00 REPLICATION

2022-08-06T02:26:40.053+01:00 2022-08-06 01:26:40,052 | TRACE | redisson-netty-2-3 | o.r.c.handler.CommandDecoder - reply: $644

2022-08-06T02:26:40.053+01:00 # Replication

2022-08-06T02:26:40.053+01:00 role:slave

2022-08-06T02:26:40.053+01:00 master_host:master.internal.qa4-util-cache-blue-qa.ajimzl.use1.cache.amazonaws.com

2022-08-06T02:26:40.053+01:00 master_port:6379

2022-08-06T02:26:40.053+01:00 master_link_status:up

2022-08-06T02:26:40.053+01:00 master_last_io_seconds_ago:0

2022-08-06T02:26:40.053+01:00 master_sync_in_progress:0

2022-08-06T02:26:40.053+01:00 slave_repl_offset:11757771692

2022-08-06T02:26:40.053+01:00 repl_sync_enabled:1

2022-08-06T02:26:40.053+01:00 slave_read_reploff:11757771692

2022-08-06T02:26:40.053+01:00 slave_priority:100

2022-08-06T02:26:40.053+01:00 slave_read_only:1

2022-08-06T02:26:40.053+01:00 connected_slaves:0

2022-08-06T02:26:40.053+01:00 master_replid:ff6c387195a717f73666baeb6f12200c6c517775

2022-08-06T02:26:40.053+01:00 master_replid2:0000000000000000000000000000000000000000

2022-08-06T02:26:40.053+01:00 master_repl_offset:11757771692

2022-08-06T02:26:40.053+01:00 second_repl_offset:-1

2022-08-06T02:26:40.053+01:00 repl_backlog_active:1

2022-08-06T02:26:40.053+01:00 repl_backlog_size:1048576

2022-08-06T02:26:40.053+01:00 repl_backlog_first_byte_offset:11756723117

2022-08-06T02:26:40.053+01:00 repl_backlog_histlen:1048576

2022-08-06T02:26:40.053+01:00

2022-08-06T02:26:40.053+01:00 , channel: [id: 0xd32b1fbc, L:/10.xx.x.97:50818 - R:qa4-util-cache.datalake.aws.qa.local/10.70.38.93:6379], command: CommandData [promise=java.util.concurrent.CompletableFuture@7f76ef08[Not completed, 2 dependents], command=(INFO REPLICATION), params=[], codec=null]

2022-08-06T02:26:40.053+01:00 2022-08-06 01:26:40,053 | TRACE | redisson-netty-2-6 | o.r.c.handler.CommandDecoder - reply: $410

2022-08-06T02:26:40.053+01:00 # Replication

2022-08-06T02:26:40.053+01:00 role:master

2022-08-06T02:26:40.053+01:00 connected_slaves:1

2022-08-06T02:26:40.053+01:00 slave0:ip=10.31.0.54,port=6379,state=online,offset=11757673265,lag=1

2022-08-06T02:26:40.053+01:00 master_replid:ff6c387195a717f73666baeb6f12200c6c517775

2022-08-06T02:26:40.053+01:00 master_replid2:0000000000000000000000000000000000000000

2022-08-06T02:26:40.053+01:00 master_repl_offset:11757771826

2022-08-06T02:26:40.053+01:00 second_repl_offset:-1

2022-08-06T02:26:40.053+01:00 repl_backlog_active:1

2022-08-06T02:26:40.053+01:00 repl_backlog_size:1048576

2022-08-06T02:26:40.053+01:00 repl_backlog_first_byte_offset:11756723251

2022-08-06T02:26:40.053+01:00 repl_backlog_histlen:1048576

2022-08-06T02:26:40.053+01:00

2022-08-06T02:26:40.053+01:00 , channel: [id: 0xbf75dc8e, L:/10.xx.x.97:46270 - R:master-qa4-util-cache.datalake.aws.qa.local/10.70.34.92:6379], command: CommandData [promise=java.util.concurrent.CompletableFuture@361be62f[Not completed, 2 dependents], command=(INFO REPLICATION), params=[], codec=null]

2022-08-06T02:26:40.053+01:00 2022-08-06 01:26:40,053 | DEBUG | redisson-netty-2-6 | o.r.c.ReplicatedConnectionManager - Current master master-qa4-util-cache.datalake.aws.qa.local/10.70.34.92:6379 unchanged

2022-08-06T02:26:40.136+01:00 2022-08-06 01:26:40,136 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1568768208 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x4b40c128, L:/10.xx.x.97:45444 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.136+01:00 2022-08-06 01:26:40,136 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@2100538042 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x206668d5, L:/10.xx.x.97:45440 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.136+01:00 2022-08-06 01:26:40,136 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1042985161 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x27eedcf9, L:/10.xx.x.97:45778 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.137+01:00 2022-08-06 01:26:40,136 | DEBUG | redisson-netty-5-9 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.137+01:00 2022-08-06 01:26:40,137 | DEBUG | redisson-netty-5-11 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.137+01:00 2022-08-06 01:26:40,136 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1154671996 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x36c7a1db, L:/10.xx.x.97:45782 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.137+01:00 2022-08-06 01:26:40,137 | DEBUG | redisson-netty-5-10 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.138+01:00 2022-08-06 01:26:40,137 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@544557879 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xcac4e2cb, L:/10.xx.x.97:45438 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.138+01:00 2022-08-06 01:26:40,138 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1470951783 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xe8e63706, L:/10.xx.x.97:45776 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.138+01:00 2022-08-06 01:26:40,138 | DEBUG | redisson-netty-5-13 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.138+01:00 2022-08-06 01:26:40,138 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1254452412 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x1027877d, L:/10.xx.x.97:45672 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.138+01:00 2022-08-06 01:26:40,138 | DEBUG | redisson-netty-5-14 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.138+01:00 2022-08-06 01:26:40,138 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@428339238 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x90379e53, L:/10.xx.x.97:45840 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.139+01:00 2022-08-06 01:26:40,138 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@846894071 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xe3c29324, L:/10.xx.x.97:45716 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.139+01:00 2022-08-06 01:26:40,139 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@2092865978 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xadea4a46, L:/10.xx.x.97:45498 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.139+01:00 2022-08-06 01:26:40,139 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@269637534 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x0f6f4a64, L:/10.xx.x.97:48974 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.139+01:00 2022-08-06 01:26:40,139 | DEBUG | redisson-netty-5-17 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.140+01:00 2022-08-06 01:26:40,140 | DEBUG | redisson-netty-5-18 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.140+01:00 2022-08-06 01:26:40,140 | DEBUG | redisson-netty-5-19 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.140+01:00 2022-08-06 01:26:40,139 | DEBUG | redisson-netty-5-15 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.140+01:00 2022-08-06 01:26:40,139 | DEBUG | redisson-netty-5-16 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.140+01:00 2022-08-06 01:26:40,140 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@239267496 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6096897a, L:/10.xx.x.97:45842 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.141+01:00 2022-08-06 01:26:40,141 | DEBUG | redisson-netty-5-20 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.141+01:00 2022-08-06 01:26:40,141 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1811188570 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xf9479402, L:/10.xx.x.97:45574 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.142+01:00 2022-08-06 01:26:40,142 | DEBUG | redisson-netty-5-21 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.142+01:00 2022-08-06 01:26:40,142 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@774502167 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xd28dd5d6, L:/10.xx.x.97:45442 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.143+01:00 2022-08-06 01:26:40,143 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@2009264056 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x0d2a0729, L:/10.xx.x.97:45668 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.143+01:00 2022-08-06 01:26:40,143 | DEBUG | redisson-netty-5-22 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.143+01:00 2022-08-06 01:26:40,143 | DEBUG | redisson-netty-5-23 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.143+01:00 2022-08-06 01:26:40,143 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1549865868 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x4320ce9f, L:/10.xx.x.97:45558 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.144+01:00 2022-08-06 01:26:40,144 | DEBUG | redisson-netty-5-24 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.145+01:00 2022-08-06 01:26:40,145 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1286570900 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xf3032d90, L:/10.xx.x.97:45658 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.145+01:00 2022-08-06 01:26:40,145 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1761581434 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6afef39f, L:/10.xx.x.97:45554 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.145+01:00 2022-08-06 01:26:40,145 | DEBUG | redisson-netty-5-25 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.146+01:00 2022-08-06 01:26:40,146 | DEBUG | redisson-netty-5-26 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.146+01:00 2022-08-06 01:26:40,146 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@740536893 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x22367fc2, L:/10.xx.x.97:45576 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.146+01:00 2022-08-06 01:26:40,146 | DEBUG | redisson-netty-5-27 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.147+01:00 2022-08-06 01:26:40,146 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@665882685 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x1e2c027d, L:/10.xx.x.97:45654 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.147+01:00 2022-08-06 01:26:40,147 | DEBUG | redisson-netty-5-28 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.147+01:00 2022-08-06 01:26:40,147 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@320748709 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6563f27f, L:/10.xx.x.97:45552 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.147+01:00 2022-08-06 01:26:40,147 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1579777745 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x25e43c76, L:/10.xx.x.97:45714 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.147+01:00 2022-08-06 01:26:40,147 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1958654679 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xcc7e26a6, L:/10.xx.x.97:45780 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.148+01:00 2022-08-06 01:26:40,148 | DEBUG | redisson-netty-5-29 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.148+01:00 2022-08-06 01:26:40,148 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@533933130 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xb9510516, L:/10.xx.x.97:45670 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.148+01:00 2022-08-06 01:26:40,148 | DEBUG | redisson-netty-5-3 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.149+01:00 2022-08-06 01:26:40,149 | DEBUG | redisson-netty-5-12 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.150+01:00 2022-08-06 01:26:40,149 | DEBUG | redisson-netty-5-2 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.150+01:00 2022-08-06 01:26:40,150 | DEBUG | redisson-netty-5-1 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.236+01:00 2022-08-06 01:26:40,236 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1271159722 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x7865eb0d, L:/10.xx.x.97:45500 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.237+01:00 2022-08-06 01:26:40,237 | DEBUG | redisson-netty-5-4 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.336+01:00 2022-08-06 01:26:40,335 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@759484136 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x4074ec3b, L:/10.xx.x.97:45626 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.336+01:00 2022-08-06 01:26:40,336 | DEBUG | redisson-netty-5-30 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.437+01:00 2022-08-06 01:26:40,436 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1568768208 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x4b40c128, L:/10.xx.x.97:45444 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.437+01:00 2022-08-06 01:26:40,437 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1042985161 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x27eedcf9, L:/10.xx.x.97:45778 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.437+01:00 2022-08-06 01:26:40,437 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@544557879 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xcac4e2cb, L:/10.xx.x.97:45438 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.437+01:00 2022-08-06 01:26:40,437 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@2100538042 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x206668d5, L:/10.xx.x.97:45440 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.437+01:00 2022-08-06 01:26:40,437 | DEBUG | redisson-netty-5-9 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.437+01:00 2022-08-06 01:26:40,437 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1470951783 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xe8e63706, L:/10.xx.x.97:45776 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,437 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@846894071 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xe3c29324, L:/10.xx.x.97:45716 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-netty-5-6 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-netty-5-7 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-netty-5-11 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@2092865978 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xadea4a46, L:/10.xx.x.97:45498 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-netty-5-31 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1254452412 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x1027877d, L:/10.xx.x.97:45672 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-netty-5-8 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-netty-5-32 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.438+01:00 2022-08-06 01:26:40,438 | DEBUG | redisson-netty-5-5 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.439+01:00 2022-08-06 01:26:40,439 | DEBUG | redisson-netty-5-10 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.439+01:00 2022-08-06 01:26:40,439 | DEBUG | redisson-netty-5-12 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.439+01:00 2022-08-06 01:26:40,439 | DEBUG | redisson-netty-5-13 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.439+01:00 2022-08-06 01:26:40,439 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@428339238 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x90379e53, L:/10.xx.x.97:45840 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.440+01:00 2022-08-06 01:26:40,440 | DEBUG | redisson-netty-5-14 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.440+01:00 2022-08-06 01:26:40,440 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@269637534 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x0f6f4a64, L:/10.xx.x.97:48974 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.440+01:00 2022-08-06 01:26:40,440 | DEBUG | redisson-netty-5-15 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.441+01:00 2022-08-06 01:26:40,440 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@239267496 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6096897a, L:/10.xx.x.97:45842 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.441+01:00 2022-08-06 01:26:40,441 | DEBUG | redisson-netty-5-16 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.441+01:00 2022-08-06 01:26:40,441 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1811188570 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xf9479402, L:/10.xx.x.97:45574 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.441+01:00 2022-08-06 01:26:40,441 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@774502167 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xd28dd5d6, L:/10.xx.x.97:45442 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.441+01:00 2022-08-06 01:26:40,441 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@2009264056 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x0d2a0729, L:/10.xx.x.97:45668 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.442+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1549865868 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x4320ce9f, L:/10.xx.x.97:45558 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.442+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1286570900 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xf3032d90, L:/10.xx.x.97:45658 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.442+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1761581434 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6afef39f, L:/10.xx.x.97:45554 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.442+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@740536893 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x22367fc2, L:/10.xx.x.97:45576 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.442+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@665882685 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x1e2c027d, L:/10.xx.x.97:45654 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.442+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1579777745 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x25e43c76, L:/10.xx.x.97:45714 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.442+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@533933130 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xb9510516, L:/10.xx.x.97:45670 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.442+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1154671996 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x36c7a1db, L:/10.xx.x.97:45782 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.447+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-netty-5-27 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.447+01:00 2022-08-06 01:26:40,447 | DEBUG | redisson-netty-5-26 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.447+01:00 2022-08-06 01:26:40,442 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@1958654679 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0xcc7e26a6, L:/10.xx.x.97:45780 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.448+01:00 2022-08-06 01:26:40,448 | DEBUG | redisson-timer-7-1 | o.r.c.handler.ConnectionWatchdog - reconnecting RedisConnection@320748709 [redisClient=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], channel=[id: 0x6563f27f, L:/10.xx.x.97:45552 ! R:master-qa4-xxx.aws.qa.local/10.xx.x.228:6379], currentCommand=null, usage=0] to master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.448+01:00 2022-08-06 01:26:40,448 | DEBUG | redisson-netty-5-18 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.449+01:00 2022-08-06 01:26:40,448 | DEBUG | redisson-netty-5-17 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.449+01:00 2022-08-06 01:26:40,448 | DEBUG | redisson-netty-5-23 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.449+01:00 2022-08-06 01:26:40,449 | DEBUG | redisson-netty-5-25 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.449+01:00 2022-08-06 01:26:40,449 | DEBUG | redisson-netty-5-20 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.450+01:00 2022-08-06 01:26:40,449 | DEBUG | redisson-netty-5-24 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.450+01:00 2022-08-06 01:26:40,449 | DEBUG | redisson-netty-5-19 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.450+01:00 2022-08-06 01:26:40,449 | DEBUG | redisson-netty-5-21 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.450+01:00 2022-08-06 01:26:40,449 | DEBUG | redisson-netty-5-22 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.450+01:00 2022-08-06 01:26:40,450 | DEBUG | redisson-netty-5-28 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.450+01:00 2022-08-06 01:26:40,450 | DEBUG | redisson-netty-5-29 | o.r.client.RedisConnection - Connection created [addr=rediss://master-qa4-xxx.aws.qa.local:6379]

2022-08-06T02:26:40.453+01:00 2022-08-06 01:26:40,452 | ERROR | redisson-netty-5-3 | o.r.c.b.LoadBalancerManager - Unable to unfreeze entry: [freeSubscribeConnectionsAmount=1, freeSubscribeConnectionsCounter=value:51:queue:0, freeConnectionsAmount=11, freeConnectionsCounter=value:64:queue:0, freezeReason=SYSTEM, client=[addr=rediss://master-qa4-xxx.aws.qa.local:6379], nodeType=MASTER, firstFail=0]

2022-08-06T02:26:40.453+01:00

Copy
java.util.concurrent.CompletionException: org.redisson.client.RedisConnectionException: Unable to connect to Redis server: master-qa4-xxx.aws.qa.local/10.xx.x.228:6379
java.util.concurrent.CompletionException: org.redisson.client.RedisConnectionException: Unable to connect to Redis server: master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)

2022-08-06T02:26:40.453+01:00 at org.redisson.connection.pool.ConnectionPool.lambda$createConnection$1(ConnectionPool.java:151)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)

2022-08-06T02:26:40.453+01:00 at org.redisson.connection.pool.ConnectionPool.promiseFailure(ConnectionPool.java:307)

2022-08-06T02:26:40.453+01:00 at org.redisson.connection.pool.ConnectionPool.lambda$createConnection$6(ConnectionPool.java:273)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)

2022-08-06T02:26:40.453+01:00 at org.redisson.client.RedisClient$1$2.run(RedisClient.java:235)

2022-08-06T02:26:40.453+01:00 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

2022-08-06T02:26:40.453+01:00 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

2022-08-06T02:26:40.453+01:00 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

2022-08-06T02:26:40.453+01:00 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

2022-08-06T02:26:40.453+01:00 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

2022-08-06T02:26:40.453+01:00 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

2022-08-06T02:26:40.453+01:00 at java.base/java.lang.Thread.run(Unknown Source)

2022-08-06T02:26:40.453+01:00 Caused by: org.redisson.client.RedisConnectionException: Unable to connect to Redis server: master-qa4-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.453+01:00 at org.redisson.connection.pool.ConnectionPool.lambda$createConnection$1(ConnectionPool.java:150)

2022-08-06T02:26:40.453+01:00 ... 18 common frames omitted

2022-08-06T02:26:40.453+01:00 Caused by: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: master-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.encodeRelay(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture.completeRelay(Unknown Source)

2022-08-06T02:26:40.453+01:00 at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(Unknown Source)

2022-08-06T02:26:40.453+01:00 ... 10 common frames omitted

2022-08-06T02:26:40.453+01:00 Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: master-xxx.aws.qa.local/10.xx.x.228:6379

2022-08-06T02:26:40.453+01:00 Caused by: java.net.ConnectException: Connection refused

2022-08-06T02:26:40.453+01:00 at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

2022-08-06T02:26:40.453+01:00 at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)

2022-08-06T02:26:40.453+01:00 at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)

2022-08-06T02:26:40.453+01:00 at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)

2022-08-06T02:26:40.453+01:00 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)

2022-08-06T02:26:40.453+01:00 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)

2022-08-06T02:26:40.453+01:00 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)

2022-08-06T02:26:40.453+01:00 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)

2022-08-06T02:26:40.453+01:00 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

2022-08-06T02:26:40.453+01:00 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

2022-08-06T02:26:40.453+01:00 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

2022-08-06T02:26:40.453+01:00 at java.base/java.lang.Thread.run(Unknown Source)

rajuraog · 2022-08-08T16:53:01Z

@mrniko Attached the TRACE logs above when issue happened. Then after this, application continuously throws the same error (Unable to connect to Redis server) over and over until restart.

Same application with singleServerConfig detects DNS change correctly and reconnects to new cache without any issues. Below logs where it's detecting correctly and reconnecting.

2022-08-06T02:29:02.825+01:00	2022-08-06 01:29:02.825 INFO 19 --- [sson-netty-2-21] org.redisson.connection.DNSMonitor : Detected DNS change. Master rediss://master-xxx.aws.qa.local:6379 has changed ip from 10.xx.xx.228 to 10.xx.xx.150

2022-08-06T02:29:02.862+01:00	2022-08-06 01:29:02.862 INFO 19 --- [sson-netty-2-14] o.r.c.pool.MasterPubSubConnectionPool : 1 connections initialized for master-xxx.aws.qa.local/10.xx.xx.xxx:6379

2022-08-06T02:29:03.366+01:00	2022-08-06 01:29:03.366 INFO 19 --- [sson-netty-2-22] org.redisson.connection.SingleEntry : master master-xxx.aws.qa.local/10.70.38.228:6379 has changed to master-xxxx.aws.qa.local/10.xx.xx.150:6379

2022-08-06T02:29:03.366+01:00	2022-08-06 01:29:03.366 INFO 19 --- [sson-netty-2-22] o.r.c.pool.MasterConnectionPool : 24 connections initialized for master-xxx.aws.qa.local/10.xx.xx.150:6379

mrniko · 2022-08-11T14:18:20Z

@rajuraog

Can you share Redisson config?

rajuraog · 2022-08-11T14:29:49Z

@mrniko Below configuration we are using, and we add the NodeAddress and Password properties dynamically.

replicatedServersConfig:
  idleConnectionTimeout: 10000
  connectTimeout: 10000
  timeout: 30000
  retryAttempts: 10
  retryInterval: 15000
  failedSlaveReconnectionInterval: 3000
  failedSlaveCheckInterval: 60000
  password: null
  subscriptionsPerConnection: 5
  clientName: null
  loadBalancer: !<org.redisson.connection.balancer.RoundRobinLoadBalancer> {}
  subscriptionConnectionMinimumIdleSize: 1
  subscriptionConnectionPoolSize: 50
  slaveConnectionMinimumIdleSize: 24
  slaveConnectionPoolSize: 64
  masterConnectionMinimumIdleSize: 24
  masterConnectionPoolSize: 64
  readMode: "SLAVE"
  subscriptionMode: "SLAVE"
  scanInterval: 1000
  pingConnectionInterval: 10000
  dnsMonitoringInterval: 5000
threads: 16
nettyThreads: 32
codec: !<org.redisson.client.codec.StringCodec> {}
transportMode: "NIO"

Note:
Also we add the below ssl properties programmatically.

SslTruststorePassword: "no_pass"
SslEnableEndpointIdentification: false
SslProvider: SslProvider.JDK

rajuraog · 2022-08-16T09:39:46Z

@mrniko Can you please take a look at my above response to the query.

rajuraog · 2022-09-01T15:12:59Z

Hi @mrniko,

Awaiting for the response on this issue. Have provided the redisson config above. Please check.

mrniko · 2022-09-01T15:33:32Z

Same application with singleServerConfig detects DNS change correctly and reconnects to new cache without any issues

The problem might be in dns itself then. If it's not report changes for some requests.

rajuraog · 2022-09-01T15:53:43Z

@mrniko Can you please elaborate the statement ? If it's not report changes for some requests

We verified that issue is not with DNS. Also teams who are using Lettuce client to connect to the same cache able to reconnect successfully without any issues.

mrniko · 2022-09-02T10:18:17Z

sorry, I meant report old host ip for some dns requests

rajuraog · 2022-09-02T10:29:29Z

@mrniko

Yes, in the TRACE logs I attached above - it's still pointing to the old host IP which is cuasing connection refused error.

java.util.concurrent.CompletionException: org.redisson.client.RedisConnectionException: Unable to connect to Redis server: master-xxx.aws.qa.local/10.xx.xx.228:6379

In case of singleServerConfig we see logs such as "DNSMonitor - Detected DNS change" and reconnects successfully to the new host ip when DNS change happens. Issue is when we use replicatedServersConfig

mrniko · 2022-09-02T11:15:04Z

Can you set trace logging level for org.redisson package and share output for replicatedServersConfig during dns change

rajuraog · 2022-09-02T11:53:53Z

@mrniko This is the TRACE log during dns change. And log also includes the error where it's unable to connect to old host since it didn't recognize the DNS change.

mrniko · 2022-09-05T12:47:02Z

@rajuraog

I need also logs for io.netty.resolver package in trace logging level.

rajuraog · 2022-09-05T15:34:20Z

Sure @mrniko. Will enable that and provide you. May need couple of days to reproduce the issue as it's not consistent.

rajuraog · 2022-09-21T14:52:20Z

@mrniko Sorry for the delayed response. Please can you take a look.

Attaching the logs included with io.netty.resolver package in trace level which starts with i.n.resolver.
At the same time ran 2 applications one with replicatedServersConfig and other with singleServerConfig pointing to the same elasticache cluster. Application running with singleServerConfig able to detect correctly when there is DNS change and reconnected successfully whereas other application still pointing to the previously resolved IP only.

mrniko · 2022-09-22T09:35:48Z

@rajuraog

Thanks for the logs.
Can you try attached version?

redisson-3.17.7-SNAPSHOT.jar.zip

rajuraog · 2022-09-22T10:23:08Z

Sure, will do.

Thank you @mrniko

rajuraog · 2022-09-27T12:17:10Z

@mrniko

Tried the attached version, but see the IP Change errors continuously though there is no such change. Attaching the INFO and TRACE logs both.
IllegalStateException: Hostname: rediss://replica-xxx..aws.dev.local:6379 has changed IP from: replica-xxx..aws.dev.local/10.xx.x.xx:6379 to rediss://10.xx.x.xxx:6379 another attempt will be made

Also may I know when this version would be available in maven repo ? Currently had to do some additional configuration in pom file to use the attached version as we are using redisson-spring-boot-starter. Not sure if that caused this issue. Please check.

mrniko · 2022-09-27T14:21:35Z

@rajuraog

Thanks for testing and logs. I attached updated version.

redisson-3.17.7-SNAPSHOT.jar.zip

rajuraog · 2022-09-27T16:47:44Z

@mrniko

It's not working as expected, attaching the INFO and TRACE level logs.

Even though no change, library saying there is a IP change and reconnecting. It's happening continuously. Please find the attached logs.

mrniko · 2022-10-10T07:13:15Z

@rajuraog Can you set TRACE logging level for io.netty.resolver package and share output?

rajuraog · 2022-10-20T13:31:28Z

@mrniko Please find the attached logs. Also attaching the info logs. I used the latest library(3.17.7). Please check.

Note: ElastiCache having one primary and 2replica nodes.
Master node: ElastiCache Primary endpoint (master-xxx..ordermanager.aws.dev.local)
Slave node: ElastiCache Reader endpoint (replica-xxx.ordermanager.aws.dev.local)
Issue: SLAVE keep getting disconnected and reconnected between 2 replicas.

rajuraog · 2022-10-24T17:12:52Z

@mrniko Please take a look at the logs attached above.

mrniko · 2022-11-07T08:29:04Z

Something wrong with DNS server. It returns different addresss for the same host - old and new.

Line 257: 2022-10-20 14:20:55,473 | INFO | redisson-netty-2-29 | o.r.c.ReplicatedConnectionManager - Hostname: rediss://replica-kauai-redis.ordermanager.aws.dev.local:6379 has changed IP from: replica-kauai-redis.ordermanager.aws.dev.local/10.68.6.63:6379 to rediss://10.68.5.191:6379 <<< NEW!!!

Line 312: 2022-10-20 14:21:02,976 | INFO | redisson-netty-2-13 | o.r.c.ReplicatedConnectionManager - Hostname: rediss://replica-kauai-redis.ordermanager.aws.dev.local:6379 has changed IP from: replica-kauai-redis.ordermanager.aws.dev.local/10.68.5.191:6379 to rediss://10.68.6.63:6379 <<< OLD!!!

Line 835: 2022-10-20 14:21:29,103 | INFO | redisson-netty-2-23 | o.r.c.ReplicatedConnectionManager - Hostname: rediss://replica-kauai-redis.ordermanager.aws.dev.local:6379 has changed IP from: replica-kauai-redis.ordermanager.aws.dev.local/10.68.6.63:6379 to rediss://10.68.5.191:6379 <<< NEW!!!
Sear

rajuraog · 2022-11-07T12:44:30Z

@mrniko
Host replica-xxx.aws.dev.local DNS configured to point to the AWS ElastiCache Reader endpoint which has 2 replica nodes underlying. So it's returning the different addresss for the same host.

rajuraog · 2022-12-29T12:14:10Z

Hi @mrniko,

Any update on the this ? Hope my last update on the issue would be helpful to debug and fix.

Thank you!

mrniko · 2023-01-21T07:53:14Z

@rajuraog

The only solution is two specify all Redis hostnames. Since "replica" hostname can't be resolved to all IPs, but only to a single IP which is constantly changed.

rajuraog added the question label Jul 21, 2022

mrniko closed this as completed Aug 5, 2022

mrniko added this to the 3.17.7 milestone Sep 22, 2022

mrniko added bug and removed question labels Sep 23, 2022

mrniko reopened this Sep 23, 2022

mrniko pushed a commit that referenced this issue Sep 26, 2022

Fixed - DNS change isn't detected in replicated mode #4437

e6e961f

mrniko pushed a commit that referenced this issue Sep 30, 2022

Fixed - DNS change isn't detected in replicated mode #4437

aadb6cf

mrniko modified the milestones: 3.17.7, 3.17.8 Oct 5, 2022

mrniko modified the milestones: 3.18.0, 3.18.1 Nov 11, 2022

mrniko added AWS and removed bug labels Nov 17, 2022

mrniko modified the milestones: 3.18.1, 3.18.2 Dec 5, 2022

mrniko modified the milestones: 3.19.0, 3.19.1 Dec 16, 2022

mrniko modified the milestones: 3.19.1, 3.19.2 Jan 6, 2023

mrniko closed this as completed Jan 21, 2023

mrniko removed this from the 3.19.2 milestone Jan 23, 2023

mrniko mentioned this issue Feb 8, 2023

[LoadBalancerManager:200][redisson-netty-2-48][]Unable to unfreeze entry #4830

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replicated mode with AWSElastiCache not detecting DNS change when Route53 record updated #4437

Replicated mode with AWSElastiCache not detecting DNS change when Route53 record updated #4437

rajuraog commented Jul 21, 2022

mrniko commented Jul 21, 2022

rajuraog commented Jul 21, 2022

rajuraog commented Aug 2, 2022

mrniko commented Aug 2, 2022

rajuraog commented Aug 8, 2022 •

edited

rajuraog commented Aug 8, 2022 •

edited

mrniko commented Aug 11, 2022

rajuraog commented Aug 11, 2022

rajuraog commented Aug 16, 2022

rajuraog commented Sep 1, 2022

mrniko commented Sep 1, 2022 •

edited

rajuraog commented Sep 1, 2022

mrniko commented Sep 2, 2022 •

edited

rajuraog commented Sep 2, 2022 •

edited

mrniko commented Sep 2, 2022

rajuraog commented Sep 2, 2022 •

edited

mrniko commented Sep 5, 2022

rajuraog commented Sep 5, 2022

rajuraog commented Sep 21, 2022 •

edited

mrniko commented Sep 22, 2022 •

edited

rajuraog commented Sep 22, 2022

rajuraog commented Sep 27, 2022 •

edited

mrniko commented Sep 27, 2022

rajuraog commented Sep 27, 2022 •

edited

mrniko commented Oct 10, 2022

rajuraog commented Oct 20, 2022 •

edited

rajuraog commented Oct 24, 2022

mrniko commented Nov 7, 2022 •

edited

rajuraog commented Nov 7, 2022 •

edited

rajuraog commented Dec 29, 2022

mrniko commented Jan 21, 2023 •

edited

Replicated mode with AWSElastiCache not detecting DNS change when Route53 record updated #4437

Replicated mode with AWSElastiCache not detecting DNS change when Route53 record updated #4437

Comments

rajuraog commented Jul 21, 2022

mrniko commented Jul 21, 2022

rajuraog commented Jul 21, 2022

rajuraog commented Aug 2, 2022

mrniko commented Aug 2, 2022

rajuraog commented Aug 8, 2022 • edited

rajuraog commented Aug 8, 2022 • edited

mrniko commented Aug 11, 2022

rajuraog commented Aug 11, 2022

rajuraog commented Aug 16, 2022

rajuraog commented Sep 1, 2022

mrniko commented Sep 1, 2022 • edited

rajuraog commented Sep 1, 2022

mrniko commented Sep 2, 2022 • edited

rajuraog commented Sep 2, 2022 • edited

mrniko commented Sep 2, 2022

rajuraog commented Sep 2, 2022 • edited

mrniko commented Sep 5, 2022

rajuraog commented Sep 5, 2022

rajuraog commented Sep 21, 2022 • edited

mrniko commented Sep 22, 2022 • edited

rajuraog commented Sep 22, 2022

rajuraog commented Sep 27, 2022 • edited

mrniko commented Sep 27, 2022

rajuraog commented Sep 27, 2022 • edited

mrniko commented Oct 10, 2022

rajuraog commented Oct 20, 2022 • edited

rajuraog commented Oct 24, 2022

mrniko commented Nov 7, 2022 • edited

rajuraog commented Nov 7, 2022 • edited

rajuraog commented Dec 29, 2022

mrniko commented Jan 21, 2023 • edited

rajuraog commented Aug 8, 2022 •

edited

rajuraog commented Aug 8, 2022 •

edited

mrniko commented Sep 1, 2022 •

edited

mrniko commented Sep 2, 2022 •

edited

rajuraog commented Sep 2, 2022 •

edited

rajuraog commented Sep 2, 2022 •

edited

rajuraog commented Sep 21, 2022 •

edited

mrniko commented Sep 22, 2022 •

edited

rajuraog commented Sep 27, 2022 •

edited

rajuraog commented Sep 27, 2022 •

edited

rajuraog commented Oct 20, 2022 •

edited

mrniko commented Nov 7, 2022 •

edited

rajuraog commented Nov 7, 2022 •

edited

mrniko commented Jan 21, 2023 •

edited