Add active detection in sentinel mode,Reconstruct sentinel mode Listener #3547

chenshi5012 · 2023-09-20T08:12:02Z

Add active detection in sentinel mode,
Reconstruct sentinel mode Listener
Add test case for master listener for sentinel

sentinel mode， lost "+switch-master" message, should client add watchdog thread #3525
please check this, JedisSentinelPool / SentineledConnectionProvider provider monitor master-slave failover by subscribe the message from “+switch-master”。

this pr modify master-slave failover monitor，Two listening mechanisms are provided, one is the default subscription mechanism and the other is the active detection mechanism;
When due to network reasons, the client loses the subscription message and the main node connection of the jedis factory is still not switched, it will cause the connection to be unavailable.

SentinelMasterSubscribeListener the default listener
SentinelMasterActiveDetectListener the active detect listener

you can specify which listener to use; or open both. default is SubscribeListener,
`

SentinelPoolConfig config = new SentinelPoolConfig();
config.setEnableActiveDetectListener(true);   // default off 
config.setEnableDefaultSubscribeListener(true); // default on 
config.setActiveDetectIntervalTimeMillis(5*1000); // default 5s 
config.setSubscribeRetryWaitTimeMillis(5*1000); // default  5s 

JedisSentinelPool pool = new JedisSentinelPool(MASTER_NAME, sentinels, config, 1000,
        "foobared", 2);

`

…ner,Add test case for master listener for sentinel Signed-off-by: c00603587 <chenshi35@huawei.com>

codecov-commenter · 2023-09-20T08:27:51Z

Codecov Report

Attention: 47 lines in your changes are missing coverage. Please review.

Comparison is base (4cd5a6a) 71.49% compared to head (33e2028) 71.58%.
Report is 4 commits behind head on master.

❗ Current head 33e2028 differs from pull request most recent head 465e5ba. Consider uploading reports for the commit 465e5ba to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3547      +/-   ##
============================================
+ Coverage     71.49%   71.58%   +0.09%     
- Complexity     4847     4875      +28     
============================================
  Files           288      291       +3     
  Lines         15450    15507      +57     
  Branches       1093     1099       +6     
============================================
+ Hits          11046    11101      +55     
+ Misses         3927     3923       -4     
- Partials        477      483       +6

Files	Coverage Δ
src/main/java/redis/clients/jedis/Connection.java	`81.17% <ø> (-1.26%)`	⬇️
...n/java/redis/clients/jedis/SentinelPoolConfig.java	`100.00% <100.00%> (ø)`
...in/java/redis/clients/jedis/JedisSentinelPool.java	`73.94% <96.15%> (+8.58%)`	⬆️
...c/main/java/redis/clients/jedis/JedisMetaInfo.java	`68.75% <40.00%> (ø)`
.../main/java/redis/clients/jedis/CommandObjects.java	`85.65% <88.88%> (-0.13%)`	⬇️
...entinel/listener/SentinelActiveDetectListener.java	`82.05% <82.05%> (ø)`
.../jedis/providers/SentineledConnectionProvider.java	`84.21% <65.21%> (+8.02%)`	⬆️
...s/sentinel/listener/SentinelSubscribeListener.java	`57.14% <57.14%> (ø)`

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

chenshi5012 · 2023-10-17T12:09:51Z

@sazzad16 thanks for your suggest . please take a look for this PR.

sazzad16 · 2023-10-17T12:48:39Z

@chenshi5012 Have you made any change after your previous PR?

Moreover, your tests are still @Ignore.

Signed-off-by: c00603587 <chenshi35@huawei.com>

* 'master' of https://github.com/chenshi5012/jedis: (21 commits) Bump org.json:json from 20230618 to 20231013 (redis#3586) Fix SORTABLE argument issue in FT.CREATE command (redis#3584) Linking to Redis resources (redis#3583) Remove patch version snapshots Make integration and snapshot actions manually triggerable (redis#3579) Missing piece from redis#3578 Release minor version snapshots (redis#3578) Allow getting schema field name (redis#3576) Fix binary variants of XRANGE and XREAD commands (redis#3571) Support GEOSHAPE field type in RediSearch (redis#3561) Bump org.apache.commons:commons-pool2 from 2.11.1 to 2.12.0 (redis#3565) Bump junixsocket-core to 2.8.1 (redis#3573) Bump org.apache.maven.plugins:maven-javadoc-plugin from 3.5.0 to 3.6.0 (redis#3539) Broadcast FUNCTION LOAD command methods (redis#3557) Encode map in encoded object (redis#3555) Polish Triggers and functions tests (redis#3554) Different variable names for json v1 and v2 interfaces (redis#3553) Fix Search/Gears test regression (redis#3552) Polish Feature: Triggers and functions commands (redis#3551) Feature: Triggers and functions commands (redis#3531) ...

Signed-off-by: c00603587 <chenshi35@huawei.com>

This reverts commit 5cd40fc. # Conflicts: # src/test/java/redis/clients/jedis/examples/GeoShapeFieldsUsageInRediSearch.java Signed-off-by: c00603587 <chenshi35@huawei.com>

chenshi5012 · 2023-10-30T10:25:29Z

@chenshi5012 Have you made any change after your previous PR?

Moreover, your tests are still @Ignore.

ignore has beenremove and the test case move to JedisSentinelTest.java

sazzad16

Many new classes only for sentinel. May consider a new package.

sazzad16 · 2023-10-30T19:09:52Z

src/main/java/redis/clients/jedis/SentinelPoolConfig.java

+
+import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
+
+public class SentinelPoolConfig extends GenericObjectPoolConfig {


GenericObjectPoolConfig takes a type argument. Declaring a sub-class without the type argument is anti-pattern.

Idea: Just declare this interface separately without sub-classing it; with a more appropriate name.

may be sub-class is the best way to avoid add more Constructor method . currently threse are 24 Constructor method for JedisSentinelPool.
use sub-class will not break the Constructor parameter type GenericObjectPoolConfig.

If that's your concern, you can introduce a Builder.

src/main/java/redis/clients/jedis/SentinelMasterActiveDetectListener.java

src/main/java/redis/clients/jedis/SentinelMasterSubscribeListener.java

sazzad16 · 2023-10-30T19:18:47Z

src/main/java/redis/clients/jedis/SentinelMasterListener.java

+ * @see SentinelMasterSubscribeListener Passive subscription
+ * @see SentinelMasterActiveDetectListener Active detection
+ */
+public interface SentinelMasterListener {


Check comment in implementing classes.

src/test/java/redis/clients/jedis/JedisSentinelTest.java

sazzad16 · 2023-10-30T19:23:08Z

@chenshi5012 This is a not so simple PR. I already have too many things in my regular job. So, would take time to properly review this. However, added some quick comments.

Signed-off-by: c00603587 <chenshi35@huawei.com> (cherry picked from commit 23813a2)

Signed-off-by: c00603587 <chenshi35@huawei.com>

chenshi5012 · 2023-11-01T07:43:00Z

Many new classes only for sentinel. May consider a new package.

thanks for your pantinet , new package "redis.clients.jedis.sentinel.listener"

yangbodong22011 · 2023-11-02T03:54:23Z

As far as I know, SentinelListener#run will always run and possibly reinitialize sentinelMaster:

Receive "+switch-master"
After receiving JedisException, it will be reinitialized after sleep subscribeRetryWaitTimeMillis.

If your client does not recover in either case, what could be the reason?

chenshi5012 · 2023-11-02T06:37:40Z

As far as I know, SentinelListener#run will always run and possibly reinitialize sentinelMaster:

Receive "+switch-master"

After receiving JedisException, it will be reinitialized after sleep subscribeRetryWaitTimeMillis.

If your client does not recover in either case, what could be the reason?

actually ，active detect will enhanced stability. while client lost subscribe message

yangbodong22011 · 2023-11-02T12:16:45Z

@chenshi5012 Yes, I know Pubsub is unreliable, and I'd like to know what specific issues you're experiencing in production?

pubsub messages lost? Or is the client always in a silent state and unable to sense that the link is disconnected?

chenshi5012 · 2023-11-03T01:42:03Z

Yes, in Sentinel mode, we've had this problem many times in the production environment.

The data load is heavy. (80G+, flush every 30mins )
the network not stable (client in other vpc. ping-latancy Occasionally very high )
The node breaks down unexpectedly (the server and sentinel are together in one node ), triggering the active/standby switchover.

Add active detection in sentinel mode,Reconstruct sentinel mode Liste…

2c247d5

…ner,Add test case for master listener for sentinel Signed-off-by: c00603587 <chenshi35@huawei.com>

sazzad16 force-pushed the master branch from 0d7e97f to 38b31f7 Compare October 12, 2023 15:15

Merge branch 'master' into master

032d017

c00603587 added 15 commits October 19, 2023 10:32

to ensure ZSentinelMasterListenerTest run as the last one

6e06c03

Signed-off-by: c00603587 <chenshi35@huawei.com>

test case for SentinelMasterListener

3ca30da

Signed-off-by: c00603587 <chenshi35@huawei.com>

test case for SentinelMasterListener

9094868

Signed-off-by: c00603587 <chenshi35@huawei.com>

remove test order

dc6ece2

Signed-off-by: c00603587 <chenshi35@huawei.com>

test case for sentinel master listener

f19c663

Signed-off-by: c00603587 <chenshi35@huawei.com>

test case for testSentinelMasterListener

23813a2

Signed-off-by: c00603587 <chenshi35@huawei.com>

build sentinel for master listeners

32f0fbd

Signed-off-by: c00603587 <chenshi35@huawei.com>

test case for sentinel master listener

773b98b

Signed-off-by: c00603587 <chenshi35@huawei.com>

fix failover timeout time

6314ea4

Signed-off-by: c00603587 <chenshi35@huawei.com>

add more sleep time to make sure failove be detect

d7cfc28

Signed-off-by: c00603587 <chenshi35@huawei.com>

modify localhost to 127

c40bfa0

Signed-off-by: c00603587 <chenshi35@huawei.com>

test case add master listener to sentinel

4f93eae

Signed-off-by: c00603587 <chenshi35@huawei.com>

1

3611c0f

Signed-off-by: c00603587 <chenshi35@huawei.com>

Revert "Support GEOSHAPE field type in RediSearch (redis#3561)"

5e4fd16

This reverts commit 5cd40fc. # Conflicts: # src/test/java/redis/clients/jedis/examples/GeoShapeFieldsUsageInRediSearch.java Signed-off-by: c00603587 <chenshi35@huawei.com>

sazzad16 requested changes Oct 30, 2023

View reviewed changes

c00603587 added 4 commits November 1, 2023 10:35

test case for testSentinelMasterListener

7e241e8

Signed-off-by: c00603587 <chenshi35@huawei.com> (cherry picked from commit 23813a2)

modify the code structure

b9f9aa8

Signed-off-by: c00603587 <chenshi35@huawei.com>

modify comment

d7e9dd2

Signed-off-by: c00603587 <chenshi35@huawei.com>

move to new package

465e5ba

Signed-off-by: c00603587 <chenshi35@huawei.com>

sazzad16 requested a review from yangbodong22011 November 1, 2023 07:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add active detection in sentinel mode,Reconstruct sentinel mode Listener #3547

Add active detection in sentinel mode,Reconstruct sentinel mode Listener #3547

chenshi5012 commented Sep 20, 2023

codecov-commenter commented Sep 20, 2023 •

edited

chenshi5012 commented Oct 17, 2023

sazzad16 commented Oct 17, 2023

chenshi5012 commented Oct 30, 2023

sazzad16 left a comment

sazzad16 Oct 30, 2023

chenshi5012 Nov 1, 2023

sazzad16 Nov 1, 2023

sazzad16 Oct 30, 2023

chenshi5012 Nov 1, 2023

sazzad16 commented Oct 30, 2023

chenshi5012 commented Nov 1, 2023

yangbodong22011 commented Nov 2, 2023

chenshi5012 commented Nov 2, 2023

yangbodong22011 commented Nov 2, 2023

chenshi5012 commented Nov 3, 2023


		import org.apache.commons.pool2.impl.GenericObjectPoolConfig;

		public class SentinelPoolConfig extends GenericObjectPoolConfig {

Add active detection in sentinel mode,Reconstruct sentinel mode Listener #3547

Are you sure you want to change the base?

Add active detection in sentinel mode,Reconstruct sentinel mode Listener #3547

Conversation

chenshi5012 commented Sep 20, 2023

codecov-commenter commented Sep 20, 2023 • edited

Codecov Report

chenshi5012 commented Oct 17, 2023

sazzad16 commented Oct 17, 2023

chenshi5012 commented Oct 30, 2023

sazzad16 left a comment

Choose a reason for hiding this comment

sazzad16 Oct 30, 2023

Choose a reason for hiding this comment

chenshi5012 Nov 1, 2023

Choose a reason for hiding this comment

sazzad16 Nov 1, 2023

Choose a reason for hiding this comment

sazzad16 Oct 30, 2023

Choose a reason for hiding this comment

chenshi5012 Nov 1, 2023

Choose a reason for hiding this comment

sazzad16 commented Oct 30, 2023

chenshi5012 commented Nov 1, 2023

yangbodong22011 commented Nov 2, 2023

chenshi5012 commented Nov 2, 2023

yangbodong22011 commented Nov 2, 2023

chenshi5012 commented Nov 3, 2023

codecov-commenter commented Sep 20, 2023 •

edited