Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition in JedisSentinelPool.initPool() causes JedisConnectionException on getResource() #1910

Closed
mp911de opened this issue Dec 6, 2018 · 2 comments
Milestone

Comments

@mp911de
Copy link
Contributor

mp911de commented Dec 6, 2018

JedisSentinelPool.initPool() is called concurrently by multiple threads during statup: The thread creating JedisSentinelPool and MasterListener. The consequence is that calls to getResource() fail with an exception.

Reproduction of the issue requires just the right timing so calling the offending code in a while(true) loop allows reproduction instead of single-shot execution.

Jedis version: 3.0.0

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

	at redis.clients.jedis.util.Pool.getResource(Pool.java:59)
	at redis.clients.jedis.JedisSentinelPool.getResource(JedisSentinelPool.java:209)
	at org.springframework.data.redis.connection.jedis.Reproducer.reproduceSentinelPoolRaceCondition(Reproducer.java:43)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.NullPointerException
	at redis.clients.jedis.util.Pool.getResource(Pool.java:50)
	... 24 more

Reproducer

public class Reproducer {

  @Test
  public void reproduceSentinelPoolRaceCondition() {

    while (true) {

      GenericObjectPoolConfig poolConfig = new JedisPoolConfig();
      JedisSentinelPool jedisSentinelPool = new JedisSentinelPool("mymaster",
          new HashSet<>(Arrays.asList("127.0.0.1:26379", "127.0.0.1:26380")), 
          poolConfig, 1000, 1000, null, 0,
          "jedis-client");

      jedisSentinelPool.getResource().close();

      jedisSentinelPool.destroy();
    }
  }
}
mp911de added a commit to spring-projects/spring-data-redis that referenced this issue Dec 6, 2018
@gkorland
Copy link
Contributor

gkorland commented Dec 9, 2018

It seems like a race condition between the main thread running the Constructor and then initPool() and the sentinel listeners threads are also trying to call initPool() in parallel

@gkorland
Copy link
Contributor

gkorland commented Dec 9, 2018

#1566 it seems like this fix is the source of this bug

@sazzad16 sazzad16 added this to the 3.0.1 milestone Dec 12, 2018
gkorland added a commit that referenced this issue Dec 25, 2018
fix #1910 protect initPool from being called in parallel
gkorland added a commit that referenced this issue Dec 25, 2018
mp911de added a commit to spring-projects/spring-data-redis that referenced this issue Jan 7, 2019
snicoll pushed a commit to snicoll/jedis that referenced this issue Mar 5, 2019
gkorland added a commit that referenced this issue Mar 5, 2019
Backport of gh-1910 to fix race condition
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants