Running out of threads during load test

[paxconG14]

Hello @vladimir-bukhtoyarov,
I am using v7.5.0 to run a 6000 tps load test using bucket4j-ignite inside a 4 node kubernetes cluster. I have 10 buckets configured like below (after reading some other discussions here):
Bandwidth classicBw = Bandwidth.classic(txAmount, Refill.intervally(txAmount, Duration.ofSeconds(wis))).withId(key); long maxUnsyncTokens = (long)(txAmount/wis) > 0 ? (long)(txAmount/wis) : 2; DelayParameters delayParams = new DelayParameters(maxUnsyncTokens, Duration.ofSeconds(1)); RemoteBucketBuilder withOptimization = proxyManager.builder().withOptimization(Optimizations.predicting(delayParams)); BucketProxy optBucketProxy = withOptimization.build(key, BucketConfiguration.builder().addLimit(classicBw).build());
after a few minutes run we get an exception like below:
Thread [name="HTTP Handler 10.128.4.1", id=5858, state=WAITING, blockCnt=1, waitCnt=2] Lock [object=java.util.concurrent.CompletableFuture$Signaller@262a52dc, ownerName=null, ownerId=-1] at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.16/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) at java.base@11.0.16/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at java.base@11.0.16/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) at java.base@11.0.16/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at java.base@11.0.16/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) at io.github.bucket4j.util.concurrent.BatchHelper$WaitingTask.waitUninterruptedly(BatchHelper.java:342) at io.github.bucket4j.util.concurrent.BatchHelper.execute(BatchHelper.java:138) at io.github.bucket4j.distributed.proxy.optimization.batch.BatchingExecutor.execute(BatchingExecutor.java:77) at io.github.bucket4j.distributed.proxy.DefaultBucketProxy.execute(DefaultBucketProxy.java:202) at io.github.bucket4j.distributed.proxy.DefaultBucketProxy.tryConsumeImpl(DefaultBucketProxy.java:81) at io.github.bucket4j.AbstractBucket.tryConsume(AbstractBucket.java:201) ...
Am I configuring the buckets somehow wrong? Can I do something to free up or unblock threads being used to consume tokens?
any hints are very much appreciated!

[vladimir-bukhtoyarov]

Currently batching optimization is not prepared for hanging of underlying backend. Most likely one of thread hangs on Ignite execution inside BatchHelper#executeBatch, if you can not find this this stack fragment inside stacktrace(of all JVM threads) then problem in something else, it is interesting to look at heapdump and full threaddump, you can share me links for dumps downloading privatelly via mail jsecoder@mail.ru

I will rewrite BatchHelper at weekend in order to add timeouts and add ability to free threads by simple interrupt.

Anyway 6000 tps does not looks huge, I have achived 1.5 milions tps on one bucket a few years ago. Did you cache buckets during the test? Optimizations work only if they applied to the same bucket instance, if you create bucket each time per request then optimizations do not work. Optimization interface has method withListener, you can use this method to collect the optimization stats.

[paxconG14]

Yes, buckets are stored in a ConcurrentHashMap and retrieved from it with every rate limit check.
The thread dumps shows

    "HTTP Handler 10.131.6.1" #472 daemon prio=5 os_prio=0 cpu=4516.40ms elapsed=2487.82s tid=0x00007fb78ca6b800 nid=0x2b2 waiting on condition  [0x00007fb61a2e0000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:323)
	at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
	at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.invoke(GridDhtAtomicCache.java:796)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.invoke(IgniteCacheProxyImpl.java:1734)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.invoke(IgniteCacheProxyImpl.java:1780)
	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.invoke(GatewayProtectedCacheProxy.java:1280)
	at io.github.bucket4j.grid.ignite.thick.IgniteProxyManager.execute(IgniteProxyManager.java:76)
	at io.github.bucket4j.distributed.proxy.AbstractProxyManager$DefaultRemoteBucketBuilder$1.execute(AbstractProxyManager.java:197)
	at io.github.bucket4j.distributed.proxy.optimization.predictive.PredictiveCommandExecutor.execute(PredictiveCommandExecutor.java:80)
	at io.github.bucket4j.distributed.proxy.optimization.batch.BatchingExecutor$3.apply(BatchingExecutor.java:58)
	at io.github.bucket4j.distributed.proxy.optimization.batch.BatchingExecutor$3.apply(BatchingExecutor.java:55)
	at io.github.bucket4j.util.concurrent.BatchHelper.execute(BatchHelper.java:132)
	at io.github.bucket4j.distributed.proxy.optimization.batch.BatchingExecutor.execute(BatchingExecutor.java:77)
	at io.github.bucket4j.distributed.proxy.DefaultBucketProxy.execute(DefaultBucketProxy.java:208)
	at io.github.bucket4j.distributed.proxy.DefaultBucketProxy.tryConsumeImpl(DefaultBucketProxy.java:83)
	at io.github.bucket4j.AbstractBucket.tryConsume(AbstractBucket.java:201) 

This problem shows up in one cluster, while in a second test cluster it works fine. Probably related to network latency, but until now I have found no proof of this.

Thanks for the hint about the optimization stats... I have been tried batch and predictive optimizations, and I am kind of curious about what the stats will show.
I am eager to test the changes on the BatchHelper. Many thanks!

[vladimir-bukhtoyarov]

@paxconG14

The hang-protection has been released with intermediate version 8.2.RC1. Please test it and provide feedback. To activate this feature you need to specify backgroundTimeBounded execution strategy for ClientSideConfig that is passed to ProxyManager

Example:

Executor executor = Executors.newFixedThreadPool(20);
Duration timeout = Duration.ofSeconds(10);
ExecutionStrategy executionStrategy = ExecutionStrategy.backgroundTimeBounded(executor, timeout);
ClientSideConfig clientSideConfig = ClientSideConfig.getDefault().withExecutionStrategy(executionStrategy);
IgniteProxyManager proxyManager = IgniteProxyManager<>(cache, clientSideConfig);

P.S. protecting client threads from hanging don't eliminate the fact that your in-memory Grid services the extra-load incorrectly, it should wake up all blocked threads and features after load became to normal, but actually it doesn't, so I think that Ignite developers should be notified about this unexpected behavior.

[paxconG14]

Hi @vladimir-bukhtoyarov , I have tried out the 8.2RC1 and using the proposed
ExecutionStrategy.backgroundTimeBounded(...)
the problem does not show up anymore :)
Many thanks!