Switch to VirtualThread for Async work/notification in JDK 21 ?

[He-Pin]

I knew Caffeine uses ForkJoinPool. common pool for scheduling/async working, but I sometimes saw large thread counts.
I want to switch to VirtualThread with the help of the .executor method, what do you think? Will caffeine change the default behavior on JDK 21, eg using VirtualThread behind the scenes?

[ben-manes]

I'd like to but for the reasons discussed in #1468, I don't consider it safe for general usage. I also don't think it should have left preview status until the pinning issue and other issues were resolved, but I understand their pressures to deliver something (especially given jigsaw, lambda, and valhalla frustrations). The next LTS seems like a good target to plan on making the switch for the default threadpool.

Javalin is an example that silently switches the default pool to virtual threads. This is a bit safer because it is an application framework, so the end user can adjust the configuration if unsuitable. Caffeine is a library that may be a transitive dependency where no configuration is exposed and the user had no knowledge or choice in the matter. Take for example NewRelic APM which uses Caffeine, while the application being monitored might not. While we could add a system property to force a global setting, there is little reason for end users to know that they need to change this or suffer production issues due to the known deadlock prone behavior in the current VT implementation.

I am personally very eager to see VTs being used as the default pool by CompletableFuture. That almost always does I/O whereas ForkJoinPool is designed for cpu-bound work and does poorly on I/O tasks. When VT mature enough that the JDK switches that default pool then it will truly be ready for production.

A lot of the time you can just set Caffeine's executor to a caller-runs policy, e.g. Runnable::run. The cache's own needs are minimal and it tries to be well behaved with direct executions. The default use of FJP for async makes sense when we are invoking foreign callbacks that the caller does not care about and the cache does not know how expensive it is. As a user if you know that you don't need async, e.g. your removal listener is very cheap, then you can disable it without any harmful effects.

[He-Pin]

Yes, that's true, a user can set abortPolicy , callerrun etc, which can cause problems, not safe for a library like caffiene.
I think I will try to replace ith .executor of caffeine with VT because I know the characters of the tasks running on the VT , currently, I'm using something like:

        SERVICE_OPTS="${SERVICE_OPTS} -Djdk.virtualThreadScheduler.maxPoolSize=256" // because of pin
        SERVICE_OPTS="${SERVICE_OPTS} -Djdk.virtualThreadScheduler.minRunnable=32"

[He-Pin]

And thanks for the detailed replay, I understand better with your input.

[ben-manes]

It sounds great for your use-case. I hope it works well. The caller-runs policy should work perfectly if there are no actual async requirements (e.g. it would make AsyncCache pointless). The unit tests mostly run with a caller-runs policy to simplify them and many users configure it for production (e.g. Infinispan, Coherence).

We do take measures to mitigate an executor that aborts by using a caller-runs when appropriate, e.g. for eviction logic or removal listener notifications. We don't do that for async loads, though, and let it fail the future and log that exception.

Executors that silently drops tasks are more problematic and is documented as unsupported. It will break async loads and notifications as never running, but async eviction will recover after a delay. We use a write buffer to avoid all writers serializing on the eviction lock and instead batch the policy updates. When that write buffer is full then a writer will barge the lock and assist by performing the pending eviction work. This was necessary due to multiple JDK bugs where ForkJoinPool lost tasks due to internal race conditions.

[He-Pin]

As a real-world critical system,(a messaging system of Taobao interactive/Live), I just saw 10pt CPU 5pt RAM reduce after migrated 80% of my thread pools to VT...

And I tried AsyncCache too, where I only got 5pt CPU and 3pt RAM reduce. (Stress testing result), So I keep the usage of normal LoadingCache instead.