Why use future over sync?

[waynerobinson]

I'm a little new to Rust specifically, but I'm a little confused as to why I'd use the async vs sync version of Moka.

Isn't Moka in-memory with all operations being synchronous against the underlying data structures?

In my experience, other languages tend to only really resort to async processing when you have to wait on IO or, in some circumstances, are spreading load across multiple threads.

Wouldn't using Moka in sync mode always be faster than async, even from within a standard async loop (i.e. from Tokio)?

[tatsuya6502]

Hi. Sorry for late reply.

Although async version of Moka cache will be slightly slower than sync cache, we would recommend to use async cache when it is used within async context (i.e. from Tokio). Async cache will never block the executor threads of async runtime, while sync cache may (i.e. when waiting for a lock). Async cache will keep the async runtime responsive.

Here are some obvious examples:

1. get_with method (doc):
   • This method takes a user function to initialize the value for the key.
   • If the key does not exist, get_with will evaluate the user function to get the value to insert.
   • Concurrent calls on the same not-existing key are coalesced into one evaluation of the user function.
     • This is achieved by using a key-level lock. One of the concurrent calls will get the write lock and start to evaluate the user function. Other calls will wait for read lock. The read lock will become available when the value is ready to read.
     • Sync cache uses regular lock so these other calls will block async runtime's executor threads until the value becomes available.
     • Async cache uses async-aware lock, which will never block the executor threads. The executor threads can run other async tasks until the value becomes available.
2. Eviction listener with the immediate mode (doc):
   • Eviction listener is a user function that will be called when a cached entry is evicted.
   • Like get_with, the immediate mode requires key-level lock. Sync cache uses regular lock so it can block the executor threads.

It will be rare but other methods, such as insert and invalidate, can block for a short period of time under heavy writes (doc). And again, sync cache can block the executor threads when it is happened.

About the speed, here is a micro-benchmark result. The async cache was about 9% slower than the sync cache. But cache read and write operation will be fast enough in both sync and async caches. 9% means 75 nanoseconds in the benchmark.

• Benchmark tool: mokabench
• Command: ./target/release/mokabench --num-clients 1 --insert-once --entry-api
• Moka v0.11.2, Tokio v1.28.2, Rust v1.70.0
• Linux x86_64 (Ubuntu 22.04 LTS)
• Intel Core i7-12700F

Cache	Max Capacity	Clients	Inserts	Reads	Hit Ratio	Duration Secs	Nanoseconds per Op
Moka Sync Cache	100,000	1	14,676,973	16,407,702	10.548%	25.766 s	828.897 ns
Moka Async Cache on Tokio	100,000	1	14,679,582	16,407,702	10.532%	28.767 s	925.362 ns	11.6% (96.465 ns) slower
Moka Sync Cache	400,000	1	9,437,929	16,407,702	42.479%	24.054 s	930.680 ns
Moka Async Cache on Tokio	400,000	1	9,423,074	16,407,702	42.569%	25.759 s	997.221 ns	7.1% (66.542 ns) slower
Moka Sync Cache	800,000	1	4,881,783	16,407,702	70.247%	15.796 s	741.963 ns
Moka Async Cache on Tokio	800,000	1	4,878,901	16,407,702	70.265%	17.111 s	803.839 ns	8.3% (61.876 ns) slower

[waynerobinson]

Thank you for the detailed response. This has made the reason for async over sync within the context of an async environment (like Tokio) much more obivous.

Cheers

[eeczw]

Speaking of the blocking behavior under heavy writes, I'm wondering if it would be better if an async-channel is adopted in the async cache.

We can call try_send() several times before calling a final send()(which might block the task, but not the thread). In this way, we can yield the thread to other async tasks and let the channel wake the task once the channel has space for new insertions. It might be a better implementation than the current lock-to-yield implementaion. I can be wrong here though.

Please let me know how you feel about it. @tatsuya6502

[tatsuya6502]

Hi. Thank you for the suggestion! Yeah, you are right; for future::Cache of current v0.12.0-beta.X, async-channel should be better and more natural solution than doing this lock-to-yield implementation. I will try it for next beta release.

In earlier versions (up to v0.11.x), the receiver side of the channel was a regular fn driven by an OS thread of scheduled-thread-pool, and future::Cache shared the same code base with sync::Cache for the area. If we used async-channel, we had to introduce more dependencies to sync::Cache, which we wanted to avoid. (future::Cache in v0.12.0 no longer shares the code base with sync::Cache in the area)

[tatsuya6502]

• Replace crossbeam-channel with async-channel for future::Cache #320

[tatsuya6502]

Hmm. I realized it will not going to work; I ended up with all async tasks waiting for send(op).await to finish while no async task is receiving from the channel: #320 (comment)

I will revisit this in a couple of days and update #320.

[tatsuya6502]

OK. I found a solution: #323.