update value / release of memory

[s-weil]

Hi

I like Moka and its API very much but found the following issue (possibly due to my own incorrect usage of it):

In my actual app I have some background job updating the values of the cache periodically.
Since the API doesn't seem to provide an actual update function, I simply insert the new value for the same key again and
before doing so, I invalidate the key. This 'update' is of course not atomic, and moreover presented the following issue:
While this indeed seems to overwrite the value, I noticed that with each update the memory of the app increases where the value itself should have a similar size.

Below is an example which should simulate this behaviour (happy to provide more details).

Is there a better way to update values or to invalidate / release the value of the key?

Thanks for your help

// moka = { version = "0.6", features = ["future"] }
// tokio = { version = "1", features = ["rt-multi-thread", "macros" ] }
// futures = "0.3"

use moka::future::Cache;
use std::sync::Arc;
use std::{thread, time::Duration};

async fn add_or_update(
    my_cache: &Cache<String, Arc<Vec<usize>>>,
    key_str: &str,
    key_value: Vec<usize>,
) {
    my_cache.invalidate(&key_str.to_string()).await;
    my_cache
        .insert(key_str.to_string(), Arc::new(key_value))
        .await;
    println!(
        "Inserted new value for {}",
        key_str
    );
}

#[tokio::main]
async fn main() {
    const NUM_TASKS: usize = 100;

    let cache = Cache::new(1);

    let single_key = "key";

    let tasks: Vec<_> = (0..NUM_TASKS)
        .map(|task_idx| {
            let my_cache = cache.clone();

            // throttle process to monitor memory
            thread::sleep(Duration::from_millis(10 * (task_idx as u64)));

            tokio::spawn(async move {
                // create some 'large' object and add it to the cache with the same key
                {
                    let key_value = (0..task_idx + 10_000_000).collect::<Vec<_>>();
                    add_or_update(&my_cache, &single_key, key_value).await;
                }

                // get back the value
                {
                    let key_value = my_cache
                        .get(&single_key.to_string())
                        .unwrap_or_else(|| Arc::new(vec![]));
                    println!(
                        "Got value for task_idx {}, len = {}",
                        task_idx,
                        key_value.len()
                    );
                }
            })
        })
        .collect();

    // wait for all tasks to complete.
    futures::future::join_all(tasks).await;

    // wait a bit longer to monitor memory
    thread::sleep(Duration::from_millis(10_000));

    println!("Finished");
}

[tatsuya6502]

Hi. Thank you for reporting the issue. I have not had a chance to run your code yet, but will try soon.

I think you are doing right, so it will be Moka's problem.

Moka uses a concurrent hash table called cht as the central storage. And I know there are a couple of places to improve in cht for better memory usage.

One such thing is a usage of crossbeam-epoch. cht uses crossbeam-epoch, which provides lock-free shared garbage collected pointers. Maybe the issue you are seeing is caused by it. By design, it will not immediately drop values pointed by its pointers, and will keep last few hundreds of them in memory after they were marked as "droppable".

I once opened a GitHub issue for it, but closed after I realized it was by design: Gregory-Meyer/cht#23 (comment)

$ cargo run --release --bin main2
...
inserted: 10000, dropped: 9832

I will check whether your issue is caused by it or not. And if so, try to come up with options to mitigate the issue. It seems cht's development has been stalled, so I forked it and will maintain it by myself.

[tatsuya6502]

By design, it will not immediately drop values pointed by its pointers, and will keep last few hundreds of them in memory after they were marked as "droppable".

Relevant issues for crossbeam-epoch:

• SkipList Memory Release Issue crossbeam-rs/crossbeam#540
• extreme memory usage with crossbeam-epoch crossbeam-rs/crossbeam#566

[s-weil]

Thanks a lot for your answer and clarification!

[tatsuya6502]

@snusnusnu — Sorry for taking long time, but I solved the issue via #169. I will publish it to crate.io soon as a part of Moka v0.9.3.

Here are the memory allocation histories captured by cargo-instruments with macOS Profiler. I ran your program with Moka v0.9.2 (before the fix) and Moka v0.9.3-dev after the fix. As you can see, while the former was using max 6.51GiB of memory, the latter used only 185.75MiB.

Moka v0.9.2 (before the fix)

Moka v0.9.3-dev (after the fix)

You can try the fix now by specifying the git URL for the Moka dependency:

[dependencies]
moka = { git = "https://github.com/moka-rs/moka", branch = "master", features = ["future"] }

Thanks!

[s-weil]

Hi @tatsuya6502

Many thanks for taking care of the issue!
Just tested it and get a similar picture, that's great.

Cheers