Strange memory usage

[ConsoleC137]

• I have looked for existing issues (including closed) about this

Bug Report

Version

axum v0.7.4
axum-core v0.4.3

Platform

Linux [..] 6.7.2-arch1-2 #1 SMP PREEMPT_DYNAMIC Wed, 31 Jan 2024 09:22:15 +0000 x86_64 GNU/Linux

Crates

axum = "0.7.4"
tokio = { version = "1.36.0", features = ["full"] }

Description

simplescreenrecorder-2024-02-07_03.49.11.webm
Greetings, everyone.
While testing axum, I noticed a strange feature: every time I ran wrk, the memory consumption increased. In the video above, each server outputs "Hello, World!". As you can see axum's memory consumption is increasing, while actix-web behaves quite predictably. I also tried adding an asynchronous wait to emulate an I/O operation, but this had no effect on memory consumption. At first I thought it looked like a memory leak, but I didn't jump to conclusions and continued testing. I noticed one detail: axum returns memory, but in my case it happened only after a few hours. I'm afraid that in production the application might crash with an OOM error.

My code:

use axum::{response::Html, routing::get, Router};

#[tokio::main]
async fn main() {
    let app = Router::new().route("/", get(handler));

    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .unwrap();
    println!("listening on {}", listener.local_addr().unwrap());
    axum::serve(listener, app).await.unwrap();
}

async fn handler() -> Html<&'static str> {
    // optional
    tokio::time::sleep(std::time::Duration::from_millis(50)).await;

    Html("<h1>Hello, World!</h1>")
}

Please let me know if I have made any errors in the design of the Bug Report.

[Kav91]

Started noticing this as well, memory increasing by ~100-200kb per request.
Very similar behavior as mentioned here: #1546

[davidpdrsn]

Its likely just that the allocator is not releasing memory back to the OS.

[dayvejones]

Try this https://docs.rs/mimalloc/latest/mimalloc/

Mimalloc allocator performed best with the axum in my projects.

[ConsoleC137]

It's really helped me. Although immediately after starting wrk I observe a large memory consumption, it comes back to normal after some time.
PS
I also tried using mimalloc with actix-web, but my program consumed 2x more memory.

[dayvejones]

Its cool. You can also try https://github.com/tikv/jemallocator , but mimalloc showed better results in my cases.

[Kav91]

I decided to put my app into a docker container and cap the memory at 50m. I used wrk to generate some load I just did 1 thread 1 conn for X minutes to watch it accumulate.

I could get the memory usage right up to 99+% but could never get it to pass 50m and fall over. This was using mimalloc.
Could see it actively releasing memory, however after stopping the requests for perhaps an hour or so it just came down to 88.48% / 44.24MiB usage and didn't go any lower.

On the host system which has 40gb of ram, I did the same request with same test configuration, but let it go further and eventually settled at 160M,

I left both running taking no requests overnight say at least 6hr+ but neither the container or host reduced any further.

It was same compiled binary in release mode, with identical requests and responses being received.

It makes it really tricky to understand the behavior as you could ramp up from very low memory usage, to an unknown idle state or configure your minimum required memory. This makes it difficult to know when you should potentially scale your resources up or just general configuration.

[dayvejones]

One week of work.

Docker with Alpine 3.19 + mimalloc.
Its a simple service wrapper over a Postgres database. 50-300 rps.