Memory leak in long-lived connection pool

[seanlinsley]

In the image below, memory usage is stable with prune_db_connections as described in bikeshedder/deadpool#273, plus malloc_trim(0). Then disabling prune_db_connections shows a clear memory leak.

Heaptrack shows two leaks, both around BytesMut usage:

Related discussions online:

• https://stackoverflow.com/questions/73725299
• docs: unclear how to free memory from a bytesmut tokio-rs/bytes#634
• Memory Leak on Actix 3.2.0 actix/actix-web#1780
• Memory leak on high number of concurrent connections hyperium/hyper#1790
• Proxy memory leak over HTTP 1 linkerd/linkerd2#11012

Possible debugging steps:

• add code to track every message sent/received, where the buffer size has grown since the last message (might require a fork of the bytes crate)
• write a reproduction script that's similar to the production workload (RDS, binary COPY FROM, queries that cause unique constraint error)

Possible fixes:

• revert ffd7245
  • this would likely fix the leak if it's actually a bug in BytesMut usage and not something like closed connections never being freed. and it'd be a good opportunity to confirm/deny the performance difference that motivated the change originally

[sfackler]

Do you have a self contained reproduction?

[seanlinsley]

I've tried to write a reproduction script, but it doesn't leak memory. If there is a bug in BytesMut usage it might be around the exact handling of messages sent/received to a remote database, using both regular queries (with heavy use of array binds) and binary COPY FROM, and with some queries resulting in unique constraint errors.