Memory leak in tokio runtime. #3550

tuxzz · 2021-02-24T13:19:21Z

Version
└── tokio v1.2.0
└── tokio-macros v1.1.0 (proc-macro)

Platform
Linux TukuZaZa-D1 5.11.1-zen1-1-zen-uksm #1 ZEN SMP PREEMPT Wed, 24 Feb 2021 07:02:49 +0000 x86_64 GNU/Linux

Description
Tokio always leak memory after drop a tokio::runtime::Runtime.

fn main() {
  for _ in 0..1000 {
    let rt = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap();
  }
}

If you increse the times of loop, the leaked memory reported by valgrind is also increasing.

Valgrind report:

$ valgrind --leak-check=full --show-leak-kinds=all target/debug/test

==3645212== Memcheck, a memory error detector
==3645212== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3645212== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==3645212== Command: target/debug/test
==3645212== 
==3645212== 
==3645212== HEAP SUMMARY:
==3645212==     in use at exit: 221,180 bytes in 2,043 blocks
==3645212==   total heap usage: 40,067 allocs, 38,024 frees, 26,859,837 bytes allocated
==3645212== 
==3645212== 32 bytes in 1 blocks are possibly lost in loss record 1 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x17A1BB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x17A279: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==3645212==    by 0x17CAB9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==3645212==    by 0x17A11C: alloc::alloc::exchange_malloc (alloc.rs:316)
==3645212==    by 0x165627: alloc::sync::Arc<T>::new (sync.rs:330)
==3645212==    by 0x16A8D2: <alloc::sync::Arc<T> as core::convert::From<T>>::from (sync.rs:2178)
==3645212==    by 0x1346C1: signal_hook_registry::register_unchecked_impl (lib.rs:572)
==3645212==    by 0x134527: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==3645212==    by 0x135090: signal_hook_registry::register (lib.rs:498)
==3645212==    by 0x133CF8: tokio::signal::unix::signal_enable::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:244)
==3645212==    by 0x19ED6B: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==3645212== 
==3645212== 40 bytes in 1 blocks are still reachable in loss record 2 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x21938C: alloc (alloc.rs:86)
==3645212==    by 0x21938C: alloc_impl (alloc.rs:166)
==3645212==    by 0x21938C: allocate (alloc.rs:226)
==3645212==    by 0x21938C: exchange_malloc (alloc.rs:316)
==3645212==    by 0x21938C: new<std::sys::unix::mutex::Mutex> (boxed.rs:186)
==3645212==    by 0x21938C: from<std::sys::unix::mutex::Mutex> (boxed.rs:1015)
==3645212==    by 0x21938C: std::sys_common::mutex::MovableMutex::new (library/std/src/sys_common/mutex.rs:64)
==3645212==    by 0x1C6BB6: std::sync::mutex::Mutex<T>::new (mutex.rs:217)
==3645212==    by 0x1CAAD5: signal_hook_registry::half_lock::HalfLock<T>::new (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/half_lock.rs:121)
==3645212==    by 0x1CC8C1: signal_hook_registry::GlobalData::ensure::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/lib.rs:286)
==3645212==    by 0x1C8F5B: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==3645212==    by 0x218701: std::sync::once::Once::call_inner (library/std/src/sync/once.rs:420)
==3645212==    by 0x1C8EDC: std::sync::once::Once::call_once (once.rs:261)
==3645212==    by 0x1CC833: signal_hook_registry::GlobalData::ensure (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/lib.rs:284)
==3645212==    by 0x134687: signal_hook_registry::register_unchecked_impl (lib.rs:571)
==3645212==    by 0x134527: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==3645212==    by 0x135090: signal_hook_registry::register (lib.rs:498)
==3645212== 
==3645212== 40 bytes in 1 blocks are still reachable in loss record 3 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x21938C: alloc (alloc.rs:86)
==3645212==    by 0x21938C: alloc_impl (alloc.rs:166)
==3645212==    by 0x21938C: allocate (alloc.rs:226)
==3645212==    by 0x21938C: exchange_malloc (alloc.rs:316)
==3645212==    by 0x21938C: new<std::sys::unix::mutex::Mutex> (boxed.rs:186)
==3645212==    by 0x21938C: from<std::sys::unix::mutex::Mutex> (boxed.rs:1015)
==3645212==    by 0x21938C: std::sys_common::mutex::MovableMutex::new (library/std/src/sys_common/mutex.rs:64)
==3645212==    by 0x1C6BB6: std::sync::mutex::Mutex<T>::new (mutex.rs:217)
==3645212==    by 0x1CAC80: signal_hook_registry::half_lock::HalfLock<T>::new (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/half_lock.rs:121)
==3645212==    by 0x1CC8E2: signal_hook_registry::GlobalData::ensure::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/lib.rs:290)
==3645212==    by 0x1C8F5B: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==3645212==    by 0x218701: std::sync::once::Once::call_inner (library/std/src/sync/once.rs:420)
==3645212==    by 0x1C8EDC: std::sync::once::Once::call_once (once.rs:261)
==3645212==    by 0x1CC833: signal_hook_registry::GlobalData::ensure (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/lib.rs:284)
==3645212==    by 0x134687: signal_hook_registry::register_unchecked_impl (lib.rs:571)
==3645212==    by 0x134527: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==3645212==    by 0x135090: signal_hook_registry::register (lib.rs:498)
==3645212== 
==3645212== 56 bytes in 1 blocks are still reachable in loss record 4 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x17A1BB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x17A279: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==3645212==    by 0x17CAB9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==3645212==    by 0x17A11C: alloc::alloc::exchange_malloc (alloc.rs:316)
==3645212==    by 0x177908: pin<tokio::signal::registry::Globals> (boxed.rs:242)
==3645212==    by 0x177908: tokio::signal::registry::globals::GLOBALS::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:169)
==3645212==    by 0x135AFD: core::ops::function::FnOnce::call_once (function.rs:227)
==3645212==    by 0x135D5A: core::ops::function::FnOnce::call_once (function.rs:227)
==3645212==    by 0x1B8828: once_cell::sync::Lazy<T,F>::force::{{closure}} (lib.rs:1023)
==3645212==    by 0x1B891E: once_cell::sync::OnceCell<T>::get_or_init::{{closure}} (lib.rs:845)
==3645212==    by 0x1A1B08: once_cell::imp::OnceCell<T>::initialize::{{closure}} (imp_std.rs:93)
==3645212==    by 0x1E3BAB: once_cell::imp::initialize_inner (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.6.0/src/imp_std.rs:168)
==3645212== 
==3645212== 64 bytes in 1 blocks are still reachable in loss record 5 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x1C2CDB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x1C2D99: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==3645212==    by 0x1C33E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==3645212==    by 0x1C2C3C: alloc::alloc::exchange_malloc (alloc.rs:316)
==3645212==    by 0x1CA90C: new<signal_hook_registry::SignalData> (boxed.rs:186)
==3645212==    by 0x1CA90C: signal_hook_registry::half_lock::WriteGuard<T>::store (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/half_lock.rs:75)
==3645212==    by 0x134DFA: signal_hook_registry::register_unchecked_impl (lib.rs:610)
==3645212==    by 0x134527: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==3645212==    by 0x135090: signal_hook_registry::register (lib.rs:498)
==3645212==    by 0x133CF8: tokio::signal::unix::signal_enable::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:244)
==3645212==    by 0x19ED6B: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==3645212==    by 0x218701: std::sync::once::Once::call_inner (library/std/src/sync/once.rs:420)
==3645212== 
==3645212== 168 bytes in 1 blocks are still reachable in loss record 6 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x17A1BB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x17A279: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==3645212==    by 0x17CAB9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==3645212==    by 0x17A11C: alloc::alloc::exchange_malloc (alloc.rs:316)
==3645212==    by 0x16E88C: new<core::option::Option<signal_hook_registry::Prev>> (boxed.rs:186)
==3645212==    by 0x16E88C: signal_hook_registry::half_lock::WriteGuard<T>::store (half_lock.rs:75)
==3645212==    by 0x134A85: signal_hook_registry::register_unchecked_impl (lib.rs:599)
==3645212==    by 0x134527: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==3645212==    by 0x135090: signal_hook_registry::register (lib.rs:498)
==3645212==    by 0x133CF8: tokio::signal::unix::signal_enable::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:244)
==3645212==    by 0x19ED6B: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==3645212==    by 0x218701: std::sync::once::Once::call_inner (library/std/src/sync/once.rs:420)
==3645212== 
==3645212== 368 bytes in 1 blocks are possibly lost in loss record 7 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x1C2CDB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x1C2D99: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==3645212==    by 0x1C33E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==3645212==    by 0x1C2C3C: alloc::alloc::exchange_malloc (alloc.rs:316)
==3645212==    by 0x1BA980: new<alloc::collections::btree::node::LeafNode<signal_hook_registry::ActionId, alloc::sync::Arc<Fn<(&libc::unix::linux_like::linux::gnu::b64::x86_64::siginfo_t)>>>> (boxed.rs:186)
==3645212==    by 0x1BA980: alloc::collections::btree::node::NodeRef<alloc::collections::btree::node::marker::Owned,K,V,alloc::collections::btree::node::marker::Leaf>::new_leaf (node.rs:136)
==3645212==    by 0x1BB619: alloc::collections::btree::node::NodeRef<alloc::collections::btree::node::marker::Owned,K,V,alloc::collections::btree::node::marker::LeafOrInternal>::new (node.rs:130)
==3645212==    by 0x1CE569: core::ops::function::FnOnce::call_once (function.rs:227)
==3645212==    by 0x1C4D4F: core::option::Option<T>::get_or_insert_with (option.rs:869)
==3645212==    by 0x1CFFEE: alloc::collections::btree::map::BTreeMap<K,V>::ensure_is_owned (map.rs:2164)
==3645212==    by 0x1A00AD: alloc::collections::btree::map::BTreeMap<K,V>::entry (map.rs:1050)
==3645212==    by 0x1A036D: alloc::collections::btree::map::BTreeMap<K,V>::insert (map.rs:796)
==3645212== 
==3645212== 788 bytes in 1 blocks are possibly lost in loss record 8 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x1C2CDB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x1C0526: hashbrown::raw::RawTable<T>::new_uninitialized (mod.rs:411)
==3645212==    by 0x19A8E6: hashbrown::raw::RawTable<T>::fallible_with_capacity (mod.rs:440)
==3645212==    by 0x19B7A2: hashbrown::raw::RawTable<T>::resize (mod.rs:873)
==3645212==    by 0x198693: hashbrown::raw::RawTable<T>::reserve_rehash (mod.rs:754)
==3645212==    by 0x19C401: hashbrown::raw::RawTable<T>::reserve (mod.rs:707)
==3645212==    by 0x1B8551: hashbrown::map::HashMap<K,V,S>::reserve (map.rs:670)
==3645212==    by 0x1B7E18: hashbrown::rustc_entry::<impl hashbrown::map::HashMap<K,V,S>>::rustc_entry (rustc_entry.rs:45)
==3645212==    by 0x182277: std::collections::hash::map::HashMap<K,V,S>::entry (map.rs:705)
==3645212==    by 0x1347E4: signal_hook_registry::register_unchecked_impl (lib.rs:580)
==3645212==    by 0x134527: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==3645212== 
==3645212== 1,320 bytes in 33 blocks are still reachable in loss record 9 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x21938C: alloc (alloc.rs:86)
==3645212==    by 0x21938C: alloc_impl (alloc.rs:166)
==3645212==    by 0x21938C: allocate (alloc.rs:226)
==3645212==    by 0x21938C: exchange_malloc (alloc.rs:316)
==3645212==    by 0x21938C: new<std::sys::unix::mutex::Mutex> (boxed.rs:186)
==3645212==    by 0x21938C: from<std::sys::unix::mutex::Mutex> (boxed.rs:1015)
==3645212==    by 0x21938C: std::sys_common::mutex::MovableMutex::new (library/std/src/sys_common/mutex.rs:64)
==3645212==    by 0x19EEDB: std::sync::mutex::Mutex<T>::new (mutex.rs:217)
==3645212==    by 0x1A092D: <std::sync::mutex::Mutex<T> as core::default::Default>::default (mutex.rs:417)
==3645212==    by 0x17799B: <tokio::signal::registry::EventInfo as core::default::Default>::default (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:20)
==3645212==    by 0x1332F0: <tokio::signal::unix::SignalInfo as core::default::Default>::default (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:196)
==3645212==    by 0x13315A: tokio::signal::unix::<impl tokio::signal::registry::Init for alloc::vec::Vec<tokio::signal::unix::SignalInfo>>::init::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:31)
==3645212==    by 0x13C534: core::iter::adapters::map::map_fold::{{closure}} (map.rs:80)
==3645212==    by 0x12BEEB: core::iter::traits::iterator::Iterator::fold (iterator.rs:2023)
==3645212==    by 0x134109: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold (map.rs:120)
==3645212==    by 0x13C1B8: core::iter::traits::iterator::Iterator::for_each (iterator.rs:678)
==3645212==    by 0x192B88: <alloc::vec::Vec<T,A> as alloc::vec::SpecExtend<T,I>>::spec_extend (vec.rs:2568)
==3645212== 
==3645212== 2,112 bytes in 1 blocks are still reachable in loss record 10 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x17A1BB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x17A279: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==3645212==    by 0x17CAB9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==3645212==    by 0x1B0EDF: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:188)
==3645212==    by 0x1B552C: alloc::raw_vec::RawVec<T,A>::with_capacity_in (raw_vec.rs:129)
==3645212==    by 0x18FF3E: alloc::vec::Vec<T,A>::with_capacity_in (vec.rs:498)
==3645212==    by 0x18E975: alloc::vec::Vec<T>::with_capacity (vec.rs:364)
==3645212==    by 0x192F8E: <alloc::vec::Vec<T> as alloc::vec::SpecFromIterNested<T,I>>::from_iter (vec.rs:2331)
==3645212==    by 0x19277A: <alloc::vec::Vec<T> as alloc::vec::SpecFromIter<T,I>>::from_iter (vec.rs:2346)
==3645212==    by 0x1939E5: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter (vec.rs:2181)
==3645212==    by 0x13C06A: core::iter::traits::iterator::Iterator::collect (iterator.rs:1670)
==3645212== 
==3645212== 8,192 bytes in 1 blocks are still reachable in loss record 11 of 13
==3645212==    at 0x4840D7B: realloc (vg_replace_malloc.c:834)
==3645212==    by 0x1D2B4C: alloc::alloc::realloc (alloc.rs:122)
==3645212==    by 0x1D282C: alloc::alloc::Global::grow_impl (alloc.rs:198)
==3645212==    by 0x1D3103: <alloc::alloc::Global as core::alloc::Allocator>::grow (alloc.rs:251)
==3645212==    by 0x1D455C: alloc::raw_vec::finish_grow (raw_vec.rs:487)
==3645212==    by 0x1B53AF: alloc::raw_vec::RawVec<T,A>::grow_amortized (raw_vec.rs:422)
==3645212==    by 0x1B1723: alloc::raw_vec::RawVec<T,A>::try_reserve (raw_vec.rs:311)
==3645212==    by 0x1B6772: alloc::raw_vec::RawVec<T,A>::reserve (raw_vec.rs:305)
==3645212==    by 0x1912B8: alloc::vec::Vec<T,A>::reserve (vec.rs:697)
==3645212==    by 0x190264: alloc::vec::Vec<T,A>::push (vec.rs:1409)
==3645212==    by 0x1772DA: tokio::signal::registry::Registry<S>::register_listener (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:71)
==3645212==    by 0x17779D: tokio::signal::registry::Globals::register_listener (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:141)
==3645212== 
==3645212== 32,000 bytes in 1,000 blocks are still reachable in loss record 12 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x17A1BB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x17A279: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==3645212==    by 0x17CAB9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==3645212==    by 0x17A11C: alloc::alloc::exchange_malloc (alloc.rs:316)
==3645212==    by 0x1ADEA4: new<tokio::sync::mpsc::block::Block<()>> (boxed.rs:186)
==3645212==    by 0x1ADEA4: tokio::sync::mpsc::list::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/list.rs:35)
==3645212==    by 0x177A04: tokio::sync::mpsc::chan::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/chan.rs:105)
==3645212==    by 0x1604CF: tokio::sync::mpsc::bounded::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/bounded.rs:93)
==3645212==    by 0x133E95: tokio::signal::unix::signal_with_handle (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:365)
==3645212==    by 0x189A0B: tokio::process::imp::driver::Driver::new (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/process/unix/driver.rs:59)
==3645212==    by 0x176EFC: tokio::runtime::driver::create_process_driver (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/runtime/driver.rs:84)
==3645212==    by 0x17676A: tokio::runtime::driver::create_io_stack (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/runtime/driver.rs:26)
==3645212== 
==3645212== 176,000 bytes in 1,000 blocks are still reachable in loss record 13 of 13
==3645212==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==3645212==    by 0x17A1BB: alloc::alloc::alloc (alloc.rs:86)
==3645212==    by 0x17A279: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==3645212==    by 0x17CAB9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==3645212==    by 0x17A11C: alloc::alloc::exchange_malloc (alloc.rs:316)
==3645212==    by 0x1658CD: alloc::sync::Arc<T>::new (sync.rs:330)
==3645212==    by 0x177BDD: tokio::sync::mpsc::chan::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/chan.rs:107)
==3645212==    by 0x1604CF: tokio::sync::mpsc::bounded::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/bounded.rs:93)
==3645212==    by 0x133E95: tokio::signal::unix::signal_with_handle (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:365)
==3645212==    by 0x189A0B: tokio::process::imp::driver::Driver::new (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/process/unix/driver.rs:59)
==3645212==    by 0x176EFC: tokio::runtime::driver::create_process_driver (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/runtime/driver.rs:84)
==3645212==    by 0x17676A: tokio::runtime::driver::create_io_stack (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/runtime/driver.rs:26)
==3645212== 
==3645212== LEAK SUMMARY:
==3645212==    definitely lost: 0 bytes in 0 blocks
==3645212==    indirectly lost: 0 bytes in 0 blocks
==3645212==      possibly lost: 1,188 bytes in 3 blocks
==3645212==    still reachable: 219,992 bytes in 2,040 blocks
==3645212==         suppressed: 0 bytes in 0 blocks
==3645212== 
==3645212== For lists of detected and suppressed errors, rerun with: -s
==3645212== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)

The text was updated successfully, but these errors were encountered:

carllerche · 2021-02-24T16:59:55Z

@ipetkov this looks signal related, are you able to take an initial look?

ipetkov · 2021-02-25T05:11:59Z

I think I know what's happening here, but first some context:

We use a global data structure to keep track of all registered signal handlers and any listeners. This structure must be global based on how Unix signals work and the data cannot be scoped to a runtime
Any previously registered signal listeners are lazily pruned the next time that signal is delivered (since we have to grab the lock anyway). When a signal listener is dropped it does not deregister itself (keeps things fast, avoids having to grab a lock to deregister)
When the process feature is enabled and the runtime's enable_io is set each runtime will eagerly create a SIGCHLD listener on construction (which internally registers a signal listener)

Given those points above, my hunch is that the "leaked" memory is from all those SIGCHLD listeners.

@tuxzz could you please confirm the following for me:

if you compile tokio without the process feature, does the leak still grow with the number of runtimes being created and dropped? (my hunch is no, the leak will remain static)
if you compile tokio without the signal feature as well, is there still a memory leak? (my hunch is no, there will be no more leak)

So what can we do about this (if we should even do anything at all)?

We can probably add a clean up hook to eagerly clean out any dropped signal listeners whenever a runtime is dropped (runtimes are unlikely to be dropped in a hot loop so maybe we can afford to try to grab the lock and do the clean up). However, we would also likely need shrink the backing vectors which would be wasted effort if other runtimes exist (since they may need to grow them back again).
As for cleaning up the signal globals it may be tricky since we'll need to keep track of when the last runtime is dropped before doing the cleanup. If the process is about to exit anyway maybe the clean up isn't even worth it. We also may not be able to free up all the memory because it's virtually impossible (i.e. unsafe) to deregister our signal handler in the general sense.

It sucks that anyone using tokio with the process or signal features enabled will see some leaked data in their valgrind output. To the extent possible I'd like to pick any low hanging fruit to avoid this, but it remains to be seen how much effort is worth it

tuxzz · 2021-02-25T05:29:18Z

@ipetkov
Without the process and signal feature, there will be no more leak.

Without the process feature, the leak remains static.

$ valgrind --leak-check=full --show-leak-kinds=all target/debug/test
 
==2076288== Memcheck, a memory error detector
==2076288== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2076288== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==2076288== Command: target/debug/test
==2076288== 
==2076288== 
==2076288== HEAP SUMMARY:
==2076288==     in use at exit: 3,488 bytes in 35 blocks
==2076288==   total heap usage: 380,049 allocs, 380,014 frees, 266,285,753 bytes allocated
==2076288== 
==2076288== 56 bytes in 1 blocks are still reachable in loss record 1 of 3
==2076288==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2076288==    by 0x156F5B: alloc::alloc::alloc (alloc.rs:86)
==2076288==    by 0x157019: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2076288==    by 0x1594E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2076288==    by 0x156EBC: alloc::alloc::exchange_malloc (alloc.rs:316)
==2076288==    by 0x174A68: pin<tokio::signal::registry::Globals> (boxed.rs:242)
==2076288==    by 0x174A68: tokio::signal::registry::globals::GLOBALS::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:169)
==2076288==    by 0x14347D: core::ops::function::FnOnce::call_once (function.rs:227)
==2076288==    by 0x14357A: core::ops::function::FnOnce::call_once (function.rs:227)
==2076288==    by 0x1813F8: once_cell::sync::Lazy<T,F>::force::{{closure}} (lib.rs:1023)
==2076288==    by 0x18145E: once_cell::sync::OnceCell<T>::get_or_init::{{closure}} (lib.rs:845)
==2076288==    by 0x190098: once_cell::imp::OnceCell<T>::initialize::{{closure}} (imp_std.rs:93)
==2076288==    by 0x1B974B: once_cell::imp::initialize_inner (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.6.0/src/imp_std.rs:168)
==2076288== 
==2076288== 1,320 bytes in 33 blocks are still reachable in loss record 2 of 3
==2076288==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2076288==    by 0x1EED3C: alloc (alloc.rs:86)
==2076288==    by 0x1EED3C: alloc_impl (alloc.rs:166)
==2076288==    by 0x1EED3C: allocate (alloc.rs:226)
==2076288==    by 0x1EED3C: exchange_malloc (alloc.rs:316)
==2076288==    by 0x1EED3C: new<std::sys::unix::mutex::Mutex> (boxed.rs:186)
==2076288==    by 0x1EED3C: from<std::sys::unix::mutex::Mutex> (boxed.rs:1015)
==2076288==    by 0x1EED3C: std::sys_common::mutex::MovableMutex::new (library/std/src/sys_common/mutex.rs:64)
==2076288==    by 0x125CDB: std::sync::mutex::Mutex<T>::new (mutex.rs:217)
==2076288==    by 0x13013D: <std::sync::mutex::Mutex<T> as core::default::Default>::default (mutex.rs:417)
==2076288==    by 0x174AFB: <tokio::signal::registry::EventInfo as core::default::Default>::default (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:20)
==2076288==    by 0x139390: <tokio::signal::unix::SignalInfo as core::default::Default>::default (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:196)
==2076288==    by 0x13923A: tokio::signal::unix::<impl tokio::signal::registry::Init for alloc::vec::Vec<tokio::signal::unix::SignalInfo>>::init::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:31)
==2076288==    by 0x13A444: core::iter::adapters::map::map_fold::{{closure}} (map.rs:80)
==2076288==    by 0x12681B: core::iter::traits::iterator::Iterator::fold (iterator.rs:2023)
==2076288==    by 0x139529: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold (map.rs:120)
==2076288==    by 0x13A0A8: core::iter::traits::iterator::Iterator::for_each (iterator.rs:678)
==2076288==    by 0x187C08: <alloc::vec::Vec<T,A> as alloc::vec::SpecExtend<T,I>>::spec_extend (vec.rs:2568)
==2076288== 
==2076288== 2,112 bytes in 1 blocks are still reachable in loss record 3 of 3
==2076288==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2076288==    by 0x156F5B: alloc::alloc::alloc (alloc.rs:86)
==2076288==    by 0x157019: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2076288==    by 0x1594E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2076288==    by 0x12944F: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:188)
==2076288==    by 0x12E34C: alloc::raw_vec::RawVec<T,A>::with_capacity_in (raw_vec.rs:129)
==2076288==    by 0x18546E: alloc::vec::Vec<T,A>::with_capacity_in (vec.rs:498)
==2076288==    by 0x184205: alloc::vec::Vec<T>::with_capacity (vec.rs:364)
==2076288==    by 0x18829E: <alloc::vec::Vec<T> as alloc::vec::SpecFromIterNested<T,I>>::from_iter (vec.rs:2331)
==2076288==    by 0x18783A: <alloc::vec::Vec<T> as alloc::vec::SpecFromIter<T,I>>::from_iter (vec.rs:2346)
==2076288==    by 0x188A15: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter (vec.rs:2181)
==2076288==    by 0x139F9A: core::iter::traits::iterator::Iterator::collect (iterator.rs:1670)
==2076288== 
==2076288== LEAK SUMMARY:
==2076288==    definitely lost: 0 bytes in 0 blocks
==2076288==    indirectly lost: 0 bytes in 0 blocks
==2076288==      possibly lost: 0 bytes in 0 blocks
==2076288==    still reachable: 3,488 bytes in 35 blocks
==2076288==         suppressed: 0 bytes in 0 blocks
==2076288== 
==2076288== For lists of detected and suppressed errors, rerun with: -s
==2076288== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Without the signal feature, the leak still grows.

$ cargo build && valgrind --leak-check=full --show-leak-kinds=all target/debug/test
==2100011== Memcheck, a memory error detector
==2100011== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2100011== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==2100011== Command: target/debug/test
==2100011== 
==2100011== 
==2100011== HEAP SUMMARY:
==2100011==     in use at exit: 2,216,060 bytes in 20,043 blocks
==2100011==   total heap usage: 400,071 allocs, 380,028 frees, 268,629,597 bytes allocated
==2100011== 
==2100011== 32 bytes in 1 blocks are possibly lost in loss record 1 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x17A1FB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x17A2B9: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2100011==    by 0x17CA79: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2100011==    by 0x17A15C: alloc::alloc::exchange_malloc (alloc.rs:316)
==2100011==    by 0x1654B7: alloc::sync::Arc<T>::new (sync.rs:330)
==2100011==    by 0x16A902: <alloc::sync::Arc<T> as core::convert::From<T>>::from (sync.rs:2178)
==2100011==    by 0x134771: signal_hook_registry::register_unchecked_impl (lib.rs:572)
==2100011==    by 0x1345D7: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==2100011==    by 0x135140: signal_hook_registry::register (lib.rs:498)
==2100011==    by 0x133DA8: tokio::signal::unix::signal_enable::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:244)
==2100011==    by 0x19EDBB: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==2100011== 
==2100011== 40 bytes in 1 blocks are still reachable in loss record 2 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x21938C: alloc (alloc.rs:86)
==2100011==    by 0x21938C: alloc_impl (alloc.rs:166)
==2100011==    by 0x21938C: allocate (alloc.rs:226)
==2100011==    by 0x21938C: exchange_malloc (alloc.rs:316)
==2100011==    by 0x21938C: new<std::sys::unix::mutex::Mutex> (boxed.rs:186)
==2100011==    by 0x21938C: from<std::sys::unix::mutex::Mutex> (boxed.rs:1015)
==2100011==    by 0x21938C: std::sys_common::mutex::MovableMutex::new (library/std/src/sys_common/mutex.rs:64)
==2100011==    by 0x1C6BB6: std::sync::mutex::Mutex<T>::new (mutex.rs:217)
==2100011==    by 0x1CAAD5: signal_hook_registry::half_lock::HalfLock<T>::new (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/half_lock.rs:121)
==2100011==    by 0x1CC8C1: signal_hook_registry::GlobalData::ensure::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/lib.rs:286)
==2100011==    by 0x1C8F5B: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==2100011==    by 0x218701: std::sync::once::Once::call_inner (library/std/src/sync/once.rs:420)
==2100011==    by 0x1C8EDC: std::sync::once::Once::call_once (once.rs:261)
==2100011==    by 0x1CC833: signal_hook_registry::GlobalData::ensure (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/lib.rs:284)
==2100011==    by 0x134737: signal_hook_registry::register_unchecked_impl (lib.rs:571)
==2100011==    by 0x1345D7: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==2100011==    by 0x135140: signal_hook_registry::register (lib.rs:498)
==2100011== 
==2100011== 40 bytes in 1 blocks are still reachable in loss record 3 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x21938C: alloc (alloc.rs:86)
==2100011==    by 0x21938C: alloc_impl (alloc.rs:166)
==2100011==    by 0x21938C: allocate (alloc.rs:226)
==2100011==    by 0x21938C: exchange_malloc (alloc.rs:316)
==2100011==    by 0x21938C: new<std::sys::unix::mutex::Mutex> (boxed.rs:186)
==2100011==    by 0x21938C: from<std::sys::unix::mutex::Mutex> (boxed.rs:1015)
==2100011==    by 0x21938C: std::sys_common::mutex::MovableMutex::new (library/std/src/sys_common/mutex.rs:64)
==2100011==    by 0x1C6BB6: std::sync::mutex::Mutex<T>::new (mutex.rs:217)
==2100011==    by 0x1CAC80: signal_hook_registry::half_lock::HalfLock<T>::new (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/half_lock.rs:121)
==2100011==    by 0x1CC8E2: signal_hook_registry::GlobalData::ensure::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/lib.rs:290)
==2100011==    by 0x1C8F5B: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==2100011==    by 0x218701: std::sync::once::Once::call_inner (library/std/src/sync/once.rs:420)
==2100011==    by 0x1C8EDC: std::sync::once::Once::call_once (once.rs:261)
==2100011==    by 0x1CC833: signal_hook_registry::GlobalData::ensure (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/lib.rs:284)
==2100011==    by 0x134737: signal_hook_registry::register_unchecked_impl (lib.rs:571)
==2100011==    by 0x1345D7: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==2100011==    by 0x135140: signal_hook_registry::register (lib.rs:498)
==2100011== 
==2100011== 56 bytes in 1 blocks are still reachable in loss record 4 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x17A1FB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x17A2B9: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2100011==    by 0x17CA79: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2100011==    by 0x17A15C: alloc::alloc::exchange_malloc (alloc.rs:316)
==2100011==    by 0x1778A8: pin<tokio::signal::registry::Globals> (boxed.rs:242)
==2100011==    by 0x1778A8: tokio::signal::registry::globals::GLOBALS::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:169)
==2100011==    by 0x135B7D: core::ops::function::FnOnce::call_once (function.rs:227)
==2100011==    by 0x135D5A: core::ops::function::FnOnce::call_once (function.rs:227)
==2100011==    by 0x1B8828: once_cell::sync::Lazy<T,F>::force::{{closure}} (lib.rs:1023)
==2100011==    by 0x1B891E: once_cell::sync::OnceCell<T>::get_or_init::{{closure}} (lib.rs:845)
==2100011==    by 0x1A1B58: once_cell::imp::OnceCell<T>::initialize::{{closure}} (imp_std.rs:93)
==2100011==    by 0x1E3BAB: once_cell::imp::initialize_inner (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.6.0/src/imp_std.rs:168)
==2100011== 
==2100011== 64 bytes in 1 blocks are still reachable in loss record 5 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x1C2CDB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x1C2D99: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2100011==    by 0x1C33E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2100011==    by 0x1C2C3C: alloc::alloc::exchange_malloc (alloc.rs:316)
==2100011==    by 0x1CA90C: new<signal_hook_registry::SignalData> (boxed.rs:186)
==2100011==    by 0x1CA90C: signal_hook_registry::half_lock::WriteGuard<T>::store (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/signal-hook-registry-1.3.0/src/half_lock.rs:75)
==2100011==    by 0x134EAA: signal_hook_registry::register_unchecked_impl (lib.rs:610)
==2100011==    by 0x1345D7: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==2100011==    by 0x135140: signal_hook_registry::register (lib.rs:498)
==2100011==    by 0x133DA8: tokio::signal::unix::signal_enable::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:244)
==2100011==    by 0x19EDBB: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==2100011==    by 0x218701: std::sync::once::Once::call_inner (library/std/src/sync/once.rs:420)
==2100011== 
==2100011== 168 bytes in 1 blocks are still reachable in loss record 6 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x17A1FB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x17A2B9: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2100011==    by 0x17CA79: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2100011==    by 0x17A15C: alloc::alloc::exchange_malloc (alloc.rs:316)
==2100011==    by 0x16E87C: new<core::option::Option<signal_hook_registry::Prev>> (boxed.rs:186)
==2100011==    by 0x16E87C: signal_hook_registry::half_lock::WriteGuard<T>::store (half_lock.rs:75)
==2100011==    by 0x134B35: signal_hook_registry::register_unchecked_impl (lib.rs:599)
==2100011==    by 0x1345D7: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==2100011==    by 0x135140: signal_hook_registry::register (lib.rs:498)
==2100011==    by 0x133DA8: tokio::signal::unix::signal_enable::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:244)
==2100011==    by 0x19EDBB: std::sync::once::Once::call_once::{{closure}} (once.rs:261)
==2100011==    by 0x218701: std::sync::once::Once::call_inner (library/std/src/sync/once.rs:420)
==2100011== 
==2100011== 368 bytes in 1 blocks are possibly lost in loss record 7 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x1C2CDB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x1C2D99: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2100011==    by 0x1C33E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2100011==    by 0x1C2C3C: alloc::alloc::exchange_malloc (alloc.rs:316)
==2100011==    by 0x1BA980: new<alloc::collections::btree::node::LeafNode<signal_hook_registry::ActionId, alloc::sync::Arc<Fn<(&libc::unix::linux_like::linux::gnu::b64::x86_64::siginfo_t)>>>> (boxed.rs:186)
==2100011==    by 0x1BA980: alloc::collections::btree::node::NodeRef<alloc::collections::btree::node::marker::Owned,K,V,alloc::collections::btree::node::marker::Leaf>::new_leaf (node.rs:136)
==2100011==    by 0x1BB619: alloc::collections::btree::node::NodeRef<alloc::collections::btree::node::marker::Owned,K,V,alloc::collections::btree::node::marker::LeafOrInternal>::new (node.rs:130)
==2100011==    by 0x1CE569: core::ops::function::FnOnce::call_once (function.rs:227)
==2100011==    by 0x1C4D4F: core::option::Option<T>::get_or_insert_with (option.rs:869)
==2100011==    by 0x1CFFEE: alloc::collections::btree::map::BTreeMap<K,V>::ensure_is_owned (map.rs:2164)
==2100011==    by 0x1A00FD: alloc::collections::btree::map::BTreeMap<K,V>::entry (map.rs:1050)
==2100011==    by 0x1A03BD: alloc::collections::btree::map::BTreeMap<K,V>::insert (map.rs:796)
==2100011== 
==2100011== 788 bytes in 1 blocks are possibly lost in loss record 8 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x1C2CDB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x1C0526: hashbrown::raw::RawTable<T>::new_uninitialized (mod.rs:411)
==2100011==    by 0x19A936: hashbrown::raw::RawTable<T>::fallible_with_capacity (mod.rs:440)
==2100011==    by 0x19B7F2: hashbrown::raw::RawTable<T>::resize (mod.rs:873)
==2100011==    by 0x1984D3: hashbrown::raw::RawTable<T>::reserve_rehash (mod.rs:754)
==2100011==    by 0x19C4F1: hashbrown::raw::RawTable<T>::reserve (mod.rs:707)
==2100011==    by 0x1B8551: hashbrown::map::HashMap<K,V,S>::reserve (map.rs:670)
==2100011==    by 0x1B7E18: hashbrown::rustc_entry::<impl hashbrown::map::HashMap<K,V,S>>::rustc_entry (rustc_entry.rs:45)
==2100011==    by 0x1822F7: std::collections::hash::map::HashMap<K,V,S>::entry (map.rs:705)
==2100011==    by 0x134894: signal_hook_registry::register_unchecked_impl (lib.rs:580)
==2100011==    by 0x1345D7: signal_hook_registry::register_sigaction_impl (lib.rs:527)
==2100011== 
==2100011== 1,320 bytes in 33 blocks are still reachable in loss record 9 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x21938C: alloc (alloc.rs:86)
==2100011==    by 0x21938C: alloc_impl (alloc.rs:166)
==2100011==    by 0x21938C: allocate (alloc.rs:226)
==2100011==    by 0x21938C: exchange_malloc (alloc.rs:316)
==2100011==    by 0x21938C: new<std::sys::unix::mutex::Mutex> (boxed.rs:186)
==2100011==    by 0x21938C: from<std::sys::unix::mutex::Mutex> (boxed.rs:1015)
==2100011==    by 0x21938C: std::sys_common::mutex::MovableMutex::new (library/std/src/sys_common/mutex.rs:64)
==2100011==    by 0x19F02B: std::sync::mutex::Mutex<T>::new (mutex.rs:217)
==2100011==    by 0x1A097D: <std::sync::mutex::Mutex<T> as core::default::Default>::default (mutex.rs:417)
==2100011==    by 0x17793B: <tokio::signal::registry::EventInfo as core::default::Default>::default (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:20)
==2100011==    by 0x1333A0: <tokio::signal::unix::SignalInfo as core::default::Default>::default (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:196)
==2100011==    by 0x13320A: tokio::signal::unix::<impl tokio::signal::registry::Init for alloc::vec::Vec<tokio::signal::unix::SignalInfo>>::init::{{closure}} (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:31)
==2100011==    by 0x13C5E4: core::iter::adapters::map::map_fold::{{closure}} (map.rs:80)
==2100011==    by 0x12BEEB: core::iter::traits::iterator::Iterator::fold (iterator.rs:2023)
==2100011==    by 0x134269: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold (map.rs:120)
==2100011==    by 0x13C258: core::iter::traits::iterator::Iterator::for_each (iterator.rs:678)
==2100011==    by 0x1929C8: <alloc::vec::Vec<T,A> as alloc::vec::SpecExtend<T,I>>::spec_extend (vec.rs:2568)
==2100011== 
==2100011== 2,112 bytes in 1 blocks are still reachable in loss record 10 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x17A1FB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x17A2B9: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2100011==    by 0x17CA79: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2100011==    by 0x1B0EDF: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:188)
==2100011==    by 0x1B562C: alloc::raw_vec::RawVec<T,A>::with_capacity_in (raw_vec.rs:129)
==2100011==    by 0x18FE0E: alloc::vec::Vec<T,A>::with_capacity_in (vec.rs:498)
==2100011==    by 0x18E9E5: alloc::vec::Vec<T>::with_capacity (vec.rs:364)
==2100011==    by 0x19339E: <alloc::vec::Vec<T> as alloc::vec::SpecFromIterNested<T,I>>::from_iter (vec.rs:2331)
==2100011==    by 0x1927FA: <alloc::vec::Vec<T> as alloc::vec::SpecFromIter<T,I>>::from_iter (vec.rs:2346)
==2100011==    by 0x193A25: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter (vec.rs:2181)
==2100011==    by 0x13C11A: core::iter::traits::iterator::Iterator::collect (iterator.rs:1670)
==2100011== 
==2100011== 131,072 bytes in 1 blocks are still reachable in loss record 11 of 13
==2100011==    at 0x4840D7B: realloc (vg_replace_malloc.c:834)
==2100011==    by 0x1D2B4C: alloc::alloc::realloc (alloc.rs:122)
==2100011==    by 0x1D282C: alloc::alloc::Global::grow_impl (alloc.rs:198)
==2100011==    by 0x1D3103: <alloc::alloc::Global as core::alloc::Allocator>::grow (alloc.rs:251)
==2100011==    by 0x1D455C: alloc::raw_vec::finish_grow (raw_vec.rs:487)
==2100011==    by 0x1B3D5F: alloc::raw_vec::RawVec<T,A>::grow_amortized (raw_vec.rs:422)
==2100011==    by 0x1B1723: alloc::raw_vec::RawVec<T,A>::try_reserve (raw_vec.rs:311)
==2100011==    by 0x1B6A12: alloc::raw_vec::RawVec<T,A>::reserve (raw_vec.rs:305)
==2100011==    by 0x191338: alloc::vec::Vec<T,A>::reserve (vec.rs:697)
==2100011==    by 0x190974: alloc::vec::Vec<T,A>::push (vec.rs:1409)
==2100011==    by 0x17727A: tokio::signal::registry::Registry<S>::register_listener (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:71)
==2100011==    by 0x17773D: tokio::signal::registry::Globals::register_listener (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/registry.rs:141)
==2100011== 
==2100011== 320,000 bytes in 10,000 blocks are still reachable in loss record 12 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x17A1FB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x17A2B9: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2100011==    by 0x17CA79: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2100011==    by 0x17A15C: alloc::alloc::exchange_malloc (alloc.rs:316)
==2100011==    by 0x1ADEA4: new<tokio::sync::mpsc::block::Block<()>> (boxed.rs:186)
==2100011==    by 0x1ADEA4: tokio::sync::mpsc::list::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/list.rs:35)
==2100011==    by 0x1779A4: tokio::sync::mpsc::chan::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/chan.rs:105)
==2100011==    by 0x16049F: tokio::sync::mpsc::bounded::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/bounded.rs:93)
==2100011==    by 0x133F45: tokio::signal::unix::signal_with_handle (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:365)
==2100011==    by 0x189A57: tokio::process::imp::driver::Driver::new (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/process/unix/driver.rs:59)
==2100011==    by 0x176E9C: tokio::runtime::driver::create_process_driver (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/runtime/driver.rs:84)
==2100011==    by 0x17670A: tokio::runtime::driver::create_io_stack (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/runtime/driver.rs:26)
==2100011== 
==2100011== 1,760,000 bytes in 10,000 blocks are still reachable in loss record 13 of 13
==2100011==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==2100011==    by 0x17A1FB: alloc::alloc::alloc (alloc.rs:86)
==2100011==    by 0x17A2B9: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==2100011==    by 0x17CA79: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==2100011==    by 0x17A15C: alloc::alloc::exchange_malloc (alloc.rs:316)
==2100011==    by 0x16470D: alloc::sync::Arc<T>::new (sync.rs:330)
==2100011==    by 0x177B7D: tokio::sync::mpsc::chan::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/chan.rs:107)
==2100011==    by 0x16049F: tokio::sync::mpsc::bounded::channel (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/sync/mpsc/bounded.rs:93)
==2100011==    by 0x133F45: tokio::signal::unix::signal_with_handle (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/signal/unix.rs:365)
==2100011==    by 0x189A57: tokio::process::imp::driver::Driver::new (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/process/unix/driver.rs:59)
==2100011==    by 0x176E9C: tokio::runtime::driver::create_process_driver (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/runtime/driver.rs:84)
==2100011==    by 0x17670A: tokio::runtime::driver::create_io_stack (/home/tuxzz/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.2.0/src/runtime/driver.rs:26)
==2100011== 
==2100011== LEAK SUMMARY:
==2100011==    definitely lost: 0 bytes in 0 blocks
==2100011==    indirectly lost: 0 bytes in 0 blocks
==2100011==      possibly lost: 1,188 bytes in 3 blocks
==2100011==    still reachable: 2,214,872 bytes in 20,040 blocks
==2100011==         suppressed: 0 bytes in 0 blocks
==2100011== 
==2100011== For lists of detected and suppressed errors, rerun with: -s
==2100011== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)

ipetkov · 2021-02-25T05:37:33Z

Great, thanks for confirming!

Without the signal feature, the leak still grows.

Worth noting that when the process feature is enabled it will internally enable signal processing on Unix platforms, that's why the leak is still growing in this scenario

carllerche · 2021-02-25T18:40:41Z

Ideally, we can avoid requiring allocating memory in the static data structure. Is there a way to refactor signal handling to use sync::Notify.

carllerche · 2021-02-25T18:58:00Z

After a quick chat in Discord, it sounds like there may be two issues.

A signal handling requires global state. As long as that state remains a constant size, cleaning it up when the process exits is not required. Valgrind may complain, but it is benign.
Currently, there is per runtime state that is created and not actively freed. This is a bug. @ipetkov has identified the source. It should be possible to refactor that aspect to use Notify (linked above) or sync::Watch to resolve the per-runtime leak.

We will first fix 2) and see how that impacts things. @ipetkov has generously volunteered to look into it over the next few days.

ipetkov · 2021-03-17T00:09:32Z

This has been fixed in #3564, now the memory "leak" is constant and does not grow with the number of created runtimes.

Going to resolve this, but feel free to reopen if you have other questions!

tuxzz added A-tokio Area: The main tokio crate C-bug Category: This is a bug. labels Feb 24, 2021

carllerche assigned ipetkov Feb 24, 2021

ipetkov added M-process Module: tokio/process M-signal Module: tokio/signal labels Feb 25, 2021

ipetkov mentioned this issue Feb 28, 2021

Fix memory leak/growth when creating many runtimes #3564

Merged

ipetkov closed this as completed Mar 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory leak in tokio runtime. #3550

Memory leak in tokio runtime. #3550

tuxzz commented Feb 24, 2021

carllerche commented Feb 24, 2021

ipetkov commented Feb 25, 2021

tuxzz commented Feb 25, 2021

ipetkov commented Feb 25, 2021

carllerche commented Feb 25, 2021

carllerche commented Feb 25, 2021

ipetkov commented Mar 17, 2021

Memory leak in tokio runtime. #3550

Memory leak in tokio runtime. #3550

Comments

tuxzz commented Feb 24, 2021

carllerche commented Feb 24, 2021

ipetkov commented Feb 25, 2021

tuxzz commented Feb 25, 2021

ipetkov commented Feb 25, 2021

carllerche commented Feb 25, 2021

carllerche commented Feb 25, 2021

ipetkov commented Mar 17, 2021