Pm2 God Daemon crashes and does not properly recover

[anonymous-sushi]

What's going wrong?

I have about 50 pm2 python processes that run and occasionally the pm2 god daemon dies with an error FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory. i have pm2 log rotation addon enabled and not sure how to troubleshoot this error. its running on a server with 128 GB ram that is never even 50% utilized, I've tried adding --max-old-space-size=2048" but i dont think it works. Ideally if this happens id like to able for pm2 to just restart all the programs but pm2 resurrect causes ghost instances of the old processes and messes up the discord bots that dont properly respond anymore even with pm2 resurrect causing me to have to do a full reboot into pm2 resurrect to properly fix it

How could we reproduce this issue?

run a bunch of python programs in pm2 that use an ok amount of memory and handles alot of redis caching + pub/sub and see if pm2 god daemon gets out of heap memory

pm2 start "python -u my_program.py" --name my_program--node-args="--max-old-space-size=2048"

Supporting information

<--- Last few GCs --->

[2159:0x664d350] 1328199 ms: Mark-Compact 4059.5 (4136.7) -> 4053.2 (4137.0) MB, 2555.0 / 0.0 ms (average mu = 0.084, current mu = 0.020) task; scavenge might not succeed
[2159:0x664d350] 1330700 ms: Mark-Compact 4059.8 (4137.0) -> 4053.4 (4137.2) MB, 2457.5 / 0.0 ms (average mu = 0.052, current mu = 0.017) task; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xbc8560 node::Abort() [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
2: 0xadc6b0 [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
3: 0xdaf470 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
4: 0xdaf826 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
5: 0xfae2a5 [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
6: 0xfae856 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
7: 0xfc0b46 [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
8: 0xfc1785 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
9: 0x1035254 v8::internal::ScavengeJob::Task::RunInternal() [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
10: 0xc33594 [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
11: 0xc369fe node::PerIsolatePlatformData::FlushForegroundTasksInternal() [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
12: 0x176dea6 [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
13: 0x17803e4 [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
14: 0x176e80e uv_run [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
15: 0xb0ba2d node::SpinEventLoopInternal(node::Environment*) [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
16: 0xc0e0f4 node::NodeMainInstance::Run() [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
17: 0xb83155 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResultImpl const*) [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
18: 0xb86ca1 node::Start(int, char**) [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
19: 0x7ff4c7300d90 [/lib/x86_64-linux-gnu/libc.so.6]
20: 0x7ff4c7300e40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
21: 0xb08f7e _start [PM2 v5.3.1: God Daemon (/home/root/.pm2)]

# Run the following commands
$ pm2 report

--- PM2 report ----------------------------------------------------------------
Date                 : Tue Apr 16 2024 15:27:24 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.3.1
node version         : 19.3.0
node path            : /home/root//.nvm/versions/node/v19.3.0/bin/pm2
argv                 : /home/root/.nvm/versions/node/v19.3.0/bin/node,/home/root/.nvm/versions/node/v19.3.0/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 1000
gid                  : 1000
uptime               : 4min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.3.1
node version         : 19.3.0
node path            : /home/root//.nvm/versions/node/v19.3.0/bin/pm2
argv                 : /home/root/.nvm/versions/node/v19.3.0/bin/node,/home/root/.nvm/versions/node/v19.3.0/bin/pm2,report
argv0                : node
user                 : root
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : AMD Ryzen 9 7950X3D 16-Core Processor
cpus nb              : 32
freemem              : 124505329664
totalmem             : 134234898432
home                 : /home/root/
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬──────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name             │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
└────┴──────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
Module
┌────┬──────────────────────────────┬───────────────┬──────────┬──────────┬──────┬──────────┬──────────┬──────────┐
│ id │ module                       │ version       │ pid      │ status   │ ↺    │ cpu      │ mem      │ user     │
├────┼──────────────────────────────┼───────────────┼──────────┼──────────┼──────┼──────────┼──────────┼──────────┤
│ 0  │ pm2-logrotate                │ 2.7.0         │ 11797    │ online   │ 0    │ 0%       │ 60.9mb   │ root    │
└────┴──────────────────────────────┴───────────────┴──────────┴──────────┴──────┴──────────┴──────────┴──────────┘
[PM2][WARN] Current process list is not synchronized with saved list. App 50+ python apps differs. Type 'pm2 save' to synchronize.
===============================================================================
--- Daemon logs --------------------------------------------
/home/root/.pm2/pm2.log last 20 lines:
PM2        | 20: 0x7ff4c7300e40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
PM2        | 21: 0xb08f7e _start [PM2 v5.3.1: God Daemon (/home/root/.pm2)]
PM2        | 2024-04-16T15:23:16: PM2 log: ===============================================================================
PM2        | 2024-04-16T15:23:16: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
PM2        | 2024-04-16T15:23:16: PM2 log: Time                 : Tue Apr 16 2024 15:23:16 GMT+0000 (Coordinated Universal Time)
PM2        | 2024-04-16T15:23:16: PM2 log: PM2 version          : 5.3.1
PM2        | 2024-04-16T15:23:16: PM2 log: Node.js version      : 19.3.0
PM2        | 2024-04-16T15:23:16: PM2 log: Current arch         : x64
PM2        | 2024-04-16T15:23:16: PM2 log: PM2 home             : /home/root/.pm2
PM2        | 2024-04-16T15:23:16: PM2 log: PM2 PID file         : /home/root/.pm2/pm2.pid
PM2        | 2024-04-16T15:23:16: PM2 log: RPC socket file      : /home/root/.pm2/rpc.sock
PM2        | 2024-04-16T15:23:16: PM2 log: BUS socket file      : /home/root/.pm2/pub.sock
PM2        | 2024-04-16T15:23:16: PM2 log: Application log path : /home/root/.pm2/logs
PM2        | 2024-04-16T15:23:16: PM2 log: Worker Interval      : 30000
PM2        | 2024-04-16T15:23:16: PM2 log: Process dump file    : /home/root/.pm2/dump.pm2
PM2        | 2024-04-16T15:23:16: PM2 log: Concurrent actions   : 2
PM2        | 2024-04-16T15:23:16: PM2 log: SIGTERM timeout      : 1600
PM2        | 2024-04-16T15:23:16: PM2 log: ===============================================================================
PM2        | 2024-04-16T15:23:16: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
PM2        | 2024-04-16T15:23:16: PM2 log: App [pm2-logrotate:0] online

Please copy/paste the above report in your issue on https://github.com/Unitech/pm2/issues