Memory issues / many instances of MaxListenersExceededWarning when using Undici via Elasticsearch

[daveyarwood]

Bug Description

This is a follow-up to #1711. @KhafraDev asked that I file a separate issue because Elasticsearch does not appear to be using fetch, which is what was fixed in that issue (via #2823).

I've been running into what I think is the same or similar issue to what was reported in #1711 - lots of MaxListenersExceededWarnings being logged, and my service eventually runs out of memory and dies. When I run with --trace-warnings, the stacktrace points to Undici:

(node:14480) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 101 abort listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit
    at _addListener (node:events:588:17)
    at EventEmitter.addListener (node:events:606:10)
    at addAbortListener (/Users/daveyarwood/code/spark/node_modules/undici/lib/core/util.js:443:10)
    at addSignal (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/abort-signal.js:36:3)
    at new RequestHandler (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:66:5)
    at Pool.request (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:171:25)
    at /Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:164:15
    at new Promise (<anonymous>)
    at Pool.request (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:163:12)

This is with Undici 6.13.0, which is a transitive dependency of @elastic/transport 8.5.0.

Reproducible By

I would be happy to provide a minimal repro case if I had any idea how to do that! :)

Please let me know if you have any ideas of things I can test in my setup.

Expected Behavior

• No MaxListenersExceededWarning messages should be logged.
• In a deployed environment, the memory graphs should not show available memory declining significantly, and the machine shouldn't run out of memory and die.

Environment

Local

2021 Macbook Pro M1 Max
macOS Sonoma 14.11
Node 20.9.0

I'm seeing the MaxListenersExceededWarnings when I run my service locally.

Deployed

My service is also running on a Linux server, in a Docker container with Node 20. I'm also seeing the MaxListenersExceededWarnings logged there, and the memory graph shows a significant decline in memory usage from the time the instance spins up to when it eventually runs out of memory and dies.

Additional context

I don't fully understand what @KhafraDev meant by Elasticsearch not using fetch. I looked through the @elastic/transport code base (on the v8.5.0 tag, which is the version I'm using) and I found UndiciConnection.ts, which appears to be where it's creating an Undici pool and calling request on it, but beyond that, I'm not sure what to look for.

[daveyarwood]

It's quite possible that there is some other reason that my service is running out of memory and dying. But at any rate, it would be good for us to get to the bottom of the MaxListenersExceededWarnings, which (as far as I can tell, from my limited understanding) appear to suggest a memory leak somewhere in Undici.

[ronag]

I believe I open end an issue over at elasticsearch a long time ago. They don't properly cleanup signals.

[ronag]

elastic/elasticsearch-js#1716

[Uzlopak]

Just one remark: when running the wpt suite for fetch, i see the MaxListenersExceededWarning emitted too.

[JoshMock]

elastic/elasticsearch-js#1716

Thanks for pointing this out @ronag. Seems I didn't address your comment there when I initially thought the problem was resolved in elasticsearch 8.8.1.

[sibelius]

is this fixed?

[JoshMock]

is this fixed?

Not in the Elasticsearch client, yet. I have to run down what @ronag suggested and establish the best way to fix that without introducing any breaking changes.

Updates will be shared on elastic/elastic-transport-js#63.