OOM in ReferenceCountedOpenSslEngine due to suppressed exceptions accumulating in pendingException

[tkowalcz]

ReferenceCountedOpenSslEngine has a pendingException field, it's purpose seems to be to store exceptions, do some cleanup and rethrow them later at a more convenient time. There is also code that appends subsequent exceptions to pendingException as suppressed:

ThrowableUtil.addSuppressed(pendingException, exception);

In my particular case this led to OOM error where ReferenceCountedOpenSslEngine occupied almost 8gb of heap with 12 million of suppressed exceptions.

Screenshot from Eclipse MAT:

Netty version where this occurred is 4.1.53.Final. It does not look like handling these exceptions is different compared to master. We also used netty-tcnative-boringssl-static in version 2.0.46.Final.

What might be interesting is that the underlying SSL error is TOO_MANY_KEY_UPDATES. I found this report similar but without OOM. Here is the BoringSSL code reporting this error: https://github.com/google/boringssl/blob/bcecc7d834fc44ad257b2f23f88e1cf597ab2736/ssl/tls13_both.cc#L677.

I have trouble reproducing TOO_MANY_KEY_UPDATES error. I will appreciate any pointers on how to trigger that, then I'll try to make a reproducer based on master branch.

[normanmaurer]

@tkowalcz converted to an issue... Also see #13351