Cannot reserve XY bytes of direct buffer memory

[Zelldon]

Describe the bug
We seeing errors "Out of memory, exiting now because we can't recover from OOM. Consider adjusting memory limits." on prod.

Error: https://console.cloud.google.com/errors/detail/CKSrwcbo6qihLg;service=zeebe;time=P7D?project=camunda-cloud-240911

The cluster seems to be completely borked. Impact as high since it looks like that cluster is unusable.

To Reproduce
I have no clue yet, it seems to be an election loop. You can see that in the metrics but also in the logs where the term is really high.

Transition to FOLLOWER on term 353 - transitioning StreamProcessor

Expected behavior

Log/Stacktrace

Full Stacktrace

java.lang.OutOfMemoryError: Cannot reserve 75498884 bytes of direct buffer memory (allocated: 2003030408, limit: 2075918336)
	at java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]
	at java.nio.DirectByteBuffer.<init>(Unknown Source) ~[?:?]
	at java.nio.ByteBuffer.allocateDirect(Unknown Source) ~[?:?]
	at io.camunda.zeebe.util.allocation.DirectBufferAllocator.allocate(DirectBufferAllocator.java:20) ~[zeebe-util-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.util.allocation.BufferAllocators.allocateDirect(BufferAllocators.java:16) ~[zeebe-util-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.dispatcher.DispatcherBuilder.initAllocatedBuffer(DispatcherBuilder.java:147) ~[zeebe-dispatcher-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.dispatcher.DispatcherBuilder.build(DispatcherBuilder.java:93) ~[zeebe-dispatcher-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.logstreams.impl.log.LogStreamImpl.createAndScheduleWriteBuffer(LogStreamImpl.java:340) ~[zeebe-logstreams-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.logstreams.impl.log.LogStreamImpl.lambda$openAppender$9(LogStreamImpl.java:297) ~[zeebe-logstreams-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.util.sched.ActorJob.invoke(ActorJob.java:79) ~[zeebe-util-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.util.sched.ActorJob.execute(ActorJob.java:44) ~[zeebe-util-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.util.sched.ActorTask.execute(ActorTask.java:122) ~[zeebe-util-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:97) ~[zeebe-util-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.util.sched.ActorThread.doWork(ActorThread.java:80) ~[zeebe-util-8.0.2.jar:8.0.2]
	at io.camunda.zeebe.util.sched.ActorThread.run(ActorThread.java:189) ~[zeebe-util-8.0.2.jar:8.0.2]

Environment:

• OS: saas
• Zeebe Version: 8.0.2
• Configuration: G3-Trial

[oleschoenburg]

This cluster ran into #9115 a while ago : Logs

[menski]

Closing for now, the cluster is in an unhealthy state, and the error didn't happen in another cluster. We are fixing an issue related to memory leaks, which might be related.