AbstractEpollStreamChannel$EpollStreamUnsafe MemLeak

[Eini-AXA]

We got the following error message after running our application for some time.

Actual behavior

LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
	io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
	io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
	io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:834)

Possible Cause

netty/transport-native-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollStreamChannel.java

Lines 761 to 812 in 88319d4

	do {
	if (sQueue != null || (sQueue = spliceQueue) != null) {
	SpliceInTask spliceTask = sQueue.peek();
	if (spliceTask != null) {
	if (spliceTask.spliceIn(allocHandle)) {
	// We need to check if it is still active as if not we removed all SpliceTasks in
	// doClose(...)
	if (isActive()) {
	sQueue.remove();
	}
	continue;
	} else {
	break;
	}
	}
	}
	// we use a direct buffer here as the native implementations only be able
	// to handle direct buffers.
	byteBuf = allocHandle.allocate(allocator);
	allocHandle.lastBytesRead(doReadBytes(byteBuf));
	if (allocHandle.lastBytesRead() <= 0) {
	// nothing was read, release the buffer.
	byteBuf.release();
	byteBuf = null;
	close = allocHandle.lastBytesRead() < 0;
	if (close) {
	// There is nothing left to read as we received an EOF.
	readPending = false;
	}
	break;
	}
	allocHandle.incMessagesRead(1);
	readPending = false;
	pipeline.fireChannelRead(byteBuf);
	byteBuf = null;
	if (shouldBreakEpollInReady(config)) {
	// We need to do this for two reasons:
	//
	// - If the input was shutdown in between (which may be the case when the user did it in the
	// fireChannelRead(...) method we should not try to read again to not produce any
	// miss-leading exceptions.
	//
	// - If the user closes the channel we need to ensure we not try to read from it again as
	// the filedescriptor may be re-used already by the OS if the system is handling a lot of
	// concurrent connections and so needs a lot of filedescriptors. If not do this we risk
	// reading data from a filedescriptor that belongs to another socket then the socket that
	// was "wrapped" by this Channel implementation.
	break;
	}
	} while (allocHandle.continueReading());

The byte buffer is set to null on line 796, but it is never released. And in each iteration we create a new byte buffer.

Netty version

4.1.67 Final

JVM version (e.g. java -version)

openjdk version "11.0.5" 2019-10-15 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.5+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode, sharing)

OS version (e.g. uname -a)

Linux 3.10.0-1160.36.2.el7.x86_64 #1 SMP Thu Jul 8 02:53:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[chrisvest]

In the "Possible Cause" snippet, the pipeline.fireChannelRead(byteBut) call takes on responsibility of releasing the buffer.

[NiteshKant]

Converting to GH discussion

[pravussum]

I face the same issue - byteBuf is set to null but still has a reference count > 0 after the call to fireChannelRead(), eventually leading to memory leaks being reported.
Even knowing the fireChannelRead() call takes responsibility to release the buffer, I'm at a loss how to continue investigation here... Would be good to have the information of who touched the buffer in which order.

Netty version is 4.1.68.Final
This happens in a Spring boot Webclient asynchronously streaming the response of a long-running request using backpressure.

[normanmaurer]

@pravussum the handler in the ChannelPipeline that "consumes" the buffer is responsible to call release()

[metolinamy]

@pravussum the handler in the ChannelPipeline that "consumes" the buffer is responsible to call release()

Help

[Eini-AXA]

We will check our InputHandlers if one of them does not release the buffer correctly.

[metolinamy]

We will check our InputHandlers if one of them does not release the buffer correctly.