Netty server getting blocked under heavy read/writes

[mayurdb]

In Uber, we are using Netty to handle the shuffle data from the Spark applications (the project is open source: RemoteShuffleService. We have a 300 node Netty cluster handling ~10-15PB, and 20000-30000 concurrent connections per server everyday. We use a 400 thread worker group and a separate boss group. At a very high level, server has an endpoint each for writing/reading data to/from disk. Data is streamed by the clients in chunks of ~4-5 MB

We face intermittent timeout issues on the cluster especially when there are heavy read/writes. After investigation we had few questions:

1. Noob question: A Channel must always be served by EventLoop on which it was originally registered. Why is that? Why not make use of the available EventLoop? We frequently see issues where only certain connections to the sever are dropped whereas the server health overall looks just fine. This would most likely be because of particular Channel blocks an EventLoop and only other Channels getting served by this EventLoop get impacted.

2. If a custom EventLoopGroup is used to handle the first ChannelHandler in the pipeline, all the requests on a Channel would be picked by a EventLoop first in the original workerGroup and then handed over to the custom EventLoopGroup or it will be directly picked by the custom EventLoopGroup?

DefaultFileRegion fileRegion = new DefaultFileRegion(new File(splitFile), 0, fileLength);
ChannelFuture sendFileChannelFuture = 
      ctx.writeAndFlush(fileRegion, ctx.newProgressivePromise() // code within the future);
// code outside of the future

Consider the case above. With some custom logging, I could see that both // code within the future and // code outside of the future get handled by the same EventLoop registered with Channel (again correct me if I'm wrong). // code outside of the future gets executed only after the future completes (there is no sendFileChannelFuture.await() or sync()). Is this expected? If it is, what's the advantage of using a Future here apart from bypassing the buffering.

4. For doing a write to disk, we simply do a sync write once a ChannelHandler has accepted the thread. A little confused here: Is there any way here to improve this? Maybe question above will clarify this a little

Expected behavior

Actual behavior

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

https://github.com/uber/RemoteShuffleService/blob/master/src/main/java/com/uber/rss/handlers/DownloadServerHandler.java#L137

Netty version

4.1.65.Final

JVM version (e.g. java -version)

Java 8

OS version (e.g. uname -a)

[normanmaurer]

These are a lot of different questions... I will try to answer all of them but please in the future create different discussion for each as it make it easer for people to find these etc.

1. This is because it simplifies the whole threading model a lot. By handling everything on the same EventLoop there is no need for any extra synchronisation etc. Also it ensures everything is delivered in the correct order. This model is widely used by network frameworks.

2. I will be handled by the assigned EventLoop of the Channel and then handed over to the EventExecutor that handles the ChannelHandler

3. While the operation might execute directly there is no guarantee that this is the case. The only way how you can be sure it is really executed is by either block on the future or add a listener that is notified once this is the case. Reasons for delayed execution might be for example if the socket buffer is full and so the write can only be executed once the remote peer did read some more data.

4. A write to the disk might block so you might want to offload these writes to another thread to ensure you not block the EventLoop and so the processing of network IO for connections.

[mayurdb]

Thanks @normanmaurer for the answers.

Just one more question, execution within the ctx.writeAndFlush() would be taken up by an EventLoop on which Channel was originally registered even if the ChannelHandler which invoked ctx.writeAndFlush() was run by a custom EventLoopGroup. How do we ensure that such executions be also taken up by a custom EventLoopGroup?

[joosing]

I've been following this conversation and the uber/rss project you linked to with interest. I'm assuming that when you refer to a custom event loop, you're referring to a dedicated event loop that you can register with a handler in your pipeline, like below.

EventLoopGroup customWorkGroup = new NioEventLoopGroup(n);
channel.pipeline().addLast(customWorkGroup, new ServerResponseHandler()); 
... 
public class ServerResponseHandler extends ChannelInboundHandlerAdapter {
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        ...
        DefaultFileRegion fileRegion = new DefaultFileRegion(somefile, 0, somefile.length());
        ctx.writeAndFlush(fileRegion, ctx.newProgressivePromise());
        ...
    }
}

In the above code, when you invoke ctx.writeAndFlush() to send a response from the ChannelHandler which is run by a custom EventLoop, the fileRegion is passed to a ChannelHandlerContext called HeadContext that exists inside the channel pipeline. The Head is the beginning of the pipeline managed by Netty (you can call it the end in the outbound direction) and is always executed by the channel's event loop. In other words, even if you assign a custom event loop to every handler you register, the user's request will be passed from the end of the pipeline to the channel's event loop for execution.

So even if you decide that the file transfer is putting a load on the channel's event loop and you separate it into a custom event loop, the core file transfer load will still be executed by the channel's event loop at the end, so I don't see much benefit in registering a custom event loop. On the other hand, if you're doing a sync write to disk, I think a custom event loop would allow you to avoid blocking and thus avoid the problematic timeout situation.

I hope this works out for you.

[mayurdb]

Awesome. Thanks for the detailed explanation!

In majority of the cases we see that its the control requests which timeout on server end. By control requests here, I mean requests which do not do any disk I/O. This problem is also faced by Netty based server used by Apache Spark to handle shuffle data. Note that the netty server in Apache Spark shuffle service only handles data reads and no data writes (and yet people run into timeout issues). You can find discussion about it here: apache/spark#22173

This could be handled to some extent if there is a way to do a soft reservation in EventLoopGroup for certain kind of requests (control requests in our case). But since the I/O work is always handled by default EventLoop, such requests can always saturate the entire EventLoopGroop.

Any way to tackle this?

[franz1981]

Move the I/O out of the event loop or split it into smaller chunks to guarantee some progress to anyone

[mayurdb]

Thanks everyone for the detailed explanation. Appreciate it :)