Connection prematurely closed BEFORE response

[martinkrc]

We're experiencing the "Connections prematurely closed" error frequently in production - most of communication succeeds but some portion of requests fails with the mentioned error.

Expected Behavior

When a connection is successfully used to exchange request/response with the server, Reactor/Netty should not cause failures.

Actual Behavior

As the situation appears to me, sometimes the server side initiates connection closing somewhere around the moment when the client acquires the pooled channel connection, but the connection is still successfully (?) used for request+response and closed at the end, however Reactor/Netty produces "Connection prematurely closed BEFORE response".

Observing from the logs, this is the rough process:

• Client creates a new connection with the server
• Client sends an HTTP request
• Client receives a response
• Client releases the channel (for later usage)
• Client acquires the channel for a new request
• (as per tcpdump) Client receives FIN segment from the server, meaning that the server closed the connection
• Client sends an HTTP request via the channel
• Client receives a response
• (as per tcpdump) Client closes its side of the connection, sending FIN segment to the server
• Client fails with "Connection prematurely closed BEFORE response"

Connection log:

TCP dump:

Steps to Reproduce

We're unable to reproduce the situation in our test environment, it happens purely in production. However, I believe that the attached log and tcpdump will help with issue resolution from Reactor/Netty side.

This is how we create the Spring WebClient used for communication:

        SslContext sslContext = NettySslContext.initialize(/*...*/);
        HttpClient httpClient = HttpClient
                .create(ConnectionProvider.elastic(
                        "abcd", Duration.of(5000, ChronoUnit.MILLIS), Duration.of(5000, ChronoUnit.MILLIS))
                )
                .keepAlive(true)
                .secure(spec -> spec.sslContext(sslContext))
                .tcpConfiguration(tcpClient -> tcpClient.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 15000)
        );

        return WebClient.builder()
                .clientConnector(new ReactorClientHttpConnector(httpClient))
                .baseUrl("https://some.host.cz:4001")
                .exchangeStrategies(exchangeStrategies)
                .build();

Your Environment

• Spring Boot 2.2.7.RELEASE
• Spring Webflux 5.2.6.RELEASE
• Reactor Netty 0.9.7.RELEASE
• Reactor Core 3.3.5.RELEASE
• Netty 4.1.49.Final
• Java SE 1.8.0_161-b12
• Linux

[violetagg]

@martinkrc Is it possible to try the latest releases?

[martinkrc]

@martinkrc Is it possible to try the latest releases?

Do you recommend any specific version? The trouble is that we already did an upgrade (from 0.8.8 to 0.9.7) recently hoping for improvements, but it didn't help and we don't want to waste release effort - the issue can't be reproduced in our test env. Do you see any ticket post 0.9.7 which could be fixing the problem?

[violetagg]

@martinkrc
I'm thinking about this
#1165
#1183
Although the root cause might be different in your case

The current releases are Reactor Netty 0.9.11, Spring Boot 2.3.3, Spring Framework 5.2.8

[violetagg]

@martinkrc One additional question: Is there any kind of timeout on the target server?
keep alive timeout/idle timeout etc. that closes the connection when it was not used for a particular time?

(For example if you have Tomcat as a target server with connectionTimeout=5s this will automatically configure keepAliveTimeout=5s by default)

From the TCP dump I see that the target server might have such timeout (5s?)

If that's the case configure the pool with max idle time below 5s, also you may want to switch the pool from FIFO to LIFO leasing strategy so that you will use the most recently used connection. (we have fixes related to FIFO/LIFO leasing in 0.9.11)

[violetagg]

@martinkrc Were you able to try the latest versions?

[martinkrc]

@violetagg Need more time, I managed to reproduce the issue somehow locally, but not in a reliable way yet. Next steps would be trying a newer version, but don't have capacity for it at the moment.
Meanwhile we're trying to find out the answer for your additional question - about timeout on the target server.

[violetagg]

@martinkrc I'm closing this, if you have additional information we can reopen it.