SSLEngine closed already in reactor-netty http client

[kushagraThapar]

Describe the bug
SSLEngine closed already gets thrown by reactor-netty http client when running ReadMyWrites in Strong Https mode

Exception or Stack Trace
2020-01-10 20:16:08,322 [reactor-http-epoll-4] WARN reactor.netty.http.client.HttpClientConnect - [id: 0x393d7ffd, L:/10.2.0.4:46496 - R:cdb-ms-prod-westus2-fd25.documents.azure.com/13.66.138.15:14939] The connection observed an error
javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:834)
at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:797)
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:778)
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:1883)
at io.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1851)
at io.netty.handler.ssl.SslHandler.close(SslHandler.java:729)
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:621)
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:605)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.close(CombinedChannelDuplexHandler.java:507)
at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:77)
at io.netty.channel.CombinedChannelDuplexHandler.close(CombinedChannelDuplexHandler.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:621)
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:605)
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:467)
at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:969)
at io.netty.channel.AbstractChannel.close(AbstractChannel.java:231)
at reactor.netty.http.client.HttpClientOperations.onInboundCancel(HttpClientOperations.java:241)
at reactor.netty.channel.FluxReceive.unsubscribeReceiver(FluxReceive.java:405)
at reactor.netty.channel.FluxReceive.lambda$new$0(FluxReceive.java:71)
at reactor.netty.channel.FluxReceive.cancelReceiver(FluxReceive.java:134)
at reactor.netty.channel.FluxReceive.cancel(FluxReceive.java:81)
at reactor.netty.channel.ChannelOperations.dispose(ChannelOperations.java:160)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.onCancel(MonoCreate.java:220)
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$TcpClientSubscriber.onNext(HttpClientConnect.java:340)
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$TcpClientSubscriber.onNext(HttpClientConnect.java:325)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156)
at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.run(PooledConnectionProvider.java:564)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

[kushagraThapar]

Reactor-netty has this issue still open. Waiting for the fix
reactor/reactor-netty#782

[kushagraThapar]

Fix has been done from reactor netty side and is supposed to be released in 0.9.7.RELEASE on 27th April.
Once it is out, we can update this on cosmos side.

[kushagraThapar]

Last checked with reactor-netty team, our issue is a little different from the general fix they have released. Our issue is related to operators cancelling the on-going request, which is causing the SSLEngine closed already exception.
More details here : reactor/reactor-netty#782 (comment)

[kushagraThapar]

This issue might be causing OOM, as observed in one of the CIs.

[kushagraThapar]

We have opened a new issue on reactor-netty github : reactor/reactor-netty#1165

[kushagraThapar]

reactor-netty has fixed the issue, and the fix will be released as part of 0.9.10.RELEASE.
Once it is out, we will update the SDK to new version and will push a new release.

[kushagraThapar]

This issue has been fixed and will be released in 4.2.1 release soon.