Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SSLException on request cancellation #1165

Closed
kushagraThapar opened this issue Jun 18, 2020 · 17 comments · Fixed by #1198
Closed

SSLException on request cancellation #1165

kushagraThapar opened this issue Jun 18, 2020 · 17 comments · Fixed by #1198
Labels
type/bug A general bug
Milestone
0.9.10.RELEASE

Comments

@kushagraThapar
Copy link

In our SDK we are seeing javax.net.ssl.SSLException: SSLEngine closed already exceptions.
We are using fixed channel pool - ConnectionProvider with 1000 as pool size.

Expected Behavior

On cancel operation, channel should be closed / disposed quietly since the operation has been cancelled. Whether that channel is reusable or not can be a point of discussion.

Actual Behavior

Combination of pagination + takeUntil operator causes SSLException. In high load, the error keeps happening, and thus causing Out of Memory issue.

Steps to Reproduce

I was able to reproduce the issue on an isolated project - which is exactly same we have reactor-netty http client in Azure CosmosDB Java SDK.

Here is the link to the project I have created : https://github.com/kushagraThapar/reactor-netty-ssl-error

Run the MainApplication.java in IDE for couple of minutes and the error starts happening.

For testing purpose, I have kept the connection pool size to 5.

Your Environment

macOS / linux machines
reactor-netty: 0.9.7.RELEASE
reactor-core: 3.3.5.RELEASE

Here is the complete stack trace:

18 Jun 2020 16:04:31,770       [reactor-http-nio-12] WARN reactor.netty.http.client.HttpClientConnect - [id: 0xbb8e3529, L:/192.168.0.10:54545 - R:rel.ink/104.24.122.41:443] The connection observed an error
javax.net.ssl.SSLException: SSLEngine closed already
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:848)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Assembly trace from producer [reactor.core.publisher.MonoIgnoreThen] :
	reactor.core.publisher.Mono.thenEmpty
	reactor.netty.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:553)
Error has been observed at the following site(s):
	|_  Mono.thenEmpty ⇢ at reactor.netty.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:553)
	|_                 ⇢ at reactor.netty.ReactorNetty$OutboundThen.then(ReactorNetty.java:598)
	|_ Mono.fromDirect ⇢ at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.lambda$onStateChange$0(HttpClientConnect.java:437)
	|_      Mono.defer ⇢ at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:437)
Stack trace:
		at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:848)
		at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:811)
		at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:792)
		at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:1931)
		at io.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1899)
		at io.netty.handler.ssl.SslHandler.close(SslHandler.java:743)
		at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:622)
		at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:606)
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.close(CombinedChannelDuplexHandler.java:505)
		at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:77)
		at io.netty.channel.CombinedChannelDuplexHandler.close(CombinedChannelDuplexHandler.java:316)
		at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:622)
		at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:606)
		at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472)
		at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:957)
		at io.netty.channel.AbstractChannel.close(AbstractChannel.java:232)
		at reactor.netty.http.client.HttpClientOperations.onInboundCancel(HttpClientOperations.java:258)
		at reactor.netty.channel.FluxReceive.unsubscribeReceiver(FluxReceive.java:436)
		at reactor.netty.channel.FluxReceive.lambda$new$0(FluxReceive.java:80)
		at reactor.netty.channel.FluxReceive.cancelReceiver(FluxReceive.java:148)
		at reactor.netty.channel.FluxReceive.cancel(FluxReceive.java:90)
		at reactor.netty.channel.ChannelOperations.dispose(ChannelOperations.java:160)
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.onCancel(MonoCreate.java:222)
		at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$TcpClientSubscriber.onNext(HttpClientConnect.java:344)
		at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$TcpClientSubscriber.onNext(HttpClientConnect.java:329)
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156)
		at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.run(PooledConnectionProvider.java:604)
		at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onNext(PooledConnectionProvider.java:485)
		at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onNext(PooledConnectionProvider.java:436)
		at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.deliver(AbstractPool.java:388)
		at reactor.netty.internal.shaded.reactor.pool.SimplePool.lambda$drainLoop$13(SimplePool.java:244)
		at reactor.core.scheduler.ImmediateScheduler.schedule(ImmediateScheduler.java:47)
		at reactor.netty.internal.shaded.reactor.pool.SimplePool.drainLoop(SimplePool.java:244)
		at reactor.netty.internal.shaded.reactor.pool.SimplePool.drain(SimplePool.java:172)
		at reactor.netty.internal.shaded.reactor.pool.SimplePool.maybeRecycleAndDrain(SimplePool.java:159)
		at reactor.netty.internal.shaded.reactor.pool.SimplePool$QueuePoolRecyclerInner.onComplete(SimplePool.java:401)
		at reactor.core.publisher.Operators.complete(Operators.java:135)
		at reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:45)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4218)
		at reactor.netty.internal.shaded.reactor.pool.SimplePool$QueuePoolRecyclerMono.subscribe(SimplePool.java:452)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4218)
		at reactor.core.publisher.Mono.subscribeWith(Mono.java:4329)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4189)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4125)
		at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:401)
		at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:426)
		at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:607)
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
		at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)
		at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267)
		at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:834)
@kushagraThapar kushagraThapar added status/need-triage A new issue that still need to be evaluated as a whole type/bug A general bug labels Jun 18, 2020
This was referenced Jun 18, 2020
Closed
Closed
@violetagg violetagg removed the status/need-triage A new issue that still need to be evaluated as a whole label Jul 1, 2020
@violetagg
Copy link
Member

@kushagraThapar From the reproducible example I do not see how this SSLEngine closed already can cause an OOM.
The exception is thrown right after acquiring from the pool and before sending any request. However I see that you use responseConnection and I do not see how you handle cancellations. Recently there was similar issue with Spring WebClient, you might want to look at it spring-projects/spring-framework#25216
Meanwhile I'm looking at SSLEngine closed already.

@kushagraThapar
Copy link
Author

@violetagg Thank you for looking at the issue.
OOM doesn't happen directly or soon, but after a very long running operation under heavy work load. We use 1000 as the number of connections for the fixed Connection pool.

Regarding cancellations, what should be the recommended way to handle cancellations. I think that is one part of the problem, or in fact could be the major issue here.

@violetagg
Copy link
Member

@kushagraThapar You do not subscribe for the inbound but you "wrap" the connection that Reactor Netty provides.
https://github.com/kushagraThapar/reactor-netty-ssl-error/blob/1d87eac3e0646160de0d62fcc689e68b2dc8cdc6/src/main/java/com/example/ReactorNettyClient.java#L99
If a cancellation is received before you are able to subscribe for the body, the cancellation is not propagated to Reactor Netty and thus we are not able to clean the memory.
When you use responseConnection you are responsible to clean the memory when a cancellation/error happens.
For the other methods that we expose response, responseSingle you do not have problems because we subscribe immediately to the inbound and we handle the memory cleanup. See here

reactor-netty/src/main/java/reactor/netty/http/client/HttpClientFinalizer.java

Lines 95 to 97 in 10c7f70

public <V> Flux<V> response(BiFunction<? super HttpClientResponse, ? super ByteBufFlux, ? extends Publisher<V>> receiver) {
return _connect().flatMapMany(resp -> Flux.from(receiver.apply(resp, resp.receive()))
.doFinally(s -> discard(resp)));
and here

reactor-netty/src/main/java/reactor/netty/http/client/HttpClientFinalizer.java

Lines 119 to 121 in 10c7f70

public <V> Mono<V> responseSingle(BiFunction<? super HttpClientResponse, ? super ByteBufMono, ? extends Mono<V>> receiver) {
return _connect().flatMap(resp -> receiver.apply(resp, resp.receive().aggregate())
.doFinally(s -> discard(resp)));

In the issue above that I mentioned the solution that was applied to WebClient is when a cancellation is received, they consume/drain the inbound.
spring-projects/spring-framework@21d0696

@kushagraThapar
Copy link
Author

@violetagg - Thank you for these wonderful insights and the explanation, it makes sense.
I updated my example with the same logic that is used in spring-projects/spring-framework@21d0696

I am still facing the issue, though one interesting thing I noticed is - for my scenario, the cancellation is happening before the response comes on the http client, because of which I am not able to consume/drain the inbound.

I added more logging to verify this. If you take a look or run the example, you will notice that this atomicReference always logs null: https://github.com/kushagraThapar/reactor-netty-ssl-error/blob/master/src/main/java/com/example/ReactorNettyClient.java#L82

@violetagg
Copy link
Member

@kushagraThapar Yes as I wrote with this concrete example I do not see how this SSLEngine closed already can cause an OOM.
The issue with responseConnection, cancellation and the possibility of not draining the incoming data is something that I saw while reviewing the example.

@kushagraThapar
Copy link
Author

@violetagg - I agree and it is possible that SSLEngine closed already might not be causing OOM directly, but rather could a be side effect of this issue.
Since this exception happens on all connections eventually (1000 connections in the fixed connection pool), causing the logger to print too many exception stack traces can also lead to using too much heap memory and thus causing the OOM.

Though at this point, our main concern is to get this issue fixed, since OOM happens rarely and it can be worked around by increasing the heap size for now.

violetagg added a commit that referenced this issue Jul 6, 2020
@violetagg
fix #1165 Change the state after notifying that the connection is acq…
d9ad6f0 
…uired successfully.
@violetagg violetagg mentioned this issue Jul 6, 2020
Merged
@violetagg violetagg linked a pull request Jul 6, 2020 that will close this issue
fix #1165 Change the state after notifying that the connection is acquired successfully. #1198
Merged
@violetagg
Copy link
Member

@kushagraThapar Will you be able to test PR #1198 ?

@violetagg violetagg added this to the 0.9.10.RELEASE milestone Jul 6, 2020
violetagg added a commit that referenced this issue Jul 6, 2020
@violetagg
fix #1165 Change the state after notifying that the connection is acq…
7500c49 
…uired successfully.
@kushagraThapar
Copy link
Author

@violetagg - Thanks for the PR, I can definitely try it out. Will post the update once I am able to test it, hopefully today.

@kushagraThapar
Copy link
Author

@violetagg - I was able to test the PR and it definitely fixes the issue in the example test application, thank you so much for that.

However, in my SDK, I am still seeing the issue with the fix PR version. Here are some interesting observations from my debugging today in SDK code.

  1. When calling doOnCancel() on http client - the response was always already subscribed before cancellation happens, so the current state was always 1 based on below code.
// 0 - not subscribed, 1 - subscribed, 2 - cancelled, 3 - cancelled via connector (before subscribe)
private final AtomicInteger state = new AtomicInteger(0);
  1. Even if I change the releaseAfterCancel method logic to compare state to 1, I still get the error.
/**
         * Called by {@link ReactorNettyClient} when a cancellation is detected
         * but the content has not been subscribed to. If the subscription never
         * materializes then the content will remain not drained. Or it could still
         * materialize if the cancellation happened very early, or the response
         * reading was delayed for some reason.
         */
        private void releaseAfterCancel(HttpMethod method) {
            boolean mayHaveBody = mayHaveBody(method);
            logger.info("May have body : {}, current state : {}, to be new state: {}", mayHaveBody, this.state.get(), 3);
            boolean newState = this.state.compareAndSet(1, 3);
            if (mayHaveBody && newState) {
                logger.info("Releasing body, not yet subscribed");
                if (logger.isDebugEnabled()) {
                    logger.debug("Releasing body, not yet subscribed");
                }
                this.bodyIntern()
                    .doOnNext(byteBuf -> {})
                    .subscribe(byteBuf -> {}, ex -> {});
            }
        }
  1. That being said, the number of SSLExceptions that I used to get earlier have reduced by a big number in your PR fix, which I am guessing is fixing most of the cancellation cases, but may be few edge cases are still remaining.

Here is the latests stack trace that I am getting now, not sure if this is same as the above one, but just wanted to provide this.

2020-07-06 17:04:03,711       [reactor-http-nio-5] WARN  reactor.netty.http.client.HttpClientConnect - [id: 0x3de7e1e9, L:/192.168.0.10:62268 - R:paged-flux-test-account-westus2.documents.azure.com/40.78.250.5:443] The connection observed an error
javax.net.ssl.SSLException: SSLEngine closed already
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:848) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Assembly trace from producer [reactor.core.publisher.MonoIgnoreThen] :
	reactor.core.publisher.Mono.thenEmpty
	reactor.netty.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:553)
Error has been observed at the following site(s):
	|_  Mono.thenEmpty ⇢ at reactor.netty.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:553)
	|_                 ⇢ at reactor.netty.ReactorNetty$OutboundThen.then(ReactorNetty.java:598)
	|_ Mono.fromDirect ⇢ at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.lambda$onStateChange$0(HttpClientConnect.java:440)
	|_      Mono.defer ⇢ at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:440)
Stack trace:
		at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:848) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:811) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:792) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:1931) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1899) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.ssl.SslHandler.close(SslHandler.java:743) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:622) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:606) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.close(CombinedChannelDuplexHandler.java:505) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:77) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.CombinedChannelDuplexHandler.close(CombinedChannelDuplexHandler.java:316) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:622) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:606) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:957) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannel.close(AbstractChannel.java:232) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at reactor.netty.http.client.HttpClientOperations.onInboundCancel(HttpClientOperations.java:260) ~[reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.channel.FluxReceive.unsubscribeReceiver(FluxReceive.java:448) ~[reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.channel.FluxReceive.lambda$new$0(FluxReceive.java:88) ~[reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.channel.FluxReceive.cancelReceiver(FluxReceive.java:187) ~[reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.channel.FluxReceive.cancel(FluxReceive.java:98) ~[reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.channel.ChannelOperations.dispose(ChannelOperations.java:165) ~[reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.core.publisher.FluxCreate$SinkDisposable.cancel(FluxCreate.java:1039) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.disposeResource(MonoCreate.java:307) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.cancel(MonoCreate.java:296) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2182) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:1963) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.cancel(FluxRetryWhen.java:151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.SerializedSubscriber.cancel(SerializedSubscriber.java:157) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.cancel(MonoFlatMapMany.java:123) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.cancel(FluxPeekFuseable.java:152) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoSingle$SingleSubscriber.cancel(MonoSingle.java:101) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators.terminate(Operators.java:1199) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:180) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:167) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2182) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:1963) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoSingle$SingleSubscriber.cancel(MonoSingle.java:101) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2182) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:1963) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.cancel(FluxRetryWhen.java:151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.SerializedSubscriber.cancel(SerializedSubscriber.java:157) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2182) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:1963) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:167) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:167) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoSingle$SingleSubscriber.cancel(MonoSingle.java:101) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:167) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2182) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:1963) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.cancel(FluxRetryWhen.java:151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.SerializedSubscriber.cancel(SerializedSubscriber.java:157) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:167) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2182) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:1963) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxExpand$ExpandBreathSubscriber.cancel(FluxExpand.java:146) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.cancel(FluxPublishOn.java:276) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:167) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.cancel(FluxMapFuseable.java:167) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2182) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2151) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:1963) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators.set(Operators.java:1119) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMergeSequential$MergeSequentialInner.cancel(FluxMergeSequential.java:593) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMergeSequential$MergeSequentialMain.cancelAll(FluxMergeSequential.java:278) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMergeSequential$MergeSequentialMain.drain(FluxMergeSequential.java:378) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMergeSequential$MergeSequentialMain.innerComplete(FluxMergeSequential.java:321) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMergeSequential$MergeSequentialInner.onComplete(FluxMergeSequential.java:576) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:1989) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.doComplete(FluxPublishOn.java:504) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.checkTerminated(FluxPublishOn.java:551) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:431) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:526) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.scheduler.ImmediateScheduler$ImmediateSchedulerWorker.schedule(ImmediateScheduler.java:79) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.trySchedule(FluxPublishOn.java:311) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.onComplete(FluxPublishOn.java:257) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxExpand$ExpandBreathSubscriber.drainQueue(FluxExpand.java:162) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxExpand$ExpandBreathSubscriber.onComplete(FluxExpand.java:141) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.SerializedSubscriber.onComplete(SerializedSubscriber.java:146) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onComplete(FluxRetryWhen.java:188) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1756) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:1989) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.SerializedSubscriber.onComplete(SerializedSubscriber.java:146) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onComplete(FluxRetryWhen.java:188) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1756) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:1989) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1756) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1755) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:78) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:270) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:270) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onComplete(FluxHandleFuseable.java:223) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onComplete(FluxMapFuseable.java:336) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onComplete(FluxHandleFuseable.java:223) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onComplete(FluxContextStart.java:115) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1756) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:121) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:270) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.3.5.RELEASE.jar:3.3.5.RELEASE]
		at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:378) [reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:372) [reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:428) [reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:617) [reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96) [reactor-netty-0.9.10.BUILD-SNAPSHOT.jar:0.9.10.BUILD-SNAPSHOT]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.49.Final.jar:4.1.49.Final]
		at java.lang.Thread.run(Thread.java:834) [?:?]

@violetagg
Copy link
Member

@kushagraThapar Thanks. I committed an additional fix to the PR, if you can test it, it will be great!

@kushagraThapar
Copy link
Author

Definitely, let me test it out .. will update you with the results.

@kushagraThapar
Copy link
Author

@violetagg - for some reason I am not able to build the latest changes in the PR. I tried multiple times, and the build is not completing..

Here are the commands I tried :

./gradlew build
AND
./gradlew build -x test

I am seeing a lot of test failures as well , which I don't remember seeing last time (but I could be wrong)

Starting a Gradle Daemon (subsequent builds will be faster)

> Task :downloadBaseline UP-TO-DATE
Will download and perform baseline comparison with 0.9.9.RELEASE

> Task :shadedJarTest

reactor.netty.tcp.TcpServerTests > testGracefulShutdown FAILED
    java.lang.AssertionError at TcpServerTests.java:929

reactor.netty.tcp.TcpServerTests > testChannelGroupClosesAllConnections FAILED
    java.lang.AssertionError at TcpServerTests.java:841

reactor.netty.tcp.TcpServerTests > testIssue688 FAILED
    java.lang.AssertionError at TcpServerTests.java:883

reactor.netty.tcp.TcpClientTests > testIssue585_1 FAILED
    reactor.core.Exceptions$ReactiveException at TcpClientTests.java:930
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.tcp.TcpClientTests > testIssue585_2 FAILED
    reactor.core.Exceptions$ReactiveException at TcpClientTests.java:988
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.tcp.TcpClientTests > testIssue600_1 FAILED
    java.lang.AssertionError at TcpClientTests.java:820

reactor.netty.tcp.TcpClientTests > testIssue600_2 FAILED
    java.lang.AssertionError at TcpClientTests.java:820

reactor.netty.http.HttpCompressionClientServerTests > serverCompressionPredicateFalse FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:292
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > serverCompressionDisabled FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:124
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > compressionServerEnabledClientDisabledIsNone FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:375
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > serverCompressionDefault FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:90
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > trueEnabledIncludeContentEncoding FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:64
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > compressionServerDefaultClientDefaultIsNone FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:406
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > serverCompressionPredicateTrue FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:245
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > serverCompressionEnabledBigResponse FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:330
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > serverCompressionEnabledSmallResponse FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:205
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > compressionActivatedOnClientAddsHeader FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:436
        Caused by: io.netty.channel.ConnectTimeoutException

reactor.netty.http.HttpCompressionClientServerTests > serverCompressionAlwaysEnabled FAILED
    reactor.core.Exceptions$ReactiveException at HttpCompressionClientServerTests.java:159
        Caused by: io.netty.channel.ConnectTimeoutException
<===========--> 87% EXECUTING [10m 0s]

reactor.netty.http.server.HttpServerTests > testGracefulShutdown FAILEDests
    org.junit.ComparisonFailure at HttpServerTests.java:1710

reactor.netty.http.server.HttpServerTests > testCustomHandlerInvokedBeforeIOHandler FAILED
    java.lang.AssertionError at HttpServerTests.java:1104

reactor.netty.http.server.HttpServerTests > testExpectErrorWhenConnectionClosed FAILED
    java.lang.AssertionError

reactor.netty.http.server.HttpServerTests > contextShouldBeTransferredFromDownStreamToUpStream FAILED
    java.lang.AssertionError at HttpServerTests.java:810

reactor.netty.http.server.HttpServerTests > testIssue186 FAILED
    java.lang.AssertionError at HttpServerTests.java:746

Please let me know if there is a faster way to build the library locally ?

@violetagg
Copy link
Member

@kushagraThapar can you try just ./gradlew publishToMavenLocal this will just build the artefact and publish it to the local maven repo

@kushagraThapar
Copy link
Author

@violetagg - I verified and this fix works well :)
I am not seeing the issue anymore at all.

Thank you so much for your efforts and fixing this issue.

Let me know once this gets released, I will update the azure java sdks to the released version.

@violetagg
Copy link
Member

@kushagraThapar Thanks for testing the fix. It will be available with next release 0.9.10.

@kushagraThapar
Copy link
Author

@violetagg - can you please let me know the tentative release date for 0.9.10.RELEASE ?
Also, once this is out, do we need to upgrade reactor-core versions as well (as part of compatible changes) ?

@violetagg
Copy link
Member

@kushagraThapar the current plan is for 20.07. We will release both reactor core + netty.

@kushagraThapar kushagraThapar mentioned this issue Jul 21, 2020
Merged
@kushagraThapar kushagraThapar mentioned this issue Jul 24, 2020
Merged
@violetagg violetagg mentioned this issue Sep 1, 2020
Closed
@FabianMeiswinkel FabianMeiswinkel mentioned this issue Oct 7, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug A general bug
Projects
None yet
Milestone
0.9.10.RELEASE
2 participants
@violetagg @kushagraThapar