UnknownHostException inside of cluster - resolved

[xsmrcek]

Hello,
I have an issue I am not able to figure out.
I have my service deployed inside of k8s cluster and I am calling only services insude of this cluster. I am getting following exception:

org.springframework.web.reactive.function.client.WebClientRequestException: Failed to resolve 'service-XXX.namespace-XXX' after 2 queries ; nested exception is java.net.UnknownHostException: Failed to resolve 'hcd-hi-dms-webapi.gma-hi-dms' after 2 queries 
	at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
	Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Request to GET http://service-XXX.namespace-XXX/endpointXXX [DefaultWebClient]
Original Stack Trace:
		at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
		at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)
		at reactor.core.publisher.Operators.error(Operators.java:198)
		at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onComplete(TracingSubscriber.java:68)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onComplete(TracingSubscriber.java:68)
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:138)
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onSubscribe(MonoIgnoreThen.java:134)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onSubscribe$0(TracingSubscriber.java:53)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onSubscribe(TracingSubscriber.java:53)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onSubscribe(FluxPeek.java:171)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onSubscribe$0(TracingSubscriber.java:53)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onSubscribe(TracingSubscriber.java:53)
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397)
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225)
		at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerError(FluxConcatMap.java:309)
		at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onError(FluxConcatMap.java:875)
		at reactor.core.publisher.Operators.error(Operators.java:198)
		at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:451)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)
		at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:537)
		at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:343)
		at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)
		at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)
		at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:308)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)
		at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:158)
		at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:475)
		at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:431)
		at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)
		at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:548)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
		at reactor.core.publisher.Operators.error(Operators.java:198)
		at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)
		at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
		at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:534)
		at reactor.netty.transport.TransportConnector.lambda$doResolveAndConnect$11(TransportConnector.java:341)
		at io.opentelemetry.javaagent.instrumentation.netty.common.FutureListenerWrappers$WrappedFutureListener.operationComplete(FutureListenerWrappers.java:106)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
		at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
		at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
		at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:109)
		at io.netty.resolver.InetSocketAddressResolver$2.operationComplete(InetSocketAddressResolver.java:86)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
		at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
		at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
		at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
		at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1055)
		at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1000)
		at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418)
		at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:971)
		at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:66)
		at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:471)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
		at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
		at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
		at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
		at io.netty.resolver.dns.DnsQueryContext.trySuccess(DnsQueryContext.java:232)
		at io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:224)
		at io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:1314)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
		at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
		at io.netty.channel.epoll.EpollDatagramChannel.processPacket(EpollDatagramChannel.java:649)
		at io.netty.channel.epoll.EpollDatagramChannel.recvmsg(EpollDatagramChannel.java:684)
		at io.netty.channel.epoll.EpollDatagramChannel.access$300(EpollDatagramChannel.java:55)
		at io.netty.channel.epoll.EpollDatagramChannel$EpollDatagramChannelUnsafe.epollInReady(EpollDatagramChannel.java:523)
		at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.UnknownHostException: Failed to resolve 'service-XXX.namespace-XXX' after 2 queries 
	at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1047)
	at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1000)
	at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418)
	at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:971)
	at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:66)
	at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:471)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at io.netty.resolver.dns.DnsQueryContext.trySuccess(DnsQueryContext.java:232)
	at io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:224)
	at io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:1314)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.epoll.EpollDatagramChannel.processPacket(EpollDatagramChannel.java:649)
	at io.netty.channel.epoll.EpollDatagramChannel.recvmsg(EpollDatagramChannel.java:684)
	at io.netty.channel.epoll.EpollDatagramChannel.access$300(EpollDatagramChannel.java:55)
	at io.netty.channel.epoll.EpollDatagramChannel$EpollDatagramChannelUnsafe.epollInReady(EpollDatagramChannel.java:523)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)

I have no idea why, as it have problem only with one service (it's 5th client request in my flow), every another request before this one is OK and this one fails in like 10% calls.

Did you ever experienced this? Or do you have any idea how to solve this issue?

My technology stack:
spring-boot-starter-parent 2.7.2 (netty 4.1.179.Final) (I am using webflux)
kotlin 1.7.10
spring-cloud 2021.0.3

My WebClient configuration:

@Bean //5s timeout
    fun connectTimeoutCustomizer(properties: DefaultHttpConnectorProperties) =
        ReactorNettyHttpClientMapper { httpClient: HttpClient ->
            properties.timeouts?.connect?.let { connectTimeout ->
                httpClient.tcpConfiguration { it.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout.milliseconds().toInt()) }
            }
        }

    @Bean //10s timeout
    fun readTimeoutCustomizer(properties: DefaultHttpConnectorProperties) =
        timeoutHandler(properties.timeouts?.read) { timeout, unit ->  ReadTimeoutHandler(timeout, unit) }

    @Bean //10s timeout
    fun writeTimeoutCustomizer(properties: DefaultHttpConnectorProperties) =
        timeoutHandler(properties.timeouts?.write) { timeout, unit ->  WriteTimeoutHandler(timeout, unit) }

    private fun timeoutHandler(timeout: DefaultHttpConnectorProperties.Timeout?, channelHandler: (timeout: Long, unit: TimeUnit) -> ChannelHandler) =
        ReactorNettyHttpClientMapper { httpClient: HttpClient ->
            timeout?.let {
                httpClient.tcpConfiguration { config ->
                    config.doOnConnected { connection -> connection.addHandlerLast(channelHandler(it.value, it.unit)) }
                }
            }
    }

    @Bean
    @ConditionalOnMissingBean
    @Suppress("NULLABILITY_MISMATCH_BASED_ON_JAVA_ANNOTATIONS")
    fun webClient(builder: WebClient.Builder, clientHttpConnector: ClientHttpConnector? = null, uriBuilderFactory: UriBuilderFactory? = DefaultUriBuilderFactory()): WebClient =
        builder.clientConnector(clientHttpConnector).uriBuilderFactory(uriBuilderFactory).build()

I did try to switch netty DNS resolver like this:

    @Bean // properties.additionOptions.switchDnsResolver was true
    fun additionalOptionsSwitchDnsResolverCustomizer(properties: DefaultHttpConnectorProperties) =
            ReactorNettyHttpClientMapper { httpClient: HttpClient ->
                properties.additionOptions?.let {
                    httpClient.tcpConfiguration { config ->
                        if(it.switchDnsResolver) {
                            config.resolver(DefaultAddressResolverGroup.INSTANCE)
                        }
                        config
                    }
                }
            }

But no success with this. I tried to switch docker base images, try some additional docnfiguration of netty DNS resolver, but with no success.

[chrisvest]

You'll need to do some more debugging before we can tell what's going on, like running dig(1) on the host and enabling debug logging in Netty.

[xsmrcek]

Hello @chrisvest ,
thank you for reply!
Here is dig from my namespace:

$ dig hcd-hi-dms-webapi.gma-hi-dms

; <<>> DiG 9.11.3-1ubuntu1.17-Ubuntu <<>> hcd-hi-dms-webapi.gma-hi-dms
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 11010
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 2a8970536af41ec699e8e24962e1a743ad5b62d7b4ed97ce (good)
;; QUESTION SECTION:
;hcd-hi-dms-webapi.gma-hi-dms.  IN      A

;; AUTHORITY SECTION:
.                       300     IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2022072701 1800 900 604800 86400

;; Query time: 5 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Jul 27 20:59:47 UTC 2022
;; MSG SIZE  rcvd: 160

Everything I see in log before exception is this:

id: 0x1adf971a] RECEIVED: UDP [26201: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 26201, QUERY(0), NXDomain(3), RD AA)
DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.gma-hi-loan-orig.svc.cluster.local. IN A)
DefaultDnsRawRecord(cluster.local. 5 IN SOA 68B)
DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] WRITE: UDP, [26201: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.gma-hi-loan-orig.svc.cluster.local. IN A)

[id: 0x1adf971a] WRITE: UDP, [13862: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.gma-hi-loan-orig.svc.cluster.local. IN CNAME)

[id: 0x1adf971a] RECEIVED: UDP [13862: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 13862, QUERY(0), NXDomain(3), RD AA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.gma-hi-loan-orig.svc.cluster.local. IN CNAME) DefaultDnsRawRecord(cluster.local. 5 IN SOA 68B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] WRITE: UDP, [60454: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.svc.cluster.local. IN A)

[id: 0x1adf971a] RECEIVED: UDP [60454: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 60454, QUERY(0), NXDomain(3), RD AA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.svc.cluster.local. IN A) DefaultDnsRawRecord(cluster.local. 5 IN SOA 68B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] WRITE: UDP, [2823: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.svc.cluster.local. IN CNAME)

[id: 0x1adf971a] WRITE: UDP, [50489: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.cluster.local. IN A)

[id: 0x1adf971a] RECEIVED: UDP [2823: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 2823, QUERY(0), NXDomain(3), RD AA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.svc.cluster.local. IN CNAME) DefaultDnsRawRecord(cluster.local. 5 IN SOA 68B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] RECEIVED: UDP [50489: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 50489, QUERY(0), NXDomain(3), RD AA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.cluster.local. IN A) DefaultDnsRawRecord(cluster.local. 5 IN SOA 68B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] WRITE: UDP, [59697: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.cluster.local. IN CNAME)

[id: 0x1adf971a] RECEIVED: UDP [59697: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 59697, QUERY(0), NXDomain(3), RD AA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.cluster.local. IN CNAME) DefaultDnsRawRecord(cluster.local. 5 IN SOA 68B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] WRITE: UDP, [8672: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.reddog.microsoft.com. IN A)

[id: 0x1adf971a] RECEIVED: UDP [8672: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 8672, QUERY(0), NXDomain(3), RD RA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.reddog.microsoft.com. IN A) DefaultDnsRawRecord(microsoft.com. 30 IN SOA 77B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] WRITE: UDP, [50536: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.reddog.microsoft.com. IN CNAME)

[id: 0x1adf971a] RECEIVED: UDP [50536: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 50536, QUERY(0), NXDomain(3), RD RA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms.reddog.microsoft.com. IN CNAME) DefaultDnsRawRecord(microsoft.com. 30 IN SOA 77B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] WRITE: UDP, [62586: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms. IN A)

[id: 0x1adf971a] RECEIVED: UDP [62586: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 62586, QUERY(0), NXDomain(3), RD RA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms. IN A) DefaultDnsRawRecord(. 30 IN SOA 64B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

[id: 0x1adf971a] WRITE: UDP, [7298: /10.10.0.10:53], DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms. IN CNAME)

[id: 0x1adf971a] RECEIVED: UDP [7298: /10.10.0.10:53], DatagramDnsResponse(from: /10.10.0.10:53, 7298, QUERY(0), NXDomain(3), RD RA) DefaultDnsQuestion(hcd-hi-dms-webapi.gma-hi-dms. IN CNAME) DefaultDnsRawRecord(. 30 IN SOA 64B) DefaultDnsRawRecord(OPT flags:0 udp:4096 0B)

"message":"client GET call failed","context":"default","exception":"org.springframework.web.reactive.function.client.WebClientRequestException: Failed to resolve 'hcd-hi-dms-webapi.gma-hi-dms' after 2 queries nested exception is java.net.UnknownHostException: Failed to resolve 'hcd-hi-dms-webapi.gma-hi-dms' after 2 queries \n\tat org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)\n\tSuppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: \nError has been observed at the following site(s):\n\t*__checkpoint ⇢ Request to GET http://hcd-hi-dms-webapi.gma-hi-dms/**** [DefaultWebClient]\nOriginal Stack Trace:\n\t\tat org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)\n\t\tat reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55)\n\t\tat reactor.core.publisher.Mono.subscribe(Mono.java:4397)\n\t\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)\n\t\tat reactor.core.publisher.Operators.error(Operators.java:198)\n\t\tat reactor.core.publisher.MonoError.subscribe(MonoError.java:53)\n\t\tat reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)\n\t\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)\n\t\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onComplete(TracingSubscriber.java:68)\n\t\tat reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onComplete(TracingSubscriber.java:68)\n\t\tat reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)\n\t\tat reactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:138)\n\t\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onSubscribe(MonoIgnoreThen.java:134)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onSubscribe$0(TracingSubscriber.java:53)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onSubscribe(TracingSubscriber.java:53)\n\t\tat reactor.core.publisher.FluxPeek$PeekSubscriber.onSubscribe(FluxPeek.java:171)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onSubscribe$0(TracingSubscriber.java:53)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onSubscribe(TracingSubscriber.java:53)\n\t\tat reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)\n\t\tat reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)\n\t\tat reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55)\n\t\tat reactor.core.publisher.Mono.subscribe(Mono.java:4397)\n\t\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)\n\t\tat reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)\n\t\tat reactor.core.publisher.Mono.subscribe(Mono.java:4397)\n\t\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)\n\t\tat reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225)\n\t\tat reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerError(FluxConcatMap.java:309)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapInner.onError(FluxConcatMap.java:875)\n\t\tat reactor.core.publisher.Operators.error(Operators.java:198)\n\t\tat reactor.core.publisher.MonoError.subscribe(MonoError.java:53)\n\t\tat reactor.core.publisher.Mono.subscribe(Mono.java:4397)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:451)\n\t\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)\n\t\tat reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:537)\n\t\tat reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:343)\n\t\tat reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)\n\t\tat reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)\n\t\tat reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)\n\t\tat reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:308)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)\n\t\tat reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:158)\n\t\tat reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:475)\n\t\tat reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:431)\n\t\tat reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)\n\t\tat reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:548)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)\n\t\tat reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)\n\t\tat reactor.core.publisher.Operators.error(Operators.java:198)\n\t\tat reactor.core.publisher.MonoError.subscribe(MonoError.java:53)\n\t\tat reactor.core.publisher.Mono.subscribe(Mono.java:4397)\n\t\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)\n\t\tat reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onError$2(TracingSubscriber.java:63)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:79)\n\t\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onError(TracingSubscriber.java:63)\n\t\tat reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)\n\t\tat reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)\n\t\tat reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:534)\n\t\tat reactor.netty.transport.TransportConnector.lambda$doResolveAndConnect$11(TransportConnector.java:341)\n\t\tat io.opentelemetry.javaagent.instrumentation.netty.common.FutureListenerWrappers$WrappedFutureListener.operationComplete(FutureListenerWrappers.java:106)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\t\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\t\tat io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)\n\t\tat io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:109)\n\t\tat io.netty.resolver.InetSocketAddressResolver$2.operationComplete(InetSocketAddressResolver.java:86)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\t\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\t\tat io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)\n\t\tat io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)\n\t\tat io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1055)\n\t\tat io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1000)\n\t\tat io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418)\n\t\tat io.netty.resolver.dns.DnsResolveContext.onResponse(DnsResolveContext.java:629)\n\t\tat io.netty.resolver.dns.DnsResolveContext.access$400(DnsResolveContext.java:66)\n\t\tat io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:462)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)\n\t\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\t\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\t\tat io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)\n\t\tat io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)\n\t\tat io.netty.resolver.dns.DnsQueryContext.trySuccess(DnsQueryContext.java:232)\n\t\tat io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:224)\n\t\tat io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:1314)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\t\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\t\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\t\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\t\tat io.netty.channel.epoll.EpollDatagramChannel.processPacket(EpollDatagramChannel.java:649)\n\t\tat io.netty.channel.epoll.EpollDatagramChannel.recvmsg(EpollDatagramChannel.java:684)\n\t\tat io.netty.channel.epoll.EpollDatagramChannel.access$300(EpollDatagramChannel.java:55)\n\t\tat io.netty.channel.epoll.EpollDatagramChannel$EpollDatagramChannelUnsafe.epollInReady(EpollDatagramChannel.java:523)\n\t\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)\n\t\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)\n\t\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\t\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\t\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\t\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.net.UnknownHostException: Failed to resolve 'hcd-hi-dms-webapi.gma-hi-dms' after 2 queries \n\tat io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1047)\n\tat io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1000)\n\tat io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418)\n\tat io.netty.resolver.dns.DnsResolveContext.onResponse(DnsResolveContext.java:629)\n\tat io.netty.resolver.dns.DnsResolveContext.access$400(DnsResolveContext.java:66)\n\tat io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:462)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\tat io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)\n\tat io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)\n\tat io.netty.resolver.dns.DnsQueryContext.trySuccess(DnsQueryContext.java:232)\n\tat io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:224)\n\tat io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:1314)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.epoll.EpollDatagramChannel.processPacket(EpollDatagramChannel.java:649)\n\tat io.netty.channel.epoll.EpollDatagramChannel.recvmsg(EpollDatagramChannel.java:684)\n\tat io.netty.channel.epoll.EpollDatagramChannel.access$300(EpollDatagramChannel.java:55)\n\tat io.netty.channel.epoll.EpollDatagramChannel$EpollDatagramChannelUnsafe.epollInReady(EpollDatagramChannel.java:523)\n\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}

[xsmrcek]

Issue was on mine side. I didn't realise that we have two cluster running at same time, one had http traffic dissabled, but I didn't dissabled rammitMQ notifications on it, so I was receiving some messages into broken cluster. Keeping this as a comment here, so it may help someone else with same issue :D