RejectedExecutionException when checking status of removed operator through BaseOperation.get

[fedinskiy]

Describe the bug

I have a code, which uses kubernetes client to deploy OpenShift operator and application and to check its status. After deleting a project and doing a final check, what everything was deleted properly, I receive a cryptic exception: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@553d2579 rejected from java.util.concurrent.ThreadPoolExecutor@2a8f6e6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 166]. The very same code forked flawlessly with the 5 version of Kubernetes client

Fabric8 Kubernetes Client version

6.1.1

Steps to reproduce

1. Clone the reproducer: git clone git@github.com:fedinskiy/quarkus-test-framework.git -b reproducer/fabric8
2. Change to the root folder of the reproducer cd quarkus-test-framework
3. Build the core components: mvn clean install -DskipTests -DskipITs
4. Run the reproducer mvn clean verify -Popenshift -Pexamples -pl examples/kafka -Dquarkus.platform.version=999-SNAPSHOT with 6.1.1
5. For comparison, run the reproducer with 5.12.3: mvn clean verify -Popenshift -Pexamples -pl examples/kafka

Expected behavior

BaseOperation.get should either return custom resource with proper status or null for non-existing resources or throw an error, which is easy to understand.

Runtime

OpenShift

Kubernetes API Server version

1.24.2@latest

Environment

Linux

Fabric8 Kubernetes Client Logs

io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Kafka]  with name: [kafka-instance]  in namespace: [ts-eogxmeeray]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:177)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:139)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:88)
	at io.quarkus.test.bootstrap.inject.OpenShiftClient.isCustomResourceReady(OpenShiftClient.java:504)
	at io.quarkus.test.services.operator.OperatorManagedResource.lambda$customResourcesAreReady$0(OperatorManagedResource.java:87)
	at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
	at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632)
	at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
	at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
	at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:533)
	at io.quarkus.test.services.operator.OperatorManagedResource.customResourcesAreReady(OperatorManagedResource.java:87)
	at io.quarkus.test.services.operator.OperatorManagedResource.isRunning(OperatorManagedResource.java:53)
	at io.quarkus.test.bootstrap.BaseService.isRunning(BaseService.java:128)
	at io.quarkus.test.bootstrap.BaseService.stop(BaseService.java:209)
	at io.quarkus.test.bootstrap.BaseService.close(BaseService.java:226)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$closeAllStoredCloseableValues$3(ExtensionValuesStore.java:68)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at java.base/java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:395)
	at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at org.junit.jupiter.engine.execution.ExtensionValuesStore.closeAllStoredCloseableValues(ExtensionValuesStore.java:68)
	at org.junit.jupiter.engine.descriptor.AbstractExtensionContext.close(AbstractExtensionContext.java:80)
	at org.junit.jupiter.engine.execution.JupiterEngineExecutionContext.close(JupiterEngineExecutionContext.java:53)
	at org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:222)
	at org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:57)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$cleanUp$10(NodeTestTask.java:167)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.cleanUp(NodeTestTask.java:167)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.io.InterruptedIOException: executor rejected
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:157)
	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:204)
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:144)
	at okhttp3.RealCall.enqueue(RealCall.java:93)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.sendAsync(OkHttpClientImpl.java:264)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:598)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:577)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:558)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:469)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:711)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:172)
	... 70 more
Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@3b7c58e7 rejected from java.util.concurrent.ThreadPoolExecutor@79627d27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 169]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:154)
	... 80 more

Additional context

$ oc version
Client Version: 4.11.0-202208020706.p0.g7075089.assembly.stream-7075089
Kustomize Version: v4.5.4
Kubernetes Version: v1.24.0+3882f8f

Related issues:
quarkusio/quarkus#28405 — client libraries are provided by Quarkus, but Quarkus team do not change them
#4448 — similar bug, but caused by usage of try-with resources.

[shawkins]

This means that the client/httpclient has been closed - we probably need a higher level check for that to make it clear. Can you find in your code where the client is being closed early?

[fedinskiy]

We closing the client twice, once as a part of Junit cleaning after all tests and once as a part of Junit cleaning on closing. But we have this problem only with strimzi operator and not, for example, with strimzi deployed in a separate container(test OpenShiftStrimziKafkaWithoutRegistryMessagingIT in the same module).

[shawkins]

But we have this problem only with strimzi operator and not, for example, with strimzi deployed in a separate container(test OpenShiftStrimziKafkaWithoutRegistryMessagingIT in the same module).

This is unrelated to the functional part of the test, it's just related to when it's being closed - this operation is being performed after it's closed. Can you put a breakpoint on the client close to see where it is happening early?

[fedinskiy]

Yes, we call close on DefaultOpenShiftClient during project deletion(method deleteProject in OpenShiftClient in the reporducer) and then we call some methods on related NamespacedOpenShiftClient (method isCustomResourceReady in OpenShiftClient ).

That surprises me, that the same process works flawlessly with non-operator workload.