Integration Test Failures

[shawkins]

After augmenting some test runs there's are oddities to how the test framework is using connections when there's a failure. Here's what appears to be a normal test run sequence:

2022-12-03T19:28:10.8669222Z [INFO] Running io.fabric8.kubernetes.PersistentVolumeClaimIT
2022-12-03T19:28:10.8789350Z Currently running calls 1 okhttp3.Dispatcher@1bec36ee io.fabric8.junit.jupiter.KubernetesNamespacedTestExtension
... repeated 4 times ...
2022-12-03T19:28:11.0796308Z Currently running calls 1 okhttp3.Dispatcher@1bec36ee io.fabric8.junit.jupiter.LoadKubernetesManifestsExtension
... repeated 3 times ...
2022-12-03T19:28:11.1372743Z Currently running calls 1 okhttp3.Dispatcher@1bec36ee null
... repeated 3 times - null means the actual test method(s) are running ...
2022-12-03T19:28:11.2798191Z Currently running calls 1 okhttp3.Dispatcher@1bec36ee io.fabric8.junit.jupiter.LoadKubernetesManifestsExtension
... repeated 3 times ...
2022-12-03T19:28:11.3511407Z Currently running calls 1 okhttp3.Dispatcher@1bec36ee io.fabric8.junit.jupiter.KubernetesNamespacedTestExtension
2022-12-03T19:28:11.3548048Z [main] WARN io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl - Shutting down dispatcher okhttp3.Dispatcher@1bec36ee
2022-12-03T19:28:11.3628138Z 	at io.fabric8.junit.jupiter.KubernetesNamespacedTestExtension.afterAll(KubernetesNamespacedTestExtension.java:75)
2022-12-03T19:28:11.3994834Z [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.525 s - in io.fabric8.kubernetes.PersistentVolumeClaimIT

Then there's a failure with the next test during the setup:

2022-12-03T19:28:11.4059353Z [INFO] Running io.fabric8.kubernetes.StorageClassIT
2022-12-03T19:28:11.4951293Z Currently running calls 1 okhttp3.Dispatcher@364b80c1 io.fabric8.junit.jupiter.KubernetesNamespacedTestExtension
... repeated 3 times ...

### There would normally be 5 total calls via KubernetesNamespacedTestExtension, but instead here the 5th call did not have a io.fabric8.junit class in it's call stack 
- it should not be a test method call because it's happening before LoadKubernetesManifestsExtension has run:

2022-12-03T19:28:11.5899827Z Currently running calls 1 okhttp3.Dispatcher@364b80c1 null

### Now when the LoadKubernetesManifestsExtension runs it's actually using the dispatcher from the previous test's client - that will result in an exception / failure 
- the Suppressed exception below confirms it's from the dispatcher that was already closed by the previous test:

2022-12-03T19:28:41.5617166Z Currently running calls 0 okhttp3.Dispatcher@1bec36ee io.fabric8.junit.jupiter.LoadKubernetesManifestsExtension

2022-12-03T19:28:41.5816681Z [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.198 s <<< FAILURE! - in io.fabric8.kubernetes.StorageClassIT
2022-12-03T19:28:41.5821911Z [ERROR] io.fabric8.kubernetes.StorageClassIT  Time elapsed: 30.198 s  <<< ERROR!
2022-12-03T19:28:41.5823086Z io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [30000] milliseconds for [ServiceAccount] with name:[default] in namespace [3949e92e-5557-4d1e-9b34-332f6e443d56].
2022-12-03T19:28:41.5824028Z 	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:862)
2022-12-03T19:28:41.5824855Z 	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:89)
2022-12-03T19:28:41.5825764Z 	at io.fabric8.junit.jupiter.KubernetesNamespacedTestExtension.initNamespace(KubernetesNamespacedTestExtension.java:114)
2022-12-03T19:28:41.5826671Z 	at io.fabric8.junit.jupiter.KubernetesNamespacedTestExtension.beforeAll(KubernetesNamespacedTestExtension.java:49)
...
2022-12-03T19:28:41.5869974Z 	Suppressed: io.fabric8.kubernetes.client.KubernetesClientException: The okhttp client executor has been shutdown.  More than likely this is because the KubernetesClient.close method has been called - please ensure that is intentional. okhttp3.Dispatcher@1bec36ee

I'm not sure what exactly is going on here - are we not scoping the storage of the client enough?

[shawkins]

Link to this occurring in a recent build: https://github.com/fabric8io/kubernetes-client/actions/runs/3620194935/jobs/6102423363#step:6:472

[manusa]

OK, so you mean that the problem happens with the JUnit extension that prepares the environment (create namespace, and so on).

Now I see this more clearly.

Let me try to refine that extension.

[shawkins]

Yes, there's a class of these failures that appear to happen during test setup. The underlying exception appears to be due to the okhttpclient being closed - https://github.com/fabric8io/kubernetes-client/actions/runs/3621420568/jobs/6105155485#step:6:302 - but that appears to be because the test is using the client from the previous test. I'm trying to confirm / address with 7fc7a24

[shawkins]

Retargeted the pr to address a regression in the watch logic that can cause missed events.