Issues with Kubernetes Mock Server since 6.4.0

[scholzj]

In Strimzi, we use for some of our tests the Kubernetes Mock Server and it worked reasonably fine for us until 6.3.1. But from 6.4.0, we seem to have two issues in many places:

• The tests are approximately 4-5 times slower compared to 6.3.1
• When calling replaceStatus() or patchStatus() on one of our custom resources, it seems there is a race condition where for some time, the mock server returns null / 404 to a get() call on a given resource which is done in parallel to the status update call.

This seems to happen with both 6.4.0 and 6.4.1 but also with the latest 6.5-SNAPSHOT builds.

Is this a known issue? Or anyone else experienced something similar? If not, I will try to put together some reproducer.

[manusa]

The tests are approximately 4-5 times slower compared to 6.3.1

This was also reported in #4786. We did have some changes recently that involved additional serialization-deserialization overhead (#4801), and also added resource version validation (for optimistic locking) in #4740. However, none of these have been released yet. If the difference in performance is between 6.4.x and 6.5-SNAPSHOT, then these are good candidates.

If the performance degradation also appears in 6.4 it can only be related to the changes in OkHttp client-side.

it seems there is a race condition

For the new race condition, if it appears in 6.5 then can be related to #4740.

In any case, having a few more hints would help in resolving these issues.

[scholzj]

This is an example of an actual Strimzi test where it is failing with 6.4 and passing with 6.3: https://github.com/strimzi/strimzi-kafka-operator/blob/main/cluster-operator/src/test/java/io/strimzi/operator/cluster/operator/assembly/KafkaAssemblyOperatorCustomCertMockTest.java => but as the reproducer suggest that is likely just because of the timing which is different. This test runs the Strimzi controllers + our own mock controllers which emulate the Pod or Service readiness in the MockKube. So they are more complicated compared to the reproducer.

[shawkins]

The fix in #4945 does address the null issue. I don't see as big of a performance difference locally, but enough of a difference between 6.3.1 and 6.4.1 that something should be attributable.

[shawkins]

The biggest difference locally was a change to move the Serialization logic handling multiple docs in a single stream to a more common location. The doc validate method probably because of the negative lookahead, is quite expensive even over relatively short inputs.

[scholzj]

Thanks a lot for looking into it.

[scholzj]

I tried it with 6.5.0 and all the issues I reported here are fixed. Also the speed seems to be (at least roughly) the same as before with 6.3.1. Thanks a lot for that 😍.

Unfortunately I run into another issue with leader election and the Mock Server - but that seems to completely unrelated to this. I opened #4965 for it.

[shawkins]

When calling replaceStatus() or patchStatus() on one of our custom resources, it seems there is a race condition where for some time, the mock server returns null / 404 to a get() call on a given resource which is done in parallel to the status update call.

A possible root issue is that there is inconsistent synchronization around the map of resources held by the mock server. The KubernetesCrudDispatcher.processEvent method will temporarily remove the resource, then add it back if it's not a delete operation so there is a possible race condition for patch and put operations. It looks like create, delete, get, and watches have appropriate locking when being processed. That doesn't seem like a new problem though - the logic has been this way at least since 6.0.

[scholzj]

Well, I never saw it before with 6.3.1 or older. But as I said, things seem to be also much slower now with 6.4.0+. So it could be that this just made it much more likely for it to happen and it was just a coincidence it seemed to work fine before.

[shawkins]

That's possible - we noticed with other testing that the okhttp client seems to some implicit flow control that would potentially avoid some concurrency issues.