NATS timeout errors when calling NatsJetStreamManagement

[joachimglink]

Observed behavior

As described in Slack channel https://natsio.slack.com/archives/CM3T6T7JQ/p1692790156946559 we´re sometimes seeing timeout errors when we try to fetch information from Jetstream.

We see that the request goes into the NATS server but the response is never received. Not 100% sure if this is a server or client issue. Please move to another repo if this one doesn´t match.

We even see these timeouts if we increase the timeout threshold to 10 or even 30 seconds.
As this is reproducable in small tests without a high load on the system, the system resources shouldn´t be the problem here.

Expected behavior

All requests against the NatsJetStreamManagement should be answered.

Server and client version

Experienced same behavior on different server (2.8.x / 2.9.x and 2.10.0) and client (jnats 2.14.x, 2.16.x) versions.

Host environment

Windows; NATS server running in Docker (Docker Desktop for Windows)
GKE cluster

Steps to reproduce

nats-timeout-reproducer.zip

The attached ZIP contains a simple reproducer:

container-starter
This module creates a Testcontainer starting the NATS server.

reproducer
A stripped down version of our code which creates a stream and registers subjects to it and also adds message consumers to them.

How to build
Simply do a mvn clean install -DskipTests in the parent folder.
After that, go into reproducer and execute the TimeoutReproducerTest.bat which starts the supplied test 25 times. A couple of executions will success, others will fail with the mentioned timeout error.
The log files of each test-run are placed under the ./logs folder.

[scottf]

@joachimglink Please see dm in slack.