[Bug] Unresponsive server

[andrewazores]

Current Behavior

@aali309 's Cryostat server (main branch as of time of writing) in smoketest.sh became unresponsive to API requests, also manifesting as a web-client UI that would get stuck trying to connect:

$ https --auth=user:pass :8181/api/v1/recordings
https: error: ConnectionError: HTTPSConnectionPool(host='localhost', port=8181): Max retries exceeded with url: /api/v1/recordings (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4920a8ca90>: Failed to establish a new connection: [Errno 111] Connection refused')) while doing a GET request to URL: https://localhost:8181/api/v1/recordings

Expected Behavior

Whatever happened to cause this condition, the server should be resilient against and remain responsive.

Steps To Reproduce

Unknown.

Tearing down the smoketest.sh, clearing state with rm -rf conf/h2.mv.db conf/rules conf/metadata archive/*, and restarting the smoketest.sh resulted in a fresh and working deployment.

Evidently the run.sh --health-cmd didn't catch that the server had become unresponsive.

Environment

main as of commit https://github.com/cryostatio/cryostat/tree/0197f7b20ca9a90c0831e4936b95c04108378afc . Podman on Fedora 38, smoketest.sh

Anything else?

unresponsive-cryostat.log