API doesn't allways succeed in removing a topic (job_id) from the RabbitMQ client when it is done with streaming logs.

[bisgaard-itis]

Is there an existing issue for this?

• I have searched the existing issues

Ideas and thoughts

• One thing which should definititely be done is to modify the current health check in the API-server so that each time the background task runs it registers the dummy job_id and unregisters it again. The "leaking" of job_ids would probably be detected by this check.
• Figure out if there is a bullet-proof way to ensure that a topic is deregistered. Would it e.g. be possible to ask the RabbitMq server periodically and check that all job_ids which are not streaming logs should be deregistered. I would like to discuss this with you @sanderegg
• Probably this will also be a bottleneck if a sufficient number of users stream logs via the API server at the same time. So it might be worth considering ways to communicate this to the orchestrator to spin up additional replicas of the api-server.

[bisgaard-itis]

@mguidon FYI
I suspect this is the reason behind the log streaming issues we face sometimes.