Consumers disconnect after long idle from RabbitMQ queue

[stefanofal-altilia]

In my company we are currently facing a strange behavour using rabbitMQ. One of our microservices hosted in a Kubernetes Cluster is built in Python (the others in Java and they don't show this problem) and communicates using Pika implementation for RabbitMQ.

During the week the microservice works just fine, in the weekend it is not used and still up and running. On monday trying to publish a message on this microservice queue results in them remaining unacked because all the consumers have disconnected. Looking in the code logs, there is absolutely nothing: no error, no exception thrown, nothing at all. The service remains idle without noticing it is disconnected from the queue.

In the first implementation we had a BlockingConnection, but referencing this issue #877 and suspecting some kind of idle termination, we moved to SelectConnection using ioloops. The heartbeat is configured with standard 60s with broker negotation. I assume it is not a problem in the rabbitMQ broker because every other microservice in Java is working just fine and the connection keeps alive for weeks.

I could not find anything relevant on the logs of rabbitMQ either because of all the other microservices' noise or due to the behavour being silent, maybe.

About our configuration:
Pika version 1.2.0
Python 3.8.3
RabbitMQ 3.8.18
Erlang 23.3.4.4

The reproducible code can be found here: github link to repo

This is a crosspost of this stackoverflow question following @lukebakken kind suggestion.

Current logs for the three RabbitMQ instances running in the Cluster.
1.txt
2.txt
3.txt

[lukebakken]

Hello! I really need to know what RabbitMQ logs in your environment. Please attach the logs (or add them to your repo), even if you don't think they contain anything relevant. Thanks.

[stefanofal-altilia]

Here the configurations, there is no advanced.config that I could find. I removed only the password for safety concerns from configuration_A.
configurations.zip

[stefanofal-altilia]

Hello @lukebakken, sorry to bother you again, but I am running in an issue. One of the microservices in Python crashed, logs are enabled so I could see what is happening (logs attached filtering for pika).

Connection._on_connection_close_from_broker of <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7f9f7ced20d0> params=<ConnectionParameters host=rabbitmq.rabbitmq-system.svc.cluster.local port=5672 virtual_host=/ ssl=False>>> for "0:Connection.Close"
DEBUG      2023-03-01 12:32:36,322 pika.connection                _on_connection_close_from_broker     1777: _on_connection_close_from_broker: frame=<METHOD(['channel_number=0', 'frame_type=1', 'method=<Connection.Close([\'class_id=0\', \'method_id=0\', \'reply_code=320\', "reply_text=CONNECTION_FORCED - broker forced connection closure with reason \'shutdown\'"])>'])>

I think this is the reason why the consumer is disconnected, but I am questioning why no exception was raised in the code. Attached there is the filtered log and the complete one from our A cluster.

If you prefer, I could open another issue here on github. Thanks.
filtered_logs.txt
complete_logs.txt

[lukebakken]

What was logged by RabbitMQ around the same point in time?

[stefanofal-altilia]

Unfortunetly, our logs don't go so far to that time, but I found something interesting. In our configuration, we have three pods running rabbitmq, and coincidentally one of them restarted exactly when this error was thrown.
This explains why the connection failed and the broker closed it, but I would like to understand if it is usually silent in Pika or an Exception is thrown.

[lukebakken]

The Pika message you provided is exactly what I would expect to see when the connected broker instance is restarted.