Celery workers not sending heartbeat

[arvindnrbt]

Similar to #3649, but I'm not using celery beat. Just with the celery worker instance.

Celery 4.2.1 I'm still facing this issue.
RabbitMQ log

rabbitmq_1  | =ERROR REPORT==== 15-Aug-2018::10:05:42 ===
rabbitmq_1  | closing AMQP connection <0.28344.0> (172.18.0.5:46918 -> 172.18.0.3:5672):
rabbitmq_1  | missed heartbeats from client, timeout: 60s

Celery -A proj inspect active

15-08-2018:10:08:41,476 DEBUG    [connection.py:360] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@09ef45b313ed', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP', 'product': 'RabbitMQ', 'version': '3.6.1'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
15-08-2018:10:08:41,490 DEBUG    [channel.py:106] using channel_id: 1
15-08-2018:10:08:41,493 DEBUG    [channel.py:443] Channel open
15-08-2018:10:08:41,516 DEBUG    [connection.py:360] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@09ef45b313ed', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP', 'product': 'RabbitMQ', 'version': '3.6.1'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
15-08-2018:10:08:41,521 DEBUG    [channel.py:106] using channel_id: 1
15-08-2018:10:08:41,524 DEBUG    [channel.py:443] Channel open
Error: No nodes replied within time constraint.

After 5 min,
RabbitMQ

rabbitmq_1  | =WARNING REPORT==== 15-Aug-2018::10:10:30 ===
rabbitmq_1  | closing AMQP connection <0.1904.1> (172.18.0.5:47004 -> 172.18.0.3:5672):
rabbitmq_1  | client unexpectedly closed TCP connection

celery -A proj inspect active

15-08-2018:10:12:03,837 DEBUG    [connection.py:360] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@09ef45b313ed', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP', 'product': 'RabbitMQ', 'version': '3.6.1'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
15-08-2018:10:12:03,844 DEBUG    [channel.py:106] using channel_id: 1
15-08-2018:10:12:03,848 DEBUG    [channel.py:443] Channel open
15-08-2018:10:12:03,869 DEBUG    [connection.py:360] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@09ef45b313ed', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP', 'product': 'RabbitMQ', 'version': '3.6.1'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
15-08-2018:10:12:03,876 DEBUG    [channel.py:106] using channel_id: 1
15-08-2018:10:12:03,879 DEBUG    [channel.py:443] Channel open
-> celery@ocrworker: OK
    - empty -

In celery log:

[2018-08-15 11:25:24,298: ERROR/MainProcess] Process 'ForkPoolWorker-1' pid:9 exited with 'signal 15 (SIGTERM)'
[2018-08-15 11:25:27,154: ERROR/MainProcess] Process 'ForkPoolWorker-2' pid:10 exited with 'signal 15 (SIGTERM)'
[2018-08-15 11:32:44,723: ERROR/MainProcess] Control command error: ConnectionResetError(104, 'Connection reset by peer')
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/celery/worker/pidbox.py", line 46, in on_message
    self.node.handle_message(body, message)
  File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 129, in handle_message
    return self.dispatch(**body)
  File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 112, in dispatch
    ticket=ticket)
  File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 135, in reply
    serializer=self.mailbox.serializer)
  File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 265, in _publish_reply
    **opts
  File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/usr/lib/python3.6/site-packages/amqp/channel.py", line 1732, in _basic_publish
    (0, exchange, routing_key, mandatory, immediate), msg
  File "/usr/lib/python3.6/site-packages/amqp/abstract_channel.py", line 50, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/usr/lib/python3.6/site-packages/amqp/method_framing.py", line 166, in write_frame
    write(view[:offset])
  File "/usr/lib/python3.6/site-packages/amqp/transport.py", line 275, in write
    self._write(s)
ConnectionResetError: [Errno 104] Connection reset by peer

I'm running celery inside docker container and I'm worried about running celery with no downtime as we had Connection reset by peer error in our tests often. Please throw some light on this.

[olii]

Releated to #4817. Heartbeat is not being called on some connections in a connection pool.

[arvindnrbt]

Is there any way to fix this now? From #4817 I can see that the only workaround is to set heartbeat to 0 and get it over with. But what about heartbeat monitoring in production applications?

[olii]

You don't need a heartbeat when running TCP connection with a keepalive interval set reasonably low. But anyway, this is a bug and it needs a proper fix.
Read a few hints here: https://www.cloudamqp.com/docs/celery.html

[sposs]

@olii what do you recommend as 'reasonably low'? I have the same sort of issue and try to apply the 'broker_heartbeat=0' fix...

[olii]

@sposs it is completely up to you. You can experiment with any value you like for your use case. Note that the default Linux TCP timeout is 2 hours. http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html

[kleysonr]

Facing the same error, using prefork.

python3
celery==4.2.1
amqp==2.3.2

With RabbitMQ running inside a docker container.

[YuanBLQ]

Facing the same dilemma as @kleysonr

python3
amqp==2.2.2
celery==4.2.1

RabbitMQ running inside a docker container.

[CrowbarKZ]

I've had this issue with all pool types with
py3.6
celery==4.2.1 + rabbitmq (both running in docker containers)

Downgrading to celery==4.1.1 seems to solve the issue for me

[avinashbakshi01]

Facing similar issue
RabbitMQ and Celery 4.1.1 in docker container

[auvipy]

celery==4.4.0 is the latest release

[auvipy]

can any of you check this celery/py-amqp#374?