Replies: 39 comments 35 replies
-
|
Facing this exact issue. |
Beta Was this translation helpful? Give feedback.
-
|
I experience the same issue with The redis service would restart but the celery worker won't consume messages from redis afterwards. |
Beta Was this translation helpful? Give feedback.
-
|
Same issue with: |
Beta Was this translation helpful? Give feedback.
-
|
same issue with: |
Beta Was this translation helpful? Give feedback.
-
|
Same issue with: although the worker continued to consume tasks for ~20 minutes after redis reconnected, then stopped. |
Beta Was this translation helpful? Give feedback.
-
|
Same issue: |
Beta Was this translation helpful? Give feedback.
-
|
I also want to know if any negative impact to run celery without the heartbeat/gossip/mingle enabled? Any idea for this ? thanks ! |
Beta Was this translation helpful? Give feedback.
-
|
Same issue here |
Beta Was this translation helpful? Give feedback.
-
|
Had the same issue. But, after some random period of time those tasks gets consume. Sometimes it takes loooong time (even hours), but they finally start to show up. |
Beta Was this translation helpful? Give feedback.
-
|
We had the same issue, for now, we're "working around it" by using --without-mingle and --without-gossip (I did not use --without-heartbeat) but seems like the problem is resolved. Hopefully we won't run into new issues because we deactivated both features. |
Beta Was this translation helpful? Give feedback.
-
|
Actually, we're still encountering the issue from time to time which is pretty critical |
Beta Was this translation helpful? Give feedback.
-
|
Tried --without-mingle and --without-gossip which won't work. How about your latest status? |
Beta Was this translation helpful? Give feedback.
-
|
celery==5.3.1 I have the same problem on production and reproduced the problem locally, adding "--without-mingle" to celery worker solved the problem. |
Beta Was this translation helpful? Give feedback.
-
|
Same issue here, using a redis cluster |
Beta Was this translation helpful? Give feedback.
-
|
Celery 5.2.7 with redis broker |
Beta Was this translation helpful? Give feedback.
-
|
Same here on |
Beta Was this translation helpful? Give feedback.
-
|
Same issue here with
|
Beta Was this translation helpful? Give feedback.
-
|
Same issue here
|
Beta Was this translation helpful? Give feedback.
-
|
I added a step by step process to reproduce the bug, please let me know if I can provide anything else to help this being resolved. Thanks a lot ! |
Beta Was this translation helpful? Give feedback.
-
|
Same issue but I'm using RabbitMQ as a broker and Redis as the backend: |
Beta Was this translation helpful? Give feedback.
-
|
same issue: |
Beta Was this translation helpful? Give feedback.
-
|
same issue: any ideas when a fix will be available? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I'm new to celery and experiencing a similar issue using celery+redis. Not sure if this can help but lowering the visibility_timeout to a low number seems to help, as per default the visibility_timeout is set to 3600. I still need to test it out a bit more, but maybe someone more experienced with celery and redis can try it out. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Agree, but it was either this or remove celery and redis from our app. Hope they fix this bug soon. |
Beta Was this translation helpful? Give feedback.
-
|
Can you please elaborate what kubernetes probes did you try? These look like they would suffice, as I'm currently testing locally:
Therefore I think liveness probe like this would do the trick and restart celery pods after connection loss (maybe several times if redis downtime was longer than probe intervals): Ideas source: #4079 . Note that some people are talking about high CPU utilization for this kind of pings, and it might require further tuning. |
Beta Was this translation helpful? Give feedback.
-
|
In my case when testing I found that it didn't work for me, to elaborate, the probes worked fine as intendent but there where still scenarios where the tasks where not executed.
In your case, do you need to restart the celery container after the redis connection is up again or when the connection is lost? Also can you check if your tasks are received again after the connection is up again? |
Beta Was this translation helpful? Give feedback.
-
From what I've tested locally, these pings return error even after the connection to Redis is up again, but worker still does not pick up the tasks from it - so after the connection is up, it should trigger one final (maybe also the only one if Redis downtime was less than probe interval) restart after restoring connection.
To be clear, did you mean "the ping toward Celery"? Because it does not ping Redis per se (but will also fail if Redis is unavailable). Also the cpu usage might be related to how you invoke the command - say, if you do
I don't mind extra restarts during Redis downtime (the worker is not doing anything anyway), but ultimately I want worker to continue executing tasks, so I need a restart after Redis connection is up as well.
I'll try to post back when I have the results. I've also applied |
Beta Was this translation helpful? Give feedback.
-
|
I've tried to implement described checks for two similar projects, in both cases restarts seem to help workers to get back to executing tasks after manually bringing Redis down and back up. Will see how it goes on a distance. Speaking about CPU usage, when worker pings happen I see spikes of about 30 mCPU usage in project A and about 250 mCPU usage in project B. Both use Celery==5.2.7, main differences from the get go is that project A works with Redis 7.0, and project B with Redis 6.2, it also uses Poetry's venv inside its container and invoking new poetry shell for each ping seems to add some overhead (roghly 100 mCPU, although my tests on that were brief). Also invoking pings like |
Beta Was this translation helpful? Give feedback.
-
Thanks for the Very helpful comment BTW |
Beta Was this translation helpful? Give feedback.
-
|
Same issue, any update for resolve? |
Beta Was this translation helpful? Give feedback.
-
|
We are working on a new testing infrastructure that will allow tackling such bugs with a better tool chain. It can already reproduce the error automatically but only once it will be released we'll be able to pay more attention to Redis, as it will be very helpful in resolving the issue inc. automatic tests which we currently don't have nor capable of having until the new infra is ready. |
Beta Was this translation helpful? Give feedback.
-
|
Change to use rabbitmq broker there is no issue |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Same issue here. we use celery in our three projects. but all of them are facing this un-expected error.. we will try replace redis to rabbitmq. however if it not work, we have to think about the usage of celery in real-production |
Beta Was this translation helpful? Give feedback.
-
|
See my response below.
I hope you’ll conclude its best to keep it! Wait around, Celery is going places ;) |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Same issue, we switched from redis to rabbitmq a few weeks ago, still the issue. |
Beta Was this translation helpful? Give feedback.
-
|
You switched to RabbitMQ and the worker stopped consuming tasks after reconnection like with Redis? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. We switched one project for now, and we still have the same issue on redis + rabbitmq projects |
Beta Was this translation helpful? Give feedback.
-
Can you please check if the bug reproduces with v5.4.0rc1? Our new tests shows the bugfix works, for both Redis and RabbitMQ so either the bug is really fixed or our tests aren't good enough. I'd appreciate any feedback. Thank you 🙏 |
Beta Was this translation helpful? Give feedback.
-
|
yes, time to shake the tree on this issue. Been listed for 2 years now. And no rectification. But I suppose removing the Redis into an independent instance that is stable and rarely reset would be the middle ground. |
Beta Was this translation helpful? Give feedback.
-
As I mentioned in a comment above (and a few other places), there is indeed a lot of work behind the scenes. YESTERDAY we’ve reached a significant milestone with the “behind the scenes” effort to improve our testing infrastructure. All in all, I want to clarify that we took a step aside so we can take many more steps forward, and our focus is building the infrastructure that will allow us to deal with many years of issues, not just one at a time, by giving the community the simplicity it needs to contribute. Once our new testing infrastructure is finalized, we’ll be able to focus on the Celery v5.4 release, which I’d hope to also include a fix for this issue as I prefer to avoid just decreasing the official support and fixing the bug instead. |
Beta Was this translation helpful? Give feedback.
-
For example, def test_redis_bug(celery_setup: CeleryTestSetup):
res = add.s(4, 2).apply_async()
assert res.get() == 6
celery_setup.broker.restart()
res = add.s(4, 2).apply_async()
assert res.get(timeout=2) == 6 -------------- smoke_tests_worker@f08c6a53dd65 v5.3.6 (emerald-rush)
--- ***** -----
-- ******* ---- Linux-6.5.11-linuxkit-aarch64-with-glibc2.36 2024-01-15 12:40:59
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: celery_test_app:0xffff94cfac50
- ** ---------- .> transport: redis://352674e72f38:6379/0
- ** ---------- .> results: redis://66857ed93ad6/0
- *** --- * --- .> concurrency: 10 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. pytest_celery.vendors.worker.tasks.ping
. t.integration.tasks.add
. t.integration.tasks.add_chord_to_chord
. t.integration.tasks.add_ignore_result
. t.integration.tasks.add_not_typed
. t.integration.tasks.add_replaced
. t.integration.tasks.add_to_all
. t.integration.tasks.add_to_all_to_chord
. t.integration.tasks.build_chain_inside_task
. t.integration.tasks.chain_add
. t.integration.tasks.chord_add
. t.integration.tasks.collect_ids
. t.integration.tasks.delayed_sum
. t.integration.tasks.delayed_sum_with_soft_guard
. t.integration.tasks.errback_new_style
. t.integration.tasks.errback_old_style
. t.integration.tasks.fail
. t.integration.tasks.fail_replaced
. t.integration.tasks.fail_unpickleable
. t.integration.tasks.identity
. t.integration.tasks.ids
. t.integration.tasks.mul
. t.integration.tasks.print_unicode
. t.integration.tasks.raise_error
. t.integration.tasks.rebuild_signature
. t.integration.tasks.redis_count
. t.integration.tasks.redis_echo
. t.integration.tasks.redis_echo_group_id
. t.integration.tasks.replace_with_chain
. t.integration.tasks.replace_with_chain_which_raises
. t.integration.tasks.replace_with_empty_chain
. t.integration.tasks.replace_with_stamped_task
. t.integration.tasks.replaced_with_me
. t.integration.tasks.retry
. t.integration.tasks.retry_once
. t.integration.tasks.retry_once_headers
. t.integration.tasks.retry_once_priority
. t.integration.tasks.retry_unpickleable
. t.integration.tasks.return_exception
. t.integration.tasks.return_nested_signature_chain_chain
. t.integration.tasks.return_nested_signature_chain_chord
. t.integration.tasks.return_nested_signature_chain_group
. t.integration.tasks.return_nested_signature_chord_chain
. t.integration.tasks.return_nested_signature_chord_chord
. t.integration.tasks.return_nested_signature_chord_group
. t.integration.tasks.return_nested_signature_group_chain
. t.integration.tasks.return_nested_signature_group_chord
. t.integration.tasks.return_nested_signature_group_group
. t.integration.tasks.return_priority
. t.integration.tasks.return_properties
. t.integration.tasks.second_order_replace1
. t.integration.tasks.second_order_replace2
. t.integration.tasks.sleeping
. t.integration.tasks.tsum
. t.integration.tasks.write_to_file_and_return_int
. t.integration.tasks.xsum
. t.smoke.tasks.long_running_task
. t.smoke.tasks.noop
. t.smoke.tasks.replace_with_task
. t.smoke.tasks.self_termination_delay_timeout
. t.smoke.tasks.self_termination_exhaust_memory
. t.smoke.tasks.self_termination_sigkill
. t.smoke.tasks.self_termination_system_exit
/celery/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
[2024-01-15 12:40:59,979: WARNING/MainProcess] /celery/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
[2024-01-15 12:40:59,988: INFO/MainProcess] Connected to redis://352674e72f38:6379/0
Connected to redis://352674e72f38:6379/0
[2024-01-15 12:40:59,988: WARNING/MainProcess] /celery/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
/celery/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
mingle: searching for neighbors
[2024-01-15 12:40:59,990: INFO/MainProcess] mingle: searching for neighbors
[2024-01-15 12:41:00,995: INFO/MainProcess] mingle: all alone
mingle: all alone
smoke_tests_worker@f08c6a53dd65 ready.
[2024-01-15 12:41:01,004: INFO/MainProcess] smoke_tests_worker@f08c6a53dd65 ready.
[2024-01-15 12:41:01,370: INFO/MainProcess] Task t.integration.tasks.add[3db7c8bc-d3d2-4c6e-bb62-04f6fb4783e1] received
Task t.integration.tasks.add[3db7c8bc-d3d2-4c6e-bb62-04f6fb4783e1] received
Task t.integration.tasks.add[3db7c8bc-d3d2-4c6e-bb62-04f6fb4783e1] succeeded in 0.006876874999989013s: 6
[2024-01-15 12:41:01,378: INFO/ForkPoolWorker-8] Task t.integration.tasks.add[3db7c8bc-d3d2-4c6e-bb62-04f6fb4783e1] succeeded in 0.006876874999989013s: 6
[2024-01-15 12:41:01,493: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/celery/celery/worker/consumer/consumer.py", line 340, in start
blueprint.start(self)
File "/celery/celery/bootsteps.py", line 116, in start
step.start(parent)
consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/celery/celery/worker/consumer/consumer.py", line 340, in start
blueprint.start(self)
File "/celery/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/celery/celery/worker/consumer/consumer.py", line 746, in start
c.loop(*c.loop_args())
File "/celery/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/usr/local/lib/python3.11/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
cb(*cbargs)
File "/usr/local/lib/python3.11/site-packages/kombu/transport/redis.py", line 1344, in on_readable
self.cycle.on_readable(fileno)
File "/usr/local/lib/python3.11/site-packages/kombu/transport/redis.py", line 569, in on_readable
chan.handlers[type]()
File "/usr/local/lib/python3.11/site-packages/kombu/transport/redis.py", line 913, in _receive
ret.append(self._receive_one(c))
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kombu/transport/redis.py", line 923, in _receive_one
response = c.parse_response()
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 824, in parse_response
response = self._execute(conn, try_read)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 800, in _execute
return conn.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 49, in call_with_retry
fail(error)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 802, in <lambda>
lambda error: self._disconnect_raise_connect(conn, error),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 789, in _disconnect_raise_connect
raise error
File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 801, in <lambda>
lambda: command(*args, **kwargs),
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 822, in try_read
return conn.read_response(disconnect_on_error=False, push_request=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 500, in read_response
response = self._parser.read_response(disable_decoding=disable_decoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 15, in read_response
result = self._read_response(disable_decoding=disable_decoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 25, in _read_response
raw = self._buffer.readline()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/_parsers/socket.py", line 115, in readline
self._read_from_socket()
File "/usr/local/lib/python3.11/site-packages/redis/_parsers/socket.py", line 68, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
File "/celery/celery/worker/consumer/consumer.py", line 746, in start
c.loop(*c.loop_args())
File "/celery/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/usr/local/lib/python3.11/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
cb(*cbargs)
File "/usr/local/lib/python3.11/site-packages/kombu/transport/redis.py", line 1344, in on_readable
self.cycle.on_readable(fileno)
File "/usr/local/lib/python3.11/site-packages/kombu/transport/redis.py", line 569, in on_readable
chan.handlers[type]()
File "/usr/local/lib/python3.11/site-packages/kombu/transport/redis.py", line 913, in _receive
ret.append(self._receive_one(c))
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kombu/transport/redis.py", line 923, in _receive_one
response = c.parse_response()
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 824, in parse_response
response = self._execute(conn, try_read)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 800, in _execute
return conn.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 49, in call_with_retry
fail(error)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 802, in <lambda>
lambda error: self._disconnect_raise_connect(conn, error),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 789, in _disconnect_raise_connect
raise error
File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 801, in <lambda>
lambda: command(*args, **kwargs),
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 822, in try_read
return conn.read_response(disconnect_on_error=False, push_request=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 500, in read_response
response = self._parser.read_response(disable_decoding=disable_decoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 15, in read_response
result = self._read_response(disable_decoding=disable_decoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 25, in _read_response
raw = self._buffer.readline()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/_parsers/socket.py", line 115, in readline
self._read_from_socket()
File "/usr/local/lib/python3.11/site-packages/redis/_parsers/socket.py", line 68, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
[2024-01-15 12:41:01,496: WARNING/MainProcess] /celery/celery/worker/consumer/consumer.py:391: CPendingDeprecationWarning:
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.
warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
/celery/celery/worker/consumer/consumer.py:391: CPendingDeprecationWarning:
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.
warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
[2024-01-15 12:41:01,497: WARNING/MainProcess] /celery/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
/celery/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
[2024-01-15 12:41:01,498: ERROR/MainProcess] consumer: Cannot connect to redis://352674e72f38:6379/0: Error 111 connecting to 352674e72f38:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)
consumer: Cannot connect to redis://352674e72f38:6379/0: Error 111 connecting to 352674e72f38:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)
[2024-01-15 12:41:03,503: INFO/MainProcess] Connected to redis://352674e72f38:6379/0
Connected to redis://352674e72f38:6379/0
[2024-01-15 12:41:03,503: WARNING/MainProcess] /celery/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
/celery/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
warnings.warn(
[2024-01-15 12:41:03,505: INFO/MainProcess] mingle: searching for neighbors
mingle: searching for neighbors
===================================================================== short test summary info =====================================================================
FAILED t/smoke/tests/test_canvas.py::test_redis_bug[celery_setup_worker-celery_redis_backend] - celery.exceptions.TimeoutError: The operation timed out.
=============================================================== 1 failed, 133 deselected in 26.62s ================================================================ |
Beta Was this translation helpful? Give feedback.
-
|
For comparison, running the same test with RabbitMQ passed: pytest -vvvv t/smoke -k test_redis_bug
======================================================================= test session starts =======================================================================
platform darwin -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0 -- /Users/nusnus/.pyenv/versions/3.12.1/envs/celery_py312/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/nusnus/dev/GitHub/celery
configfile: pyproject.toml
plugins: docker-tools-3.1.3, timeout-2.2.0, order-1.2.0, click-1.1.0, subtests-0.11.0, rerunfailures-13.0, celery-1.0.0b1, xdist-3.5.0
collected 134 items / 133 deselected / 1 selected
t/smoke/tests/test_canvas.py::test_redis_bug[celery_setup_worker-celery_redis_backend] PASSED [100%]
=============================================================== 1 passed, 133 deselected in 50.95s ================================================================ |
Beta Was this translation helpful? Give feedback.
-
|
Just to confirm, if the Redis server is run from outside a typical Docker compose implementation and is reasonably static and rarely restarted, this would largely avoid this issue? I mainly have this issue when a main application, that has the Redis instance as part of the docker-compose file, is restarted, usually because I am updating the code. (every few weeks is common). When I do this I have 2 other containers that run as celery workers based on separate docker-compose files. They will be disconnected on Redis restart and fail to reconnect until I do a complete restart of those containers. (Which I commonly forget to do) So technically, if I rarely stop Redis, I should avoid the problem most of the time. If this is the case, it should be a recommendation in the docs until rectified. |
Beta Was this translation helpful? Give feedback.
-
|
As you said, this is an avoidance and in that case this solution is specific for your environment so I prefer to avoid recommending workarounds and if time permits focus, invest it in solving the issue and not to offer ways to live with it 😉 |
Beta Was this translation helpful? Give feedback.
-
The tree has been shaken! |
Beta Was this translation helpful? Give feedback.
-
|
Celery v5.4.0rc1 is ready for testing! |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately I can still observe that with v5.4.0rc1 workers do not sync with other workers after reconnect and do not pick up new tasks. We have multiple celery workers and redis as part of a docker swarm stack. |
Beta Was this translation helpful? Give feedback.
-
|
HI all, |
Beta Was this translation helpful? Give feedback.
-
|
I also tried upgrading to v5.4.0rc1 but unfortunately the issue still persists for me as well. On reconnection to redis, the workers just hang and do not pick any tasks. The only solution is to restart the workers as with the previous version 5.3.1 i had. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Same problem with This is my import configparser
config = configparser.ConfigParser()
config.read('config.ini')
CELERY_CONFIG = config['celery']
broker_url = CELERY_CONFIG['broker']
result_backend = CELERY_CONFIG['backend']
worker_cancel_long_running_tasks_on_connection_loss = True
broker_connection_retry_on_startup = True
# FIXME: work-around for sudden connection drop on Redis.
# Track this problem on:
# - https://groups.google.com/g/celery-users/c/6yF34oA30Ys
# - https://github.com/celery/celery/discussions/7276
broker_connection_max_retries = None
broker_pool_limit = None
worker_deduplicate_successful_tasks = True
worker_concurrency = 1
worker_prefetch_multiplier = 1
worker_state_db = "state.db"
worker_send_task_events = True
worker_pool = 'prefork'
task_time_limit = 3600My worker starting command are: Running
After my worker stopped consuming task, I tried to
It seems like the worker couldn't close the connection and stuck there. I haven't tried the |
Beta Was this translation helpful? Give feedback.
-
|
We had the same problem i don't know how related it is with the original post.I could reproduce it on our setup, because of a firewall reload could interrupt the network for few seconds. To reproduce the problem:
Based on all the information i could find, I think this is what happens on a network interrupt:
Only thing is don't understand why ping and inspect command still seems to work. Are these broadcasted via a different way? Solution: Default the tcp keepalive is not enabled, this is something that is mentioned in a issue at Redis: redis/redis#7855 I have added the following settings: import socket
CELERY_BROKER_TRANSPORT_OPTIONS = {
"socket_keepalive": True,
"socket_keepalive_options": {
socket.TCP_KEEPIDLE: 60,
socket.TCP_KEEPCNT: 5,
socket.TCP_KEEPINTVL: 10,
},
}If I try to reproduce the problem with these settings, the worker stops processing jobs and after 60 seconds it starts processing jobs again. |
Beta Was this translation helpful? Give feedback.
-
|
Very interesting! @krukas May I ask that you also try this workaround on v5.4.0rc1? Thinking out loud - if this gives a general enough remediation for this infamous bug, maybe it's worth making it the default settings for 5.4.0, at least to release the pressure..
|
Beta Was this translation helpful? Give feedback.
-
|
@Nusnus No, I have only tested this on 5.3.6 Based on issue of Redis I think it should be a good idea to have some default for keepalive. |
Beta Was this translation helpful? Give feedback.
-
|
Sure np - I’ll check it out myself then, hopefully soon enough 🙏 |
Beta Was this translation helpful? Give feedback.
-
|
I am also experiencing this issue. Simple setup: Celery worker, one Redis database as broker, one as results backend, both on the same Redis instance. When restarting Redis, meaning it comes back up within seconds or even less, this is logged: Sometimes, tasks 'work' again, but more often, they don't. Whether they do or don't, seems random. When tasks are received, nothing is logged. Contrary to some other messages in this thread, Celery shuts down gracefully when stopping it. What I've tried: Set
... so it's no surprise it doesn't work, as restarting the single Redis instance causes the broker to become unavailable too. Set keepalive on the socket, according to the configuration at #7276 (reply in thread). That doesn't help either. |
Beta Was this translation helpful? Give feedback.
-
|
I have a related problem, or the same problem but perhaps a different root cause. In a kubernetes deployment, I have a redis pod, a django pod and a second deployment for celery (with autoscale process workers). In my logs I get socket timeout errors often while trying to get a redis lock. Then it all goes wrong. No more tasks are every executed. I have tried everything I can think of to get the worker to retry the redis connection, but the logs do not show any retry behaviour. The redis pod has liveness checking and it never fails. I have resorted to running a one a minute celery beat task which gets the worker to touch a file, and my liveness check in kubernetes checks how fresh that file is. I do not know how to reproduce this error. I have celery 5.3.6 and redis 7.2.3 these are my configuration options: |
Beta Was this translation helpful? Give feedback.
and others
-
I am experiencing an issue with celery==5.2.3 that I did not experience with celery 4.4.7 which I have recently migrated from.
I am using redis (5.0.9) as the message broker. When I manually restart redis, the celery worker from time to time (after the restart of redis) stops consuming tasks indefinitely. Celery beat is able to publish tasks to the broker without any problem after the redis restarts. Once I force a restart of the worker, it will get all the past scheduled tasks by beat.
Only if I run celery 5 worker without heartbeat/gossip/mingle this does not happen and I can restart redis without the worker stopping to consume tasks after it reconnects to it.
I am running the worker with the following options to "make it work":
celery -A proj worker -l info --without-heartbeat --without-gossip --without-mingleWhen I try running celery with rabbitmq as the message broker and with mingle/gossip/heartbeat I cannot reproduce the bug (this only happens with redis). But for the scenario I am using I need to keep using redis.
I have 2 questions:
Logs prior to when it get's stuck. I did wait for half an hour and tasks (periodic task are scheduled every 5 minutes) were not consumed by the worker, then I did hit ctrl+c. There is no logs when it stops consuming messages, it just "freezes":
Celery report
Beta Was this translation helpful? Give feedback.
All reactions