-
-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
celery ping doesn't work when using SQS #6727
Comments
Hey @WhyNotHugo 👋, We also offer priority support for our sponsors. |
@celery/aws-task-force It seems like SQS doesn't support gossip and no PID queues are being created. |
@thedrow @WhyNotHugo It does not make sense to have a separate queue for gossip, it's not like with Rabbit/Redis where you set up a single instance supporting multiple queues. |
I think we should not close this but instead reopen and emit a proper error message. |
Per my comment on celery/kombu#1337 [0], we may be able to begin work on a change in celery to avoid doing gossip stuff if the transport being used only support direct queues (which I am assuming is a valid property proxy for transports which can't make an extra queue for gossip purposes). If someone has brainspace to do so, it would be useful to start thinking about what code in celery should change to avoid doing gossip in such a situation. |
I'm still a bit confused on the state of the issue. I see that
However, I'm using |
As far as I know ping is an inspect command which goes through the PIDBox. |
@thedrow Yeah, but is there an alternate celery command to check health status of celery worker on the same host itself? I understand it's a doubt unrelated to this issue, but it'd be awesome if you can help. Right now, I'm unable to add health checks to the worker container (on Kubernetes) since I use SQS broker for tasks. |
Hi! |
up |
Related #4079 |
Checklist
master
branch of Celery.This has already been asked to the discussion group first.It looks like I need a Google account for that. Do I really need to sign up there first?contribution guide
on reporting bugs.
for similar or identical bug reports.
for existing proposed fixes.
to find out if the bug was already fixed in the master branch.
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
celery -A proj report
in the issue.(if you are not able to do this, then at least specify the Celery
version affected).
master
branch of Celery.pip freeze
in the issue.to reproduce this bug.
Optional Debugging Information
and/or implementation.
result backend.
broker and/or result backend. Note: it only happens on SQS
ETA/Countdown & rate limits disabled.
and/or upgrading Celery and its dependencies. Note: note sure how far back I should retest
Related Issues and Possible Duplicates
Couldn't find any directly related.
Related Issues
Couldn't find any directly related.
Possible Duplicates
Couldn't find any directly related.
Environment & Settings
Celery version:
5.0.5 (singularity)
celery report
Output:Steps to Reproduce
celery --app myapp inspect ping
Required Dependencies
Python Packages
pip freeze
Output:Other Dependencies
The issue only seems to happen on SQS.
Minimally Reproducible Test Case
Given that this only happens when running SQS, a minimal example it pretty challenging, since the code needs to be set up in a way that it can run on AWS. The issue is also when a ping is sent, so you'd need a server running and to run a command on the same host, right?
Expected Behavior
Celery ping should work as with RabbitMQ. It should send the message through the broken, and the worker should reply.
Actual Behavior
Celery attempts to use another, inexitent queue for the message:
I've tried various ways of routing including:
However, the message still always go to a new, randomly-named, inexistent queue. I can neither create the queue ahead of time (the name is new each time), and I also can't seem to route ping to go through a specific queue (the default).
I've looked online a lot, but issues discussing
celery ping
,sqs
or routing never overlap so as to cover this. I tried looking at the code both in Celery and Kombu, but can't even quite figure out where the queue name comes from or if (how?) it can be overridden.I've found no workaround, so it seems that
celery inspect ping
can't work with SQS...?The text was updated successfully, but these errors were encountered: