Does celery support kafka as broker yet?

[Lunaticf]

I can see there is some PR about kafka support merged. Thanks

[open-collective-bot[bot]]

Hey @Lunaticf 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

[yataosu]

@Lunaticf From the kombu pr celery/kombu#1506, kombu support kafka in https://github.com/celery/kombu/releases/tag/v5.3.0a1.
Maybe Celery will support kafka in future.

[pengtaochang]

Hi @Lunaticf, good news! Kombu supports Kafka, enabling Celery to seamlessly utilize Kafka functionality. I've successfully tested this on my local machine.

Here's my local configuration:

broker_url = "confluentkafka://192.168.16.130:9092/"

To further optimize your setup, consider utilizing the broker_transport_options setting to provide additional configuration to the Kafka client pool. The key features include:

Features:

Type: Virtual
Supports Direct: Yes
Supports Topic: Yes
Supports Fanout: No
Supports Priority: No
Supports TTL: No
Feel free to leverage these features for enhanced performance and flexibility in your Celery setup with Kafka. Let me know if you have any questions or need further assistance!

[pengtaochang]

To get started, make sure to add confluentkafka to your environment by running:

pip install confluent-kafka

[auvipy]

thanks for checking and confirming!

[glivne]

Would you subscribe to a topic with /{topic} at the end of the connection string like vhost in mq?

[glivne]

Nevermind, I found the solution but I'll leave it here for people who need it. A celery Queue = Kafka Topic. If you want a specific topic, task_default_queue={topic} does the trick. not sure what broker_vhost is used for if at all yet.