Using events in celery batch

[Anghille]

Hi everyone,

I have been working on a module called celery-batch for some days now. We are using this custom Batch task to buffer celery tasks before processing them.

The problem is that this module isn't sending events at all, and all tasks in monitoring tools (flower, or celery-exporter with prometheus) are in PENDING state, even if I see them beeing processed in my code.

Do you have any idea How those events could be added to the celery-batch? (see. https://github.com/clokep/celery-batches).

I tried to use EventDispatcher(), self.app.events.send(xxxx), and other stuff, without success. My goal is to be able to have multiple events : one event for each task in the batch (task-successed, task-failed ...) but also for the batching task itself.

I have tried multiple implementation here : https://github.com/Anghille/celery-batches/tree/failed-%40, but I am not an expert on celery at all. I read the source code of the celery-batch thoroughly, as well as the Celery doc and lots of part of the celery code. But I can't figure how to implement this. There is too many blind spot in my knowledge :(

Any idea, lead or even code that could help me? :) Cheers

[clokep]

I think we can handle this in the issue you already have open: clokep/celery-batches#83.