Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use a group periodically - missing documentation #8909

Open
4 tasks done
jonasN5 opened this issue Mar 13, 2024 · 4 comments
Open
4 tasks done

How to use a group periodically - missing documentation #8909

jonasN5 opened this issue Mar 13, 2024 · 4 comments

Comments

@jonasN5
Copy link

jonasN5 commented Mar 13, 2024

  • I have checked the issues list
    for similar or identical bug reports.
  • I have checked the pull requests list
    for existing proposed fixes.
  • I have checked the commit log
    to find out if the bug was already fixed in the main branch.
  • I have included all related issues and possible duplicate issues in this issue
    (If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Description

In the documentation, there is indication on how to add a single periodic task but I didn't find any indication on how to schedule a periodic group.
Doing sender.add_periodic_task(10, my_group, name='group task name') raises an exception.
So I ended up creating one main task that spawns many subtasks, which is discouraged in the docs:

job = group([subtask.s(item) for item in range(10)])
result = job.apply_async()
ret_val = result.get(disable_sync_subtasks=False)

Suggestions

Provide information on how to schedule a group job periodically.

@vesnikos
Copy link

Where in documentation is that indication?

@jonasN5
Copy link
Author

jonasN5 commented Apr 21, 2024

https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html

This shows how to add a single periodic task.
I'm looking for documentation on how to schedule a periodic group.

@vesnikos
Copy link

vesnikos commented Apr 21, 2024

You can't. Groups/Chords/Chains and others are part of what is called workflows of tasks [0]. beat schedules are designed to initiate a individual task as far as I know.

It's very much possible to use initiating tasks to kick start your workflows:

@task
def op_workflow_a():
  from celery import group
  from my_app.tasks import send_email
  emails = ["...@gmail.com", "...@outlook.com"]
  group_task = group(send_email.s(email=email) for email in emails)
  group_result = group_task.apply_async()
  
  return group_result.id

[0] https://docs.celeryq.dev/en/stable/userguide/canvas.html#groups

@jonasN5
Copy link
Author

jonasN5 commented May 6, 2024

That is already what I'm doing. However, after the group_result = group_task.apply_async() line, I have a group_result.get(disable_sync_subtasks=False) because disable_sync_subtasks is necessary here. What happens if I do not await for completion? Is there a risk one of the subtasks is interrupted/killed depending on the circumstances?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants