Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pending task results are missing task name and kwargs #3820

Closed
bjester opened this issue Nov 17, 2022 · 1 comment
Closed

Pending task results are missing task name and kwargs #3820

bjester opened this issue Nov 17, 2022 · 1 comment
Assignees
Labels
DEV: backend P0 - critical Priority: Release blocker or regression

Comments

@bjester
Copy link
Member

bjester commented Nov 17, 2022

Observed behavior

A chef script produced thousands of enqueued calculate_user_storage_tasks for the same user, which overwhelmed the task workers

Expected behavior

For most tasks, any attempts to enqueue the task should only enqueue it if there isn't already a queued or currently executing task for the same arguments

User-facing consequences

Other users actively editing channels experience long delays for operations like copying and publishing, where the UI clearly shows the progress of the operation

Errors and logs

Seems as though the fetch_or_enqueue behavior isn't working properly

INFO:root:Enqueuing task:id calculate_user_storage_task:3a505d01cb9e447f9360158b0dad61e1 for user:channel 2858:None | {'user_id': 2858}
INFO:root:Didn't fetch matching task calculate_user_storage_task for user 2858 | {'user_id': 2858}

Additional information

https://learningequality.slack.com/archives/C0LK8QS9J/p1668640815323059
celery/django-celery-results#334

Despite enabling extended results, the pending result does not have the 'extended' information

@bjester bjester self-assigned this Nov 17, 2022
@bjester bjester added P0 - critical Priority: Release blocker or regression DEV: backend labels Nov 17, 2022
@bjester bjester changed the title Concurrent and repeated attempts to calculate_user_storage_task through cheffing API Pending task results are missing task name and kwargs Nov 17, 2022
@bjester
Copy link
Member Author

bjester commented Nov 18, 2022

Closed by #3822

@bjester bjester closed this as completed Nov 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DEV: backend P0 - critical Priority: Release blocker or regression
Projects
None yet
Development

No branches or pull requests

1 participant