fix: run Celery tasks asynchronously in dev mode #928
Conversation
In `tutor dev`, Celery tasks were running eagerly (in-process) rather than being delegated to the worker containers for asynchronous processing. This difference in behavior between `tutor dev` and `tutor local` meant that certain code paths and race-condition bugs would not surface when testing in dev. The cause is that CELERY_ALWAYS_EAGER is set to True in the upstream devstack settings file. That's because Devstack does not support async LMS/CMS workers; it runs all those tasks in-process. The fix is to simply override CELERY_ALWAYS_EAGER to False. We do this in the common settings file for simplicity, but the change should not affect production mode, which has always had CELERY_ALWAYS_EAGER equal False. For unit tests, we override CELERY_ALAYS_EAGER back to True, because many of them are written to assume in-process tasks.
|
I needed this fix in order to test the upcoming Content Libraries V2 changes, which moves all of the update-from-library operations into Celery tasks. FYI @ormsbee. |
There was a problem hiding this comment.
This is a major change. I'm assuming you tested this and it works, right? It will make troubleshooting celery tasks more difficult for developers (no pdb for them). But on the other hand it will get us closer to production. If you think that this will be an overall improvement for developer experience, then please merge.
@regisb Let me check this out. I think devs should be able to set breakpoints in Celery tasks and then interact with them using |
In
tutor dev, Celery tasks were running eagerly (in-process) rather than being delegated to the worker containers for asynchronous processing. This difference in behavior betweentutor devandtutor localmeant that certain code paths and race-condition bugs would not surface when testing in dev.The cause is that CELERY_ALWAYS_EAGER is set to True in the upstream devstack settings file. That's because Devstack does not support async LMS/CMS workers; it runs all those tasks in-process.
The fix is to simply override CELERY_ALWAYS_EAGER to False. We do this in the common settings file for simplicity, but the change should not affect production mode, which has always had CELERY_ALWAYS_EAGER equal False.
For unit tests, we override CELERY_ALAYS_EAGER back to True, because many of them are written to assume in-process tasks.