Define DB config earlier for django-celery-results, fix INSTALLED_APPS

[juspence]

@Elkasitu I found two possible issues, but I'm not sure if fixing them will actually make the monitoring email work:

1. INSTALLED_APPS ordering was different compared to Corgi, and django_celery_beat was not present so all the django_celery_beat.models couldn't be imported.
2. CELERY_RESULT_BACKEND was set to "django-db", but Django's DATABASES setting hadn't been defined yet, so maybe django-celery-results didn't know where to store each TaskResult. I'd expect this to raise an error instead of silently failing, but 🤷

[Elkasitu]

I am not convinced that either of these will change anything, as the task results seem to be working locally without django-celery-beat, with the current INSTALLED_APPS order and without the DATABASES setting being defined in the base settings.pyfile.

This seems to be a recurring error [1][2] and there doesn't seem to be a definitive solution, I don't have time for this right now either so let's leave it for Friday.

[1] celery/django-celery-results#19
[2] celery/django-celery-results#102

[juspence]

The docs above mention this is usually a config error. I checked in stage after I manually ran the first email task (which should create its own TaskResult), but TaskResult.objects.count() was still 0.

I don't have a working local OSIDB env set up (and don't have time to fix this unfortunately), so in this PR I just tried to minimize config differences between OSIDB and Corgi (which works with this config).

I do see that the Celery script you run in Kubernetes is almost identical to the Corgi one, and I don't think the differences would explain this bug. However, it looks like in your script you use "-P gevent", but the Celery app config's worker_pool setting is still set to "prefork".

So it looks like either the settings in your run script aren't being applied, or they're being overridden somewhere else. I also see the result_backend is still set to a redis:// URL and not to "django-db" like we have in Corgi. Below shows current OSIDB settings in the Celery worker pod in stage:

>>> from config.celery import app
>>> app.conf.get("worker_pool")
'prefork'
>>> app.conf.get("broker_url")
'rediss://secret@redis:6379/'
>>> app.conf.get("result_backend")
'rediss://secret@redis:6379/'
>>> app.conf.get("result_backend_max_retries")
2
>>> app.conf.get("result_backend_always_retry")
True

Compare with Corgi:

>>> app.conf.get("broker_url")
'redis://redis:6379'
>>> app.conf.get("result_backend")
'django-db'

I can't guarantee this change will fix the issue, but in Corgi we do define the Django database settings before we tell Celery to use Django's DB for storing TaskResults. I'm not familiar enough with OSIDB to know how you apply Celery settings or what else might override the stuff in config/settings.py though.

[Elkasitu]

Closed by #296