Can we use django celery results for remote workers?

[guillaumeldc]

Greetings!

I'm encountering an issue with my Django app that I'm hoping to get some assistance with. I've noticed that when I connect a remote worker that doesn't have access to my Django app configuration, the result cannot be stored correctly. Strangely enough, everything seems to be working perfectly fine when I run everything locally.

I was wondering if anyone has had success with a similar setup and could provide some guidance or resources. Is there any documentation out there that might be useful in my situation?
I just need to know what setting I need to enter for my remote worker configuration. Currently using postgresql that can be opened to outside connections if needed.

I would greatly appreciate any help or insights you might have on this matter. Thank you in advance for your time and attention.

Best regards.
GL.

[urzbs]

Hey there,
I was woundering the same and its not really well documented how to do this with remote workers.
For me it seems django-celery-results was developed with local workers in mind.

When i was using django-celery-results I was hoping that the results that are updated in the redis_backend would be sychronized with the django-db but that is not the case.

However as a workaround:
I have created a blank django project, that does only contain the database credentials, aswell an python environment with

dependencies:
django, django-celery-results, celery

settings.py:

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "django_celery_results",
]

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'my_db',
        'USER': 'postgres',
        'PASSWORD': 'pw',
        'HOST': '127.0.0.1',
        'PORT': '5432',
    }
}

you can start pure celery workers and the worker will update the results into the django-db of your given db-connection from settings.py

# app.py
from __future__ import absolute_import, unicode_literals
from django.conf import settings
from celery import Celery
import os

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings") 

app = Celery('myapp', broker='redis://localhost:6379/0', backend='django-db://')

app.autodiscover_tasks(["path.to.taskfile",])

There is maybe a better way of doing this.

[urzbs]

As a small followup, I have reduced almost everything from the created django-project.
you dont even need the manage py

this is my structure now:

.
├── celery_runner
│   ├── app.py
│   └── tasks.py
└── config
    └── settings
        └── local.py

settings.py (local.py) - but a lot of stuff removed

INSTALLED_APPS = [
    "django_celery_results",
]

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'my_db',
        'USER': 'postgres',
        'PASSWORD': 'dev',
        'HOST': '127.0.0.1',
        'PORT': '5432',
    }
}

requirements.txt

amqp==5.2.0
asgiref==3.7.2
async-timeout==4.0.3
billiard==4.2.0
celery==5.3.6
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
Django==5.0.1
django-celery-results==2.5.1
django-redis==5.4.0
kombu==5.3.5
prompt-toolkit==3.0.43
psycopg2-binary==2.9.9
python-dateutil==2.8.2
redis==5.0.1
six==1.16.0
sqlparse==0.4.4
typing_extensions==4.9.0
tzdata==2023.4
vine==5.1.0
wcwidth==0.2.13

so when using backend='django-db://' in your Celery setting, it will use your installed version of django-celery-results with given database info