Celery using defaults URL while connecting to redis over TLS

[suryan-s]

Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Issue

While trying to connect my redis instance in Azure to my django celery instance, even after setting the correct backend and broker URL, they are not being used and is set to the default url of redis://redis:6379/0. The given below is the celery code i used in the django:

from __future__ import absolute_import, unicode_literals

import os
from celery import Celery
from django.conf import settings
from dotenv import load_dotenv

load_dotenv()

REDIS_ACCESS_KEY = os.getenv('REDIS_ACCESS_KEY')
REDIS_HOST = os.getenv('REDIS_HOST')
REDIS_PORT = os.getenv('REDIS_PORT')
REDIS_DB = os.getenv('REDIS_DB')

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'TestProject.settings')

broker_url = f'rediss://:{REDIS_ACCESS_KEY}@{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}?ssl_cert_reqs=required'
result_backend = f'rediss://:{REDIS_ACCESS_KEY}@{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}?ssl_cert_reqs=required'
print(f"broker_url: {broker_url}")
print(f"result_backend: {result_backend}")

settings.CELERY_BROKER_URL = broker_url
settings.CELERY_RESULT_BACKEND = result_backend
app = Celery('TestProject',
             broker=broker_url,
             backend=result_backend,
             include=['ReservationManager.tasks'],
             queue='central',
             timezone='Asia/Kolkata',
             broker_heartbeat=10,
             backend_heartbeat=10,
             broker_connection_max_retries=100,
             broker_connection_timeout=30,
             broker_connection_retry=True
             )
# app.config_from_object("django.conf:settings")

print(f"CELERY_BROKER_URL: {app.conf.broker_url}")
print(f"CELERY_RESULT_BACKEND: {app.conf.result_backend}")

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


@app.task(name='ping_central')
def ping_central_server():
    """
    This function is used to test the celery task in the main server.
    """
    return {'status': 'pong'}

i have commented the app.config_from_object("django.conf:settings") as im using all the neccessary configs in this file itself. But my debug results is showing that the URLs i have provided are not being used and instead the default one is being used:

celery -A TestProject worker -l info broker_connection_retry_on_startup=True
broker_url: rediss://:<access_key>@<host>/0?ssl_cert_reqs=required
result_backend: rediss://:<access_key>@<host>/0?ssl_cert_reqs=required
CELERY_BROKER_URL: redis://redis:6379/0
CELERY_RESULT_BACKEND: redis://redis:6379/0
 
 -------------- celery@s-suryan-82FG v5.3.6 (emerald-rush)
--- ***** ----- 
-- ******* ---- Linux-6.5.0-26-generic-x86_64-with-glibc2.38 2024-04-01 22:19:24
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         TestProject:0x7d281bc1d550
- ** ---------- .> transport:   redis://redis:6379/0
- ** ---------- .> results:     redis://redis:6379/0
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . ping_central

[2024-04-01 22:19:26,933: ERROR/MainProcess] consumer: Cannot connect to redis://redis:6379/0: Error -3 connecting to redis:6379. Temporary failure in name resolution..
Trying again in 2.00 seconds... (1/100)

I tried many ways to assign the values of the broker and backend URL. eventually i created another python file outside of the django project with this code:

from celery import Celery

broker_url = rediss://:<access_key>@<host>/0?ssl_cert_reqs=required
result_backend = rediss://:<access_key>@<host>/0?ssl_cert_reqs=required

app = Celery(
    broker_url=broker_url,
    backend=result_backend,
    queue='micro1',
    timezone='Asia/Kolkata',
    broker_heartbeat=10,
    backend_heartbeat=10,
    broker_connection_max_retries=100,
    broker_connection_timeout=30,
    broker_connection_retry=True,
    broker_pool_limit=10000,
    broker_connection_retry_on_startup=True
)


@app.task(name='test')
def test(x, y):
    print('The sum received is {}'.format(x + y))
    return x + y

surprisingly this worked perfectly fine. And this is the debug info:

celery -A consumer worker -l info broker_connection_retry_on_startup=True
 
 -------------- celery@s-suryan-82FG v5.3.6 (emerald-rush)
--- ***** ----- 
-- ******* ---- Linux-6.5.0-26-generic-x86_64-with-glibc2.38 2024-04-01 22:24:23
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         __main__:0x7afcf7924610
- ** ---------- .> transport:   rediss://:**@<host>:6380/0
- ** ---------- .> results:     rediss://:**@<host>:6380/0
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . test

[2024-04-01 22:24:24,391: INFO/MainProcess] Connected to rediss://:**@<host>:6380/0
[2024-04-01 22:24:25,077: INFO/MainProcess] mingle: searching for neighbors
[2024-04-01 22:24:26,967: INFO/MainProcess] mingle: all alone
[2024-04-01 22:24:28,405: INFO/MainProcess] celery@s-suryan-82FG ready.

i tried printing the env values in the django side program and everything is working fine. But dont know why its going back to using the default URL.

this is the dependecy details:

aiohttp                       3.9.3
aiosignal                     1.3.1
amqp                          5.2.0
asgiref                       3.7.2
attrs                         23.2.0
billiard                      4.2.0
celery                        5.3.6
certifi                       2024.2.2
cffi                          1.16.0
charset-normalizer            3.3.2
click                         8.1.7
click-didyoumean              0.3.0
click-plugins                 1.1.1
click-repl                    0.3.0
cryptography                  42.0.2
Django                        5.0.2
django-cors-headers           4.3.1
django-oauth-toolkit          2.3.0
django-redis                  5.4.0
djangorestframework           3.14.0
djangorestframework-simplejwt 5.3.1
frozenlist                    1.4.1
gevent                        23.9.1
greenlet                      3.0.3
gunicorn                      21.2.0
idna                          3.6
jwcrypto                      1.5.3
kombu                         5.3.5
multidict                     6.0.5
oauthlib                      3.2.2
packaging                     23.2
pika                          1.3.2
pip                           24.0
prompt-toolkit                3.0.43
psycopg2-binary               2.9.9
pycparser                     2.21
PyJWT                         2.8.0
python-dateutil               2.8.2
python-dotenv                 1.0.1
pytz                          2024.1
redis                         5.0.0
requests                      2.31.0
setuptools                    69.0.3
six                           1.16.0
sqlparse                      0.4.4
typing_extensions             4.9.0
tzdata                        2023.4
urllib3                       2.2.0
vine                          5.1.0
wcwidth                       0.2.13
wheel                         0.42.0
whitenoise                    6.6.0
yarl                          1.9.4
zope.event                    5.0
zope.interface                6.1

[aayushostwal]

Observed this same issue w.r.t to CELERY_BROKER_URL
What I found is when you configure celery, whatever args you pass about broker and reds host, it creates a preconf (https://github.com/celery/celery/blob/main/celery/app/base.py#L272)

Then it initialise Setting objects for celery in which its not using the preconf, instead using env variables directly. (https://github.com/celery/celery/blob/main/celery/app/utils.py#L70)

I have tried avoiding naming the env variables as
CELERY_BROKER_URL
CELERY_RESULT_BACKEND
or any such variables which are present in Setting configuration. This worked

[suryan-s]

Observed this same issue w.r.t to CELERY_BROKER_URL What I found is when you configure celery, whatever args you pass about broker and reds host, it creates a preconf (https://github.com/celery/celery/blob/main/celery/app/base.py#L272)

Then it initialise Setting objects for celery in which its not using the preconf, instead using env variables directly. (https://github.com/celery/celery/blob/main/celery/app/utils.py#L70)

I have tried avoiding naming the env variables as CELERY_BROKER_URL CELERY_RESULT_BACKEND or any such variables which are present in Setting configuration. This worked

Thank you @aayushostwal , the error I was getting was similar except I had a similar variable in my .env file. Since both the .env and settings had the same variable, the one from the env was used. Took a long time until I realised the issue. .so that solved the issue.