celery 5.0.0 with Django==3.0.10 emits CDeprecationWarning. Following the advise breaks the app

[bastbnl]

Congrats on the v5 release 👏 and thank you for maintaining such a wonderful piece of software. Much appreciated.

I updated from celery==4.4.7 to celery==5.0.0 today and I'm using celery with Django 3.0.10. Celery produces this message - and others - after starting:

[2020-09-25 11:31:17,362: WARNING/MainProcess] /var/www/brwnppr/dev/env/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning:
    The 'CELERY_TASK_SERIALIZER' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_serializer instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

[2020-09-25 11:31:17,364: WARNING/MainProcess] /var/www/brwnppr/dev/env/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning:
    The 'CELERYBEAT_SCHEDULE' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the beat_schedule instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

[2020-09-25 11:31:17,364: WARNING/MainProcess] /var/www/brwnppr/dev/env/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning:
    The 'CELERY_DEFAULT_QUEUE' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_default_queue instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

[2020-09-25 11:31:17,365: WARNING/MainProcess] /var/www/brwnppr/dev/env/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning:
    The 'CELERY_ACCEPT_CONTENT' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the accept_content instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

[2020-09-25 11:31:17,365: WARNING/MainProcess] /var/www/brwnppr/dev/env/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning:
    The 'BROKER_URL' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the broker_url instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

[2020-09-25 11:31:17,366: WARNING/MainProcess] /var/www/brwnppr/dev/env/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning:
    The 'CELERY_RESULT_SERIALIZER' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the result_serializer instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

[2020-09-25 11:31:17,367: WARNING/MainProcess] /var/www/brwnppr/dev/env/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning:
    The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the result_serializer instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

[2020-09-25 11:31:17,367: WARNING/MainProcess] /var/www/brwnppr/dev/env/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning:
    The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the result_backend instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

[2020-09-25 11:31:17,367: WARNING/MainProcess] Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.

I ran the celery upgrade command on the Django project settings and it worked beautifully. Unfortunately the application appears to block after the upgrade command and I think that blocking is caused by celery falling back to amqp while the project I'm working on uses redis as the broker. The Django application itself blocks due to it trying to starting a task which fails for the same reason.

Reverted the settings - thanks for keeping a .orig and it's up and running again.

Now I have been looking through the code to see if I could come up with a PR to suppress this message, but that appears to be kind of hard due to the way the configuration is processed. I'll see if I can come up with something.

[craigds]

also getting this (with django 2.2.15)

/venv/lib/python3.7/site-packages/celery/app/utils.py:206: CDeprecationWarning: 
    The 'CELERY_TIMEZONE' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the timezone instead

  alternative=f'Use the {_TO_NEW_KEY[setting]} instead')

for all settings. That particular setting is defined like

CELERY_TIMEZONE = 'UTC'

Running celery upgrade settings --django on the settings file produces no changes.

[auvipy]

task_serializer is suggested from celery 4.0 which you are not using. not the caps CELERY_TASK_SERIALIZER

[craigds]

We're using the django-style configuration suggested in the docs, which has uppercase names prefixed with CELERY_: https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html

The uppercase name-space means that all Celery configuration options must be specified in uppercase instead of lowercase, and start with CELERY_, so for example the task_always_eager setting becomes CELERY_TASK_ALWAYS_EAGER, and the broker_url setting becomes CELERY_BROKER_URL. This also applies to the workers settings, for instance, the worker_concurrency setting becomes CELERY_WORKER_CONCURRENCY.

Perhaps I should have led with that, I guess I assumed it. Code looks like:

app.config_from_object('django.conf:settings', namespace='CELERY')

[auvipy]

We're using the django-style configuration suggested in the docs, which has uppercase names prefixed with CELERY_: https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html

The uppercase name-space means that all Celery configuration options must be specified in uppercase instead of lowercase, and start with CELERY_, so for example the task_always_eager setting becomes CELERY_TASK_ALWAYS_EAGER, and the broker_url setting becomes CELERY_BROKER_URL. This also applies to the workers settings, for instance, the worker_concurrency setting becomes CELERY_WORKER_CONCURRENCY.

Perhaps I should have led with that, I guess I assumed it. Code looks like:

app.config_from_object('django.conf:settings', namespace='CELERY')

Does this seem to be inconsistent? @thedrow @clokep

[thedrow]

This is indeed a problem.
The code which detects the deprecated settings does not distinguish between Django style settings and Celery's deprecated settings.
This is exactly one of the reasons I didn't remove support for the deprecated setting names.

celery/celery/app/utils.py

Lines 233 to 287 in f05e82a

	def detect_settings(conf, preconf=None, ignore_keys=None, prefix=None,
	all_keys=None, old_keys=None):
	preconf = {} if not preconf else preconf
	ignore_keys = set() if not ignore_keys else ignore_keys
	all_keys = SETTING_KEYS if not all_keys else all_keys
	old_keys = _OLD_SETTING_KEYS if not old_keys else old_keys
	source = conf
	if conf is None:
	source, conf = preconf, {}
	have = set(source.keys()) - ignore_keys
	is_in_new = have.intersection(all_keys)
	is_in_old = have.intersection(old_keys)
	info = None
	if is_in_new:
	# have new setting names
	info, left = _settings_info, is_in_old
	if is_in_old and len(is_in_old) > len(is_in_new):
	# Majority of the settings are old.
	info, left = _old_settings_info, is_in_new
	if is_in_old:
	# have old setting names, or a majority of the names are old.
	if not info:
	info, left = _old_settings_info, is_in_new
	if is_in_new and len(is_in_new) > len(is_in_old):
	# Majority of the settings are new
	info, left = _settings_info, is_in_old
	else:
	# no settings, just use new format.
	info, left = _settings_info, is_in_old
	if prefix:
	# always use new format if prefix is used.
	info, left = _settings_info, set()
	# only raise error for keys that the user didn't provide two keys
	# for (e.g., both ``result_expires`` and ``CELERY_TASK_RESULT_EXPIRES``).
	really_left = {key for key in left if info.convert[key] not in have}
	if really_left:
	# user is mixing old/new, or new/old settings, give renaming
	# suggestions.
	raise ImproperlyConfigured(info.mix_error.format(renames='\n'.join(
	FMT_REPLACE_SETTING.format(replace=key, with_=info.convert[key])
	for key in sorted(really_left)
	)))
	preconf = {info.convert.get(k, k): v for k, v in preconf.items()}
	defaults = dict(deepcopy(info.defaults), **preconf)
	return Settings(
	preconf, [conf, defaults],
	(_old_key_to_new, _new_key_to_old),
	deprecated_settings=is_in_old,
	prefix=prefix,
	)

We can adjust it to ensure it doesn't warn incorrectly, though I'm not sure how currently.

[thedrow]

Running celery upgrade settings --django on the settings file produces no changes.

@craigds TBH, I'm not familiar with what the command does in detail.
I do believe that if the command makes no changes, then we should not be issuing a deprecation warning for those settings.

[bastbnl]

@thedrow I was thinking about that and I found a couple of possible fixes. The first is really simple and it just doesn't invoke maybe_warn_deprecated_settings (https://github.com/celery/celery/blob/v5.0.0/celery/apps/worker.py#L144) when app._config_source == 'django.conf:settings'. I didn't really like it, but it would work

The next, and that's the one I'm currently working on, is a fix where instead of using django.conf:settings you would use something like celery.contrib.django:settings which takes care of a lot of things, including changing the case on the keys. A difficulty with this approach is the prefix or namespace is not easily shared with the module.

[thedrow]

At this point I prefer quick solutions as we are working on our configuration subsystem as well.

The first solution you suggested certainly works.

[bastbnl]

@thedrow I added PR #6385 containing the really simple "fix"

[asottile-sentry]

is there an eventual "fix" for this? will capitalized settings be supported indefinitely?

I'm still seeing this warning in celery==5.3.5 and django==3.2.23 and applying the suggestion breaks django due to it requiring upper-cased settings