Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CELERY_RESULT_BACKEND = 'django-cache' is not working when CELERY_CACHE_BACKEND='default' points to a Database #230

Closed
ShivKJ opened this issue Aug 30, 2021 · 2 comments · Fixed by #242
Labels

Comments

@ShivKJ
Copy link

ShivKJ commented Aug 30, 2021

Hi,

I am trying to use Database for CELERY_RESULT_BACKEND and following this doc.

I set the following in settings.py

CELERY_RESULT_BACKEND = 'django-cache'
CELERY_CACHE_BACKEND = 'default'

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": 'redis://localhost/0',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            'MAX_ENTRIES': 1000
        },
    }
}

results are getting saved in redis correctly.

However, when I use Database in caching,

CELERY_RESULT_BACKEND = 'django-cache'
CELERY_CACHE_BACKEND = 'default'

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
        'LOCATION': 'cacher',
    }
}

then I get the following error (after doing django migration),

[ ERROR/MainProcess] Pool callback raised exception: TypeError('ord() expected string of length 1, but int found')
Traceback (most recent call last):
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/billiard/pool.py", line 1796, in safe_apply_callback
    fun(*args, **kwargs)
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/celery/worker/request.py", line 571, in on_failure
    self.task.backend.mark_as_failure(
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/celery/backends/base.py", line 171, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/celery/backends/base.py", line 482, in store_result
    self._store_result(task_id, result, state, traceback,
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/celery/backends/base.py", line 903, in _store_result
    current_meta = self._get_task_meta_for(task_id)
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/celery/backends/base.py", line 925, in _get_task_meta_for
    meta = self.get(self.get_key_for_task(task_id))
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/django_celery_results/backends/cache.py", line 18, in get
    return self.cache_backend.get(key)
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/django/core/cache/backends/db.py", line 51, in get
    return self.get_many([key], version).get(key, default)
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/django/core/cache/backends/db.py", line 59, in get_many
    self.validate_key(key)
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/django/core/cache/backends/base.py", line 249, in validate_key
    for warning in memcache_key_warnings(key):
  File "/home/abc/.virtualenvs/exp/lib/python3.9/site-packages/django/core/cache/backends/base.py", line 287, in memcache_key_warnings
    if ord(char) < 33 or ord(char) == 127:
TypeError: ord() expected string of length 1, but int found

Is this the expected behavior with Database as backend cache?


Please note that if I use,

CELERY_RESULT_BACKEND = 'django-db'

Then it would work, as django_celery_results creates tables for it and results will be stored correctly. But this method is different from that of using django-cache

@badeendjuh
Copy link
Contributor

I'm running into the same error. From my experimenting I have found so far that:

  • The problem occurs only when using CELERY_RESULT_BACKEND='django-cache' and the django cache driver django.core.cache.backends.db.DatabaseCache.
  • Naming the cache entry anything other then 'default' does not change the error.
  • Using a different cache driver like django.core.cache.backends.filebased.FileBasedCache works fine
  • The stack trace is about retrieving the results but the problem seems to be that no results are stored to begin with. The cache table always remains empty.
  • Setting and getting cache items using the Django Cache API directly works fine.

@badeendjuh
Copy link
Contributor

badeendjuh commented Oct 11, 2021

I've narrowed down the problem to django_celery_results/backends/cache.py. The CacheBackend class feeds a key item to get/set that is of type byte. The Django Cache API does not except this, it wants a str type key.

The following code django_celery_results/backends/cache.py fixes it for me.
This is similar to a fix done for CoucheDB: celery/celery#6781

"""Celery cache backend using the Django Cache Framework."""
from __future__ import absolute_import, unicode_literals

from django.core.cache import cache as default_cache, caches
from kombu.utils.encoding import bytes_to_str
from celery.backends.base import KeyValueStoreBackend


class CacheBackend(KeyValueStoreBackend):
   """Backend using the Django cache framework to store task metadata."""

   def __init__(self, *args, **kwargs):
       super(CacheBackend, self).__init__(*args, **kwargs)

       # Must make sure backend doesn't convert exceptions to dict.
       self.serializer = 'pickle'

   def get(self, key):
       key = bytes_to_str(key)
       return self.cache_backend.get(key)

   def set(self, key, value):
       key = bytes_to_str(key)
       self.cache_backend.set(key, value, self.expires)

   def delete(self, key):
       key = bytes_to_str(key)
       self.cache_backend.delete(key)

   def encode(self, data):
       return data

   def decode(self, data):
       return data

   @property
   def cache_backend(self):
       backend = self.app.conf.cache_backend
       return caches[backend] if backend else default_cache

I will see if I can put this is a pull request, but no promises because I'm not familiar with the github pull mechanism

auvipy pushed a commit that referenced this issue Oct 11, 2021
* Use string values for django-chache keys.

When django cache backend tests the key provided by django_celery_results, It wants to have str type value. 
byte values result in the following exception being raised:

  File "/opt/exaboard/eXaboard/lib/python3.6/site-packages/django/core/cache/backends/base.py", line 287, in memcache_key_warnings
    if ord(char) < 33 or ord(char) == 127:
TypeError: ord() expected string of length 1, but int found

* Remove old super call argument

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add test that saves key value in byte format

* Syntax fix for test_convert_key_from_byte_to_str

* Fixes based on flake8 failures

* Fix t/unit/backends/test_cache.py:64:50: W291 trailing whitespace

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@auvipy auvipy added the bug label Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants