Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reaching maximum value for TaskResult primary key field causes task failures #307

Open
tmarice opened this issue Apr 25, 2022 · 4 comments · May be fixed by #426
Open

Reaching maximum value for TaskResult primary key field causes task failures #307

tmarice opened this issue Apr 25, 2022 · 4 comments · May be fixed by #426

Comments

@tmarice
Copy link

tmarice commented Apr 25, 2022

When using django-celery-results with the Postgres backend, it's possible to reach the maximum integer value for the primary key column, and this causes every task to fail.

We're using Python 3.9.7, Django 2.2.28, Django Celery Results 2.2.0, Postgres 12.

Traceback:

TaskResult.DoesNotExist: TaskResult matching query does not exist.
  File "django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "django/db/models/query.py", line 406, in get
    raise self.model.DoesNotExist(
SequenceGeneratorLimitExceeded: nextval: reached maximum value of sequence "django_celery_results_taskresult_id_seq" (2147483647)

  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
DataError: nextval: reached maximum value of sequence "django_celery_results_taskresult_id_seq" (2147483647)

  File "billiard/pool.py", line 1796, in safe_apply_callback
    fun(*args, **kwargs)
  File "celery/worker/request.py", line 571, in on_failure
    self.task.backend.mark_as_failure(
  File "celery/backends/base.py", line 171, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "celery/backends/base.py", line 482, in store_result
    self._store_result(task_id, result, state, traceback,
  File "django_celery_results/backends/database.py", line 66, in _store_result
    self.TaskModel._default_manager.store_result(
  File "django_celery_results/managers.py", line 46, in _inner
    return fun(*args, **kwargs)
  File "django_celery_results/managers.py", line 168, in store_result
    obj, created = self.using(using).get_or_create(task_id=task_id,
  File "django/db/models/query.py", line 541, in get_or_create
    return self._create_object_from_params(kwargs, params)
  File "django/db/models/query.py", line 575, in _create_object_from_params
    obj = self.create(**params)
  File "django/db/models/query.py", line 422, in create
    obj.save(force_insert=True, using=self.db)
  File "django/db/models/base.py", line 743, in save
    self.save_base(using=using, force_insert=force_insert,
  File "django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "django/db/models/base.py", line 910, in _do_insert
    return manager._insert([self], fields=fields, return_id=update_pk,
  File "django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "cachalot/monkey_patch.py", line 37, in inner
    return original(compiler, *args, **kwargs)
  File "cachalot/monkey_patch.py", line 113, in inner
    return original(write_compiler, *args, **kwargs)
  File "django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "cachalot/monkey_patch.py", line 137, in inner
    return original(cursor, sql, *args, **kwargs)
  File "django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

The fix was to manually ALTER SEQUENCE and ALTER COLUMN to bigint. After this, tasks continued to work as expected.

Although our problem is now solved, I'm not sure if there was a way to prevent this, or at least handle the issue in a better way.
@auvipy
Copy link
Member

auvipy commented Apr 27, 2022

that could be making default to bigautofield in this package

@tmarice
Copy link
Author

tmarice commented May 10, 2022

Django 3.2 allows configuring the type of AutoFields, so it seems there's no need for action from django-celery-results side, aside from perhaps mentioning this as a possible issue in the documentation.

@richardjmarini
Copy link

We run into this problem at least twice a year. It could be solved by just using a uuid field and populating it with a uuid4 value.

browniebroke added a commit to browniebroke/django-celery-results that referenced this issue Mar 20, 2024
@browniebroke
Change default_auto_field to BigAutoField
4d4dc84 
Fix celery#307
@browniebroke browniebroke linked a pull request Mar 20, 2024 that will close this issue
Open
@browniebroke
Copy link
Contributor

browniebroke commented Mar 20, 2024 •
edited

I've changed the default primary keys to BigAutoField in #426

It feels like the primary keys of these models shouldn't be integers, but rather some sort of UUID. That seems like a much difficult change to make as a third party package.

I don't know what mechanisms Django migrations gives that can be applied here. Doing a data migration on tables with millions of rows might take a while to run, so that seems out of question...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Change default_auto_field to BigAutoField browniebroke/django-celery-results
4 participants
@browniebroke @tmarice @auvipy @richardjmarini