WorkerLostError billiard.pool in mark_as_worker_lost SIGNAL 11

[vcarrasc]

From time to time Sentry reports the exception below. I'm not sure how to replicate it and which one could be the issue.
There are some mentions that the issue could be related to psycopg2 and psycopg-binary2 but I do not have psycopg2-binary as part of the libraries installed on the server.

WorkerLostError

"""
Traceback (most recent call last):
  File "/home/user-service/.cache/pypoetry/virtualenvs/morta-9TtSrW0h-py3.11/lib/python3.11/site-packages/billiard/pool.py", line 1264, in mark_as_worker_lost
    raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV) Job: 2434.
"""

Process "ForkPoolWorker-11" pid:33013 exited with "signal 11 (SIGSEGV)"

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.3.6 (emerald-rush) kombu:5.3.5 py:3.11.2
            billiard:4.2.0 py-amqp:5.2.0
platform -> system:Linux arch:64bit
            kernel version:6.4.16-linuxkit imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: N/A or Unknown
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

alembic                  1.13.1   A database migration tool for SQLAlchemy.
amqp                     5.2.0    Low-level AMQP client for Python (fork of amqplib).
anyio                    4.3.0    High level compatibility layer for multiple asynchronous event loop implementations
apispec-webframeworks    0.5.2    Web framework plugins for apispec.
asttokens                2.4.1    Annotate AST trees with source code positions
async-timeout            4.0.3    Timeout context manager for asyncio programs
attrs                    23.2.0   Classes Without Boilerplate
bcrypt                   4.1.2    Modern password hashing for your software and your servers
billiard                 4.2.0    Python multiprocessing fork with improvements and bugfixes
black                    23.3.0   The uncompromising code formatter.
blinker                  1.7.0    Fast, simple object-to-object and broadcast signaling
cachetools               5.3.2    Extensible memoizing collections and decorators
celery                   5.3.6    Distributed Task Queue.
celery-once              3.0.1    Allows you to prevent multiple execution and queuing of celery tasks.
certifi                  2024.2.2 Python package for providing Mozilla's CA Bundle.
cffi                     1.16.0   Foreign Function Interface for Python calling C code.
chardet                  3.0.4    Universal encoding detector for Python 2 and 3
charset-normalizer       3.3.2    The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
click                    8.1.7    Composable command line interface toolkit
click-didyoumean         0.3.0    Enables git-like *did-you-mean* feature in click
click-plugins            1.1.1    An extension module for click to enable registering CLI commands via setuptools entry-points.
click-repl               0.3.0    REPL plugin for Click
cycler                   0.12.1   Composable style cycles
decorator                5.1.1    Decorators for Humans
deepdiff                 5.8.1    Deep Difference and Search of any Python object/data.
deprecated               1.2.14   Python @deprecated decorator to deprecate old python classes, functions or methods.
dnspython                2.6.1    DNS toolkit
executing                2.0.1    Get the currently executing AST node of a frame, and other information
faker                    4.18.0   Faker is a Python package that generates fake data for you.
flask                    1.1.2    A simple framework for building complex web applications.
flask-cors               3.0.10   A Flask extension adding a decorator for CORS support
flask-limiter            1.5      Rate limiting for flask applications
flask-shell-ipython      0.4.1    Replace default `flask shell` command by similar command running IPython.
flask-sqlalchemy         2.4.3    Adds SQLAlchemy support to your Flask application.
fonttools                4.49.0   Tools to manipulate font files
google-api-core          2.17.1   Google API client core library
google-auth              2.28.1   Google Authentication Library
google-cloud             0.34.0   API Client library for Google Cloud
google-cloud-core        2.0.0    Google Cloud API client core library
google-cloud-storage     1.42.1   Google Cloud Storage API client library
google-crc32c            1.1.2    A python wrapper of the C library 'Google CRC32C'
google-resumable-media   2.0.2    Utilities for Google Media Downloads and Resumable Uploads
googleapis-common-protos 1.62.0   Common protobufs used in Google APIs
gunicorn                 20.1.0   WSGI HTTP Server for UNIX
h11                      0.14.0   A pure-Python, bring-your-own-I/O implementation of HTTP/1.1
httpcore                 1.0.4    A minimal low-level HTTP client.
httpx                    0.25.2   The next generation HTTP client.
idna                     3.6      Internationalized Domain Names in Applications (IDNA)
importlib-metadata       4.13.0   Read metadata from Python packages
importlib-resources      6.1.2    Read resources from Python packages
iniconfig                2.0.0    brain-dead simple config-ini parsing
ipython                  8.22.1   IPython: Productive Interactive Computing
itsdangerous             2.0.1    Safely pass data to untrusted environments and back.
jedi                     0.19.1   An autocompletion tool for Python that can be used for text editors.
jinja2                   2.11.3   A very fast and expressive template engine.
kiwisolver               1.4.5    A fast implementation of the Cassowary constraint solver
kombu                    5.3.5    Messaging library for Python.
limits                   3.9.0    Rate limiting utilities
lxml                     5.1.0    Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API.
mako                     1.3.2    A super-fast templating language that borrows the best ideas from the existing templating languages.
markupsafe               1.1.1    Safely add untrusted strings to HTML/XML markup.
marshmallow              3.6.0    A lightweight library for converting complex datatypes to and from native Python datatypes.
matplotlib               3.5.0    Python plotting package
matplotlib-inline        0.1.6    Inline Matplotlib backend for Jupyter
msgpack                  1.0.7    MessagePack serializer
mypy                     0.910    Optional static typing for Python
mypy-extensions          0.4.4    Experimental type system extensions for programs checked with the mypy typechecker.
numpy                    1.26.4   Fundamental package for array computing in Python
ollama                   0.1.7    The official Python client for Ollama.
ordered-set              4.1.0    An OrderedSet is a custom MutableSet that remembers its order, so that every
packaging                23.2     Core utilities for Python packages
parso                    0.8.3    A Python Parser
passlib                  1.7.4    comprehensive password hashing framework supporting over 30 schemes
pathspec                 0.12.1   Utility library for gitignore style pattern matching of file paths.
pexpect                  4.9.0    Pexpect allows easy control of interactive console applications.
pgvector                 0.2.5    pgvector support for Python
pillow                   10.2.0   Python Imaging Library (Fork)
platformdirs             4.2.0    A small Python package for determining appropriate platform-specific dirs, e.g. a "user data dir".
pluggy                   1.4.0    plugin and hook calling mechanisms for python
prompt-toolkit           3.0.43   Library for building powerful interactive command lines in Python
protobuf                 4.25.3   
psycopg2                 2.9.9    psycopg2 - Python-PostgreSQL Database Adapter
ptyprocess               0.7.0    Run a subprocess in a pseudo terminal
pure-eval                0.2.2    Safely evaluate AST nodes without side effects
py                       1.11.0   library with cross-python path, ini-parsing, io, code, log facilities
py-cpuinfo               9.0.0    Get CPU info with pure Python
pyasn1                   0.5.1    Pure-Python implementation of ASN.1 types and DER/BER/CER codecs (X.208)
pyasn1-modules           0.3.0    A collection of ASN.1-based protocols modules
pycparser                2.21     C parser in Python
pycryptodome             3.20.0   Cryptographic library for Python
pygments                 2.17.2   Pygments is a syntax highlighting package written in Python.
pymongo                  4.6.1    Python driver for MongoDB <http://www.mongodb.org>
pyotp                    2.9.0    Python One Time Password Library
pyparsing                3.1.1    pyparsing module - Classes and methods to define and execute parsing grammars
pytest                   6.2.5    pytest: simple powerful testing with Python
pytest-benchmark         4.0.0    A ``pytest`` fixture for benchmarking code. It will group the tests into rounds that are calibrated to the chosen timer.
pytest-celery            0.0.0    pytest-celery a shim pytest plugin to enable celery.contrib.pytest
python-dateutil          2.8.2    Extensions to the standard Python datetime module
python-docx              0.8.11   Create and update Microsoft Word .docx files.
python-slugify           4.0.1    A Python Slugify application that handles Unicode
pyyaml                   6.0.1    YAML parser and emitter for Python
redis                    4.6.0    Python client for Redis database and key-value store
reportlab                4.1.0    The Reportlab Toolkit
requests                 2.31.0   Python HTTP for Humans.
restrictedpython         7.0      RestrictedPython is a defined subset of the Python language which allows to provide a program input into a trusted environment.
rsa                      4.9      Pure-Python RSA implementation
sentry-sdk               1.43.0   Python client for Sentry (https://sentry.io)
setuptools-scm           8.0.4    the blessed package to manage your versions by scm tags
six                      1.16.0   Python 2 and 3 compatibility utilities
sniffio                  1.3.1    Sniff out which async library your code is running under
sqlalchemy               1.3.17   Database Abstraction Library
stack-data               0.6.3    Extract data from python stack frames and tracebacks for informative displays
text-unidecode           1.3      The most basic Text::Unidecode port
toml                     0.10.2   Python Library for Tom's Obvious, Minimal Language
traitlets                5.14.1   Traitlets Python configuration system
typer                    0.10.0   Typer, build great CLIs. Easy to code. Based on Python type hints.
typing-extensions        4.10.0   Backported and Experimental Type Hints for Python 3.8+
tzdata                   2024.1   Provider of IANA time zone data
urllib3                  2.2.1    HTTP library with thread-safe connection pooling, file post, and more.
vine                     5.1.0    Python promises.
wcwidth                  0.2.13   Measures the displayed width of unicode strings in a terminal
webargs                  6.1.1    Declarative parsing and validation of HTTP request objects, with built-in support for popular web frameworks, including Flask, Django, Bottle, Tornado, Pyramid, webapp2, Falcon, and aiohttp.
werkzeug                 1.0.1    The comprehensive WSGI web application library.
wrapt                    1.16.0   Module for decorators, wrappers and monkey patching.
xmltodict                0.13.0   Makes working with XML feel like you are working with JSON
zipp                     3.17.0   Backport of pathlib-compatible object wrapper for zip files

Other Dependencies

N/A

[jacklinke]

It's really helpful when you provide as much detail as possible. You left the dependency version information blank and gave no details about your setup. That makes it really hard for others who are unfamiliar with your particular setup to help or diagnose the issue. Are you running on Linux or windows? Are you using recent versions of celery, kombu, etc? Is this running directly on the machine or in docker/docker compose? Help others help you.

---

From what I've read about this issue, it's typically caused by system resource issues (e.g.: using too much memory)

If you are running your workers in docker/docker compose, try running docker stats and see how much of the available cpus & memory your worker(s) are using.

If you temporarily adjust your settings/initialization to limit resource usage (for instance, use concurrency of 1, 1 worker, and no autoscaling if you were using autoscaling), does the problem persist?

[vcarrasc]

Hi @jacklinke,
Thank you for your comment. I will give it a go to what you suggested!
Regarding your note about the dependency version, when I opened the issue I've placed the details below in Celery report. Isn't that the right place?

software -> celery:5.3.6 (emerald-rush) kombu:5.3.5 py:3.11.2
            billiard:4.2.0 py-amqp:5.2.0
platform -> system:Linux arch:64bit
            kernel version:6.4.16-linuxkit imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

[prav2019]

Experiencing the same issue in python 3.8.2 and celery 4.4.7

[vcarrasc]

Seems like this issue might be related #7007