Possible causes of WORKER TIMEOUT

[dbartenstein]

Hello Gunicorn community! 👋

Today WORKER TIMEOUTs happened on our server. There was pretty high load due to many webhook events being processed and after a while Gunicorn workers started to fail with WORKER TIMEOUT (see log below).

Our Gunicorn configuration:
gunicorn --workers 5 --worker-connections=1000 --worker-class=gevent --bind localhost:8000 --disable-redirect-access-to-syslog

Questions:

• Should timeout be set explicitly when using gevent, e.g. --timeout 600?
• Is it wise to set max-requests as recommended by Adam? https://adamj.eu/tech/2019/09/19/working-around-memory-leaks-in-your-django-app/
• Gunicorn + NGINX are running on a virtual server and there was enough memory available - so "Perhaps out of memory?" probably does not point to the the possible solution
• When using gevent + Gunicorn: is the following paragraph of the Gunicorn documentation still relevant? For full greenlet support applications might need to be adapted. When using, e.g., Gevent and Psycopg it makes sense to ensure psycogreen is installed and setup.
  Last update to psycogreen was in 2022.

Grateful for any hints and recommendations! 🙇

/var/log/syslog:

Sep  5 09:38:38 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:38 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043765)
Sep  5 09:38:39 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:39 +0200] [413510] [ERROR] Worker (pid:2043765) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:38:39 prod-main-zemtu gunicorn[2043946]: [2023-09-05 09:38:39 +0200] [2043946] [INFO] Booting worker with pid: 2043946
Sep  5 09:38:58 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:58 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043479)
Sep  5 09:38:58 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:58 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043840)
Sep  5 09:38:59 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:59 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043858)
Sep  5 09:38:59 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:59 +0200] [413510] [ERROR] Worker (pid:2043479) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:38:59 prod-main-zemtu gunicorn[2044005]: [2023-09-05 09:38:59 +0200] [2044005] [INFO] Booting worker with pid: 2044005
Sep  5 09:38:59 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:59 +0200] [413510] [ERROR] Worker (pid:2043840) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:38:59 prod-main-zemtu gunicorn[2044007]: [2023-09-05 09:38:59 +0200] [2044007] [INFO] Booting worker with pid: 2044007
Sep  5 09:38:59 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:38:59 +0200] [413510] [ERROR] Worker (pid:2043858) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:38:59 prod-main-zemtu gunicorn[2044008]: [2023-09-05 09:38:59 +0200] [2044008] [INFO] Booting worker with pid: 2044008
Sep  5 09:39:10 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:39:10 +0200] [413510] [CRITICAL] WORKER TIMEOUT (pid:2043842)
Sep  5 09:39:11 prod-main-zemtu gunicorn[413510]: [2023-09-05 09:39:11 +0200] [413510] [ERROR] Worker (pid:2043842) was sent SIGKILL! Perhaps out of memory?
Sep  5 09:39:11 prod-main-zemtu gunicorn[2044099]: [2023-09-05 09:39:11 +0200] [2044099] [INFO] Booting worker with pid: 2044099

[dbartenstein]

Hello 👋!

After doing quite some research, we today deployed the following configuration on our servers:
gunicorn --workers=5 --worker-connections=1000 --worker-class=gevent --bind=localhost:8000 --name=ZEMTU zemtu.wsgi --disable-redirect-access-to-syslog --timeout=600 --max-requests=5000 --max-requests-jitter=250 --config=gunicorn.conf.py

Additional notes:

• gunicorn.conf.py is almost exactly like https://gist.github.com/jaddison/4506070

I’ll keep you posted about the success of this new configuration.

[akshaybaj]

Hi!.

Did this configuration work without any error?

[dbartenstein]

Yes, for us it this configuration has been working. We only tweaked the config a bit:

gunicorn --env DJANGO_SETTINGS_MODULE=zemtu.settings.production --workers=5 --worker-connections=1000 --worker-class=gevent --bind=localhost:8000 --name=ZEMTU zemtu.wsgi --disable-redirect-access-to-syslog --timeout=600 --max-requests=50000 --max-requests-jitter=3000 --config=zemtu/settings/gunicorn.conf.py

And yes, we have https://github.com/psycopg/psycogreen/ installed.