Redis-py client connection_pool release causes SystemExit and Celery beat to stop scheduling tasks

[keaton-dataprophet]

Version: Celery: 5.3.6, Redis-py 5.0.1 and 5.1.0b3

Platform: Python 3.11.0 on Ubuntu 22.04 / AWS.

Description:

I am using redis as the message broker for celery-beat. The celery beat scheduler will intermittently get into a state where the following exception is raised:

Exception ignored in: 
<function PubSub.__del__ at 0x7f4931cbeac0>
Traceback (most recent call last):
  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/redis/client.py", line 745, in __del__
    
self.reset()
  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/redis/client.py", line 753, in reset
    
self.connection_pool.release(self.connection)
  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/redis/connection.py", line 1224, in release
    
with self._lock:
  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/celery/apps/beat.py", line 158, in _sync
    
service.sync()
  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/celery/beat.py", line 656, in sync
    
self.scheduler.close()
  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/celery/beat.py", line 437, in close
    
self.sync()
  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/django_celery_beat/schedulers.py", line 298, in sync
    
close_old_connections()
  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/django/db/__init__.py", line 56, in close_old_connections
    
for conn in connections.all(initialized_only=True):
 File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/django/utils/connection.py", line 76, in all
    
return [

  File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/django/utils/connection.py", line 80, in <listcomp>
    
if not initialized_only or hasattr(self._connections, alias)

File "/var/dataprophet/connect-web/py3.11.3_venv/lib/python3.11/site-packages/celery/apps/beat.py", line 159, in _sync
    
raise SystemExit()
SystemExit

causing it to stop scheduling tasks but where the systemd service remains in an active state and the service does not restart. Below is a sample of the .service file

[Unit]
Description=Beat
After=network.target

[Service]
Type=simple

Restart=always
User=beat
Group=beat

EnvironmentFile=/etc/default/beat

ExecStart=/bin/sh -c '/var/service/py3.11.3_venv/bin/celery -A config.celery_app \
  beat \
  --pidfile=/run/service/beat.pid \
  --logfile=/var/beat.log \
  --loglevel=${CELERYD_LOG_LEVEL}'

RuntimeDirectory=beat
WorkingDirectory=/var/backend
LogsDirectory=beat


[Install]
WantedBy=multi-user.target

The celery-beat service has to be manually restarted in order for it to start scheduling tasks again.
This behaviour started around the time we updated redis-py from 3.5.2 to 5.0.1 and celery from 4.4.6 to 5.3.6