Skip to content

Commit

Permalink
[FIX] queue_job: runner - filedescriptor out of range in select
Browse files Browse the repository at this point in the history
Use the most efficient Selector implementation available on the current platform

Odoo supports only SelectSelector but it is a little obsolete

python >= 3.4 supports a new high-level library Selectors:
 - https://docs.python.org/es/3/library/selectors.html

It could to auto-choose the following ones:
 - SelectSelector
 - PollSelector
 - EpollSelector
 - DevpollSelector
 - KqueueSelector

Using the DefaultSelector class the most efficient implementation available on the current platform will be use:
 - https://docs.python.org/3/library/selectors.html#selectors.DefaultSelector

It helps to support better the resources of the system

Using SelectSelector you are not able to run workers >=255

If you set `ulimit -n 10240` and run `odoo-bin --workers=255`
the following error is raised:

    Traceback (most recent call last):
    File "odoo/service/server.py", line 926, in run
        self.sleep()
    File "odoo/service/server.py", line 852, in sleep
        sel.select(self.beat)
    File "python3.8/lib/python3.8/selectors.py", line 323, in select
        r, w, _ = self._select(self._readers, self._writers, [], timeout)
    ValueError: filedescriptor out of range in select()

But using PollSelector it is not reproduced even using more workers

Most of platform supports PollSelector but using DefaultSelector we can be sure
that even too old system are supported too

And using this High-level library will allow to use the future new improvements

e.g. Epoll has better performance improvements

More info about:
 - https://devarea.com/linux-io-multiplexing-select-vs-poll-vs-epoll
 - redis/redis-py#486
 - odoo/odoo#84684
  • Loading branch information
moylop260 authored and gurneyalex committed Jun 28, 2023
1 parent 1f52367 commit 34269ee
Showing 1 changed file with 12 additions and 4 deletions.
16 changes: 12 additions & 4 deletions queue_job/jobrunner/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@
import datetime
import logging
import os
import select
import selectors
import threading
import time
from contextlib import closing, contextmanager
Expand All @@ -162,6 +162,8 @@

_logger = logging.getLogger(__name__)

select = selectors.DefaultSelector


# Unfortunately, it is not possible to extend the Odoo
# server command line arguments, so we resort to environment variables
Expand Down Expand Up @@ -485,10 +487,16 @@ def wait_notification(self):
# probably a bug
_logger.debug("select() timeout: %.2f sec", timeout)
if timeout > 0:
conns, _, _ = select.select(conns, [], [], timeout)
if conns and not self._stop:
for conn in conns:
conn.poll()
with select() as sel:
for conn in conns:
sel.register(conn, selectors.EVENT_READ)
events = sel.select(timeout=timeout)
for key, _mask in events:
if key.fileobj == self._stop_pipe[0]:
# stop-pipe is not a conn so doesn't need poll()
continue
key.fileobj.poll()

def stop(self):
_logger.info("graceful stop requested")
Expand Down

0 comments on commit 34269ee

Please sign in to comment.