EOFError when using multiprocessing in 3.1.0 #7802

robsan00 · 2020-06-08T15:22:55Z

Describe the bug
When I am using multiprocessing (option "-j auto") in my local repositories, at least under Linux I am getting pickle errors. I tried to reproduce this with a simple test repository (creating 20 dummy files), but that was not enough to reproduce the error. Maybe I need 100s of files or cross references or similar to get the error.

Anyway, the error:

Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/util/parallel.py", line 91, in _process
    pipe.send((failed, collector.logs, ret))
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle traceback objects

Exception occurred:
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
The full traceback has been saved in /tmp/sphinx-err-4oh9kpix.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Process Process-6:
(...)

The content in the logfile:

# Sphinx version: 3.1.0
# Python version: 3.6.7 (CPython)
# Docutils version: 0.14 
# Jinja2 version: 2.10
# Last messages:
#   reading sources... [ 89%] pyprep/pyprep.mode .. test_pyframe/test_pyframe.test_sigop.test_siggen
#   
#   reading sources... [ 94%] test_pyframe/test_pyframe.test_sigop.test_sigproc .. tools/tools.device_setup.prep_device
#   
#   reading sources... [100%] tools/tools.errorcheck .. tools/tools.upgrade_pacmon
#   
#   
#   
#   waiting for workers...
#   
# Loaded extensions:
#   sphinx.ext.mathjax (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /home/robsan00/venv/python3/lib/python3.6/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.type_comment (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.viewcode (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/viewcode.py
#   sphinx.ext.intersphinx (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.todo (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/todo.py
Traceback (most recent call last):
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/application.py", line 348, in build
    self.builder.build_update()
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 299, in build_update
    len(to_build))
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 416, in read
    self._read_parallel(docnames, nproc=self.app.parallel)
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 467, in _read_parallel
    tasks.join()
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/util/parallel.py", line 107, in join
    self._join_one()
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/util/parallel.py", line 112, in _join_one
    exc, logs, result = pipe.recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

This works fine when using Sphinx 3.0.4 or "-j 1" instead of "-j auto".

Environment info

OS: Linux
Python version: 3.6.7
Sphinx version: 3.1.0 (worked fine in 3.1.0)
Sphinx extensions:
sphinx-rtd-theme==0.4.3
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport==1.1.0
Extra tools: -

The text was updated successfully, but these errors were encountered:

tk0miya · 2020-06-08T16:30:33Z

Sorry for the inconvenience. Could you add a debug log to sphinx.util/parallel.py to investigate the error, please?

--- a/sphinx/util/parallel.py
+++ b/sphinx/util/parallel.py
@@ -88,6 +88,7 @@ class ParallelTasks:
             errmsg = traceback.format_exception_only(err.__class__, err)[0].strip()
             ret = (errmsg, traceback.format_exc())
         logging.convert_serializable(collector.logs)
+        print(failed, collector.logs, ret)
         pipe.send((failed, collector.logs, ret))

     def add_task(self, task_func: Callable, arg: Any = None, result_func: Callable = None) -> None:  # NOQA

Please open /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/util/parallel.py and enable the debug log.

robsan00 · 2020-06-08T17:39:22Z

This gives me a 60 MByte file which (1) I am not sure if you can do anything with and (2) basically contains our source code, which I don't really to give out completely. The last things printed out before the whole thing goes down is something like this:

return settings\n\x94}\x94(\x8c\t_NicField\x94j\x02\x01\x00\x00K\x1eK,\x87\x94\x8c\x15create_mach_info_file\x94j=3\x01\x00K/Kp\x87\x94\x8c\x10get_nic_settings\x94j=3\x01\x00KsK\xaa\x87\x94u}\x94(\x8c\t_NicField\x94j)\x03\x00\x00\x8c\x15create_mach_info_file\x94j)\x03\x00\x00\x8c\x10get_nic_settings\x94j)\x03\x00\x00u\x8c\x0fpyprep.machinfo\x94t\x94uub.'

The next line is the error from above.

I also tried to just print in case of an exception:

try:
    pipe.send((failed, collector.logs, ret))
except:
    print(failed, collector.logs, ret)
    raise

This did not look better, though.

webknjaz · 2020-06-11T10:39:35Z

FTR here's another repo where it's reproducible: https://github.com/cherrypy/cheroot/runs/761272594#step:9:203 (Python 3.7).

Since it's a public repo, it should be possible to see the issue locally by cloning the repo and running tox -e build-docs (pip install tox first, of course). I hope having a public source will help you see it first-hands.

P.S. I confirm that w/o parallelism it works: https://readthedocs.org/projects/cheroot/builds/11233786/.

tk0miya · 2020-06-11T13:58:38Z

@webknjaz Thank you for info. I reproduced the error on my local!

sphinx-build has crashed with EOFError when autodoc raises a warning having exc_info under parallel mode. In parallel mode, all messages are pickled to transfer logs to parent process. But the warning is not picklable because it contains a traceback object. This removes exc_info from warning messages to prevent crashes.

tk0miya · 2020-06-11T14:10:58Z

I posted #7802 to fix this.

Note: I used this Dockerfile to reproduce the error.

FROM python:3.7-slim

RUN apt update; apt install -y git make build-essential vim
RUN git clone https://github.com/cherrypy/cheroot
WORKDIR /cheroot
RUN pip3 install -e .
RUN pip3 install -U Sphinx
RUN pip3 install tox
RUN apt install -y enchant
RUN tox -e build-docs --notest

webknjaz · 2020-06-11T16:05:45Z

s/#7802/#7818

Fix #7802: autodoc: EOFError is raised on parallel build

robsan00 · 2020-06-15T07:45:02Z

@webknjaz Thanks for reproducing this!

@tk0miya The fix works for us, thanks!

webknjaz · 2020-06-15T10:43:56Z

I also confirm that this is fixed for me. Great job, @tk0miya!

robsan00 added the type:bug label Jun 8, 2020

tk0miya added this to the 3.1.1 milestone Jun 8, 2020

tk0miya added the prio:high label Jun 8, 2020

vytas7 mentioned this issue Jun 10, 2020

Work around sphinx 3.1.0 bug causing parallel build to fail on Linux falconry/falcon#1729

Merged

This was referenced Jun 11, 2020

sphinxcontrib-spelling==5.1.0 + Sphinx==3.1.0 explode under Python 3.7 sphinx-contrib/spelling#49

Closed

Properly close SSL socket in _loopback_for_cert cherrypy/cheroot#291

Merged

tk0miya mentioned this issue Jun 11, 2020

Fix #7802: autodoc: EOFError is raised on parallel build #7818

Merged

tk0miya added a commit that referenced this issue Jun 13, 2020

Merge pull request #7818 from tk0miya/7802_EOFError_on_parallel_build

46f79c5

Fix #7802: autodoc: EOFError is raised on parallel build

tk0miya closed this as completed Jun 13, 2020

sciunto mentioned this issue Aug 12, 2020

Block sphinx versions for which links from search engine are broken scikit-image/scikit-image#4910

Draft

SeanDS mentioned this issue Mar 8, 2021

EOFError when using multiprocessing in 3.5.1 #8973

Closed

github-actions bot locked as resolved and limited conversation to collaborators Jul 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EOFError when using multiprocessing in 3.1.0 #7802

EOFError when using multiprocessing in 3.1.0 #7802

robsan00 commented Jun 8, 2020

tk0miya commented Jun 8, 2020

robsan00 commented Jun 8, 2020

webknjaz commented Jun 11, 2020 •

edited

tk0miya commented Jun 11, 2020

tk0miya commented Jun 11, 2020

webknjaz commented Jun 11, 2020

robsan00 commented Jun 15, 2020

webknjaz commented Jun 15, 2020

EOFError when using multiprocessing in 3.1.0 #7802

EOFError when using multiprocessing in 3.1.0 #7802

Comments

robsan00 commented Jun 8, 2020

tk0miya commented Jun 8, 2020

robsan00 commented Jun 8, 2020

webknjaz commented Jun 11, 2020 • edited

tk0miya commented Jun 11, 2020

tk0miya commented Jun 11, 2020

webknjaz commented Jun 11, 2020

robsan00 commented Jun 15, 2020

webknjaz commented Jun 15, 2020

webknjaz commented Jun 11, 2020 •

edited