Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EOFError when using multiprocessing in 3.5.1 #8973

Closed
SeanDS opened this issue Mar 8, 2021 · 6 comments
Closed

EOFError when using multiprocessing in 3.5.1 #8973

SeanDS opened this issue Mar 8, 2021 · 6 comments
Labels
type:bug

Comments

@SeanDS
Copy link

SeanDS commented Mar 8, 2021

Describe the bug
Similar to #7802, I am seeing an EOFError in the Sphinx HTML build when enabling the -j auto flag, but this is with the newest version of Sphinx and Python 3.9, so it appears the fix for that issue is not relevant or not working now.

The error occurs intermittently - it sometimes works, and sometimes fails with the error below:

waiting for workers...
Exception occurred:
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
    raise EOFError
EOFError
The full traceback has been saved in /tmp/sphinx-err-giu99hwq.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [Makefile:37: html_build] Error 2

The logfile:

# Sphinx version: 3.5.1
# Python version: 3.9.2 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.3
# Last messages:
#   writing output... [ 92%] api/utilities/generated/finesse.utilities.blockdiag.display_loops_blockdiag .. api/utilities/generated/finesse.utilities.misc.logs
#   writing output... [ 93%] api/utilities/generated/finesse.utilities.misc.opened_file .. api/utilities/optimise/core/finesse.utilities.optimise.core.BaseOptimiser
#   writing output... [ 94%] api/utilities/optimise/core/finesse.utilities.optimise.core.BaseQuantity .. api/utilities/optimise/core/generated/finesse.utilities.optimise.core.BaseOptimiser.x
#   writing output... [ 95%] api/utilities/optimise/core/generated/finesse.utilities.optimise.core.BaseQuantity.__init__ .. developer/codeguide/cython/annotations
#   writing output... [ 96%] developer/codeguide/cython/cmath_extension .. developer/testing/index
#   writing output... [ 97%] getting_started/example2 .. manual/mathematical_description/plane-waves/modulation_light_fields
#   writing output... [ 98%] manual/radiation_pressure/index .. usage/internal_differences
#   writing output... [100%] usage/legacy_mode .. zglossary/index
#   
#   waiting for workers...
# Loaded extensions:
#   sphinx.ext.mathjax (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /usr/local/lib/python3.9/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.type_comment (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.autosummary (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/autosummary/__init__.py
#   sphinx.ext.coverage (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/coverage.py
#   sphinx.ext.doctest (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/doctest.py
#   sphinx.ext.intersphinx (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.todo (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/todo.py
#   sphinx.ext.ifconfig (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/ifconfig.py
#   sphinx.ext.viewcode (3.5.1) from /usr/local/lib/python3.9/site-packages/sphinx/ext/viewcode.py
#   sphinxcontrib.bibtex (unknown version) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/bibtex/__init__.py
#   sphinxcontrib.katex (0.6.1) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/katex.py
#   sphinx_click (unknown version) from /usr/local/lib/python3.9/site-packages/sphinx_click/__init__.py
#   finesse_sphinx (0.5.0) from /usr/local/lib/python3.9/site-packages/finesse_sphinx/__init__.py
#   numpydoc (1.1.0) from /usr/local/lib/python3.9/site-packages/numpydoc/__init__.py
#   jupyter_sphinx (0.3.2) from /usr/local/lib/python3.9/site-packages/jupyter_sphinx/__init__.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/usr/local/lib/python3.9/site-packages/sphinx/application.py", line 352, in build
    self.builder.build_update()
  File "/usr/local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 296, in build_update
    self.build(to_build,
  File "/usr/local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 360, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 531, in write
    self._write_parallel(sorted(docnames),
  File "/usr/local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 575, in _write_parallel
    tasks.join()
  File "/usr/local/lib/python3.9/site-packages/sphinx/util/parallel.py", line 107, in join
    self._join_one()
  File "/usr/local/lib/python3.9/site-packages/sphinx/util/parallel.py", line 112, in _join_one
    exc, logs, result = pipe.recv()
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 255, in recv
    buf = self._recv_bytes()
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
    raise EOFError
EOFError

Your project
Can provide if required, but can't upload publicly.

Environment info

  • OS: Linux host, running the python:3.9 Docker image.
  • Python version: 3.9
  • Sphinx version: 3.5.1
  • Sphinx extensions: see the log above
@SeanDS SeanDS added the type:bug label Mar 8, 2021
@tk0miya
Copy link
Member

tk0miya commented Mar 8, 2021

The EOFError says the child process was died by unexpected reason. It's not the root reason for the crash. Is there any output in your console? What happened if you give -v to the sphinx-build? In addition to this, it would be better to check if you downgrade Sphinx to 3.4.3.

@SeanDS
Copy link
Author

SeanDS commented Mar 8, 2021

I downgraded to 3.4.3, added the -v flag and ran the build a few more times until it failed again with the same error. The new log is:

# Sphinx version: 3.4.3
# Python version: 3.9.2 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.3
# Last messages:
#   writing output... [ 90%] api/tracing/generated/finesse.tracing.ctracer.get_last_left_branch .. api/tree/generated/finesse.tree.TreeNode.__init__
#   writing output... [ 91%] api/tree/generated/finesse.tree.TreeNode.add .. api/utilities/generated/finesse.utilities.blockdiag.display_blockdiag_output
#   writing output... [ 92%] api/utilities/generated/finesse.utilities.blockdiag.display_loops_blockdiag .. api/utilities/generated/finesse.utilities.misc.logs
#   writing output... [ 93%] api/utilities/generated/finesse.utilities.misc.opened_file .. api/utilities/optimise/core/finesse.utilities.optimise.core.BaseOptimiser
#   writing output... [ 94%] api/utilities/optimise/core/finesse.utilities.optimise.core.BaseQuantity .. api/utilities/optimise/core/generated/finesse.utilities.optimise.core.BaseOptimiser.x
#   writing output... [ 95%] api/utilities/optimise/core/generated/finesse.utilities.optimise.core.BaseQuantity.__init__ .. developer/codeguide/cython/annotations
#   writing output... [ 96%] developer/codeguide/cython/cmath_extension .. developer/testing/index
#   writing output... [ 97%] getting_started/example2 .. manual/mathematical_description/plane-waves/modulation_light_fields
#   writing output... [ 98%] manual/radiation_pressure/index .. usage/internal_differences
#   writing output... [100%] usage/legacy_mode .. zglossary/index
# Loaded extensions:
#   sphinx.ext.mathjax (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /usr/local/lib/python3.9/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.type_comment (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.autosummary (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/autosummary/__init__.py
#   sphinx.ext.coverage (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/coverage.py
#   sphinx.ext.doctest (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/doctest.py
#   sphinx.ext.intersphinx (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.todo (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/todo.py
#   sphinx.ext.ifconfig (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/ifconfig.py
#   sphinx.ext.viewcode (3.4.3) from /usr/local/lib/python3.9/site-packages/sphinx/ext/viewcode.py
#   sphinxcontrib.bibtex (unknown version) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/bibtex/__init__.py
#   sphinxcontrib.katex (0.6.1) from /usr/local/lib/python3.9/site-packages/sphinxcontrib/katex.py
#   sphinx_click (unknown version) from /usr/local/lib/python3.9/site-packages/sphinx_click/__init__.py
#   finesse_sphinx (0.5.0) from /usr/local/lib/python3.9/site-packages/finesse_sphinx/__init__.py
#   numpydoc (1.1.0) from /usr/local/lib/python3.9/site-packages/numpydoc/__init__.py
#   jupyter_sphinx (0.3.2) from /usr/local/lib/python3.9/site-packages/jupyter_sphinx/__init__.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/usr/local/lib/python3.9/site-packages/sphinx/application.py", line 352, in build
    self.builder.build_update()
  File "/usr/local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 296, in build_update
    self.build(to_build,
  File "/usr/local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 360, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 531, in write
    self._write_parallel(sorted(docnames),
  File "/usr/local/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 571, in _write_parallel
    tasks.add_task(write_process, arg)
  File "/usr/local/lib/python3.9/site-packages/sphinx/util/parallel.py", line 103, in add_task
    self._join_one()
  File "/usr/local/lib/python3.9/site-packages/sphinx/util/parallel.py", line 112, in _join_one
    exc, logs, result = pipe.recv()
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 255, in recv
    buf = self._recv_bytes()
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 388, in _recv
    raise EOFError
EOFError

The full output in the console is here. I don't see any error messages.

I should point out that we execute a few long running (30-60 second) scripts using jupyter-sphinx. Could these be reaching some execution timeout perhaps?

@tk0miya
Copy link
Member

tk0miya commented Mar 9, 2021

Thank you for the info. But I don't still find the reason of the crash... I need to reproduce the error on my local. Could you share minimal example of this case? Full document is not needed.

I should point out that we execute a few long running (30-60 second) scripts using jupyter-sphinx. Could these be reaching some execution timeout perhaps?

No. Parallel build feature does not have any timeouts. And the error says the message pipe between sub-process was closed unexpectedly. It means the sub-process is died.

@SeanDS
Copy link
Author

SeanDS commented Mar 10, 2021

It will be difficult to provide you with a minimum example since we use jupyter-execute which requires the our Python code to be on the Python path. The main project uses C extensions so I'd need to also build a bdist wheel for your OS (if I am able to, for your architecture). The docs + wheel are probably around 60 MB. Are you willing to continue debugging this, given the effort it will require to install our environment? Or is there anything I can still try first, like enabling further debug/logging inside Sphinx?

I also just removed all uses of jupyter-execute in the docs (extension is still installed) and I've so far not seen this error again.

@JordanYates
Copy link

I am seeing this when building docs under WSL2 for the Zephyr open source project.
My logs are no more informative than those posted by Sean.

Python 3.8.10
Sphinx v4.3.2

Source can be found here: https://github.com/zephyrproject-rtos/zephyr
Build info here: https://docs.zephyrproject.org/latest/guides/docs/index.html

@AA-Turner
Copy link
Member

Closing as there's no reproducer. If the issue persists in 4.5 or 5.0 please open a new issue.

A

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tk0miya @SeanDS @JordanYates @AA-Turner