Circular import with sphinxcontrib.serializinghtml

[progval]

Describe the bug
I can't run Sphinx because a circular import:

• sphinx.builders.html import sphinxcontrib.serializinghtml unconditionally: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/builders/html.py#L1142
• sphinxcontrib.serializinghtml import sphinx.builders.html unconditionally: https://github.com/sphinx-doc/sphinxcontrib-serializinghtml/blob/master/sphinxcontrib/serializinghtml/__init__.py#L15

To Reproduce
Steps to reproduce the behavior:

$ git clone https://github.com/t-b/sphinxcontrib-images.git
$ cd sphinxcontrib-images
$ git checkout port-to-sphinx-2.x
$ git submodule update --init
$ pip3 install --user sphinx==2.1 sphinx-rtd-theme==0.1.6
$ cd docs
$ make html

-> ImportError on some computers

Expected behavior
no ImportError

Your project
https://github.com/t-b/sphinxcontrib-images/ branch port-to-sphinx-2.x
(that's not my projet, I'm just trying to make it work)

Environment info

• OS: Linux
• Python version: 3.5
• Sphinx version: 2.1
• Sphinx extensions: sphinx.ext.autodoc, sphinx.ext.todo, sphinx.ext.viewcode

Additional context

For some reason, the build does work on some computers. But when reading the source code, I don't understand why it does work on these.

(issue copied from #6245 (comment) )

[tk0miya]

I tried to reproduce the error with following Dockefile. But it does not raise ImportError...

FROM python:3.5

RUN git clone https://github.com/t-b/sphinxcontrib-images.git
WORKDIR /sphinxcontrib-images
RUN git checkout port-to-sphinx-2.x
RUN git submodule update --init
RUN pip3 install sphinx==2.1 sphinx-rtd-theme==0.1.6
RUN pip3 install .
WORKDIR /sphinxcontrib-images/docs
RUN make html

note: I need to install sphinxcontrib-images itself before building doc.

[tk0miya]

sphinx.builders.html import sphinxcontrib.serializinghtml unconditionally: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/builders/html.py#L1142
sphinxcontrib.serializinghtml import sphinx.builders.html unconditionally: https://github.com/sphinx-doc/sphinxcontrib-serializinghtml/blob/master/sphinxcontrib/serializinghtml/__init__.py#L15

If my understanding is correct, this will not cause circular imports. sphinx.builders.html goes to import serializinghtml module at almost tail of the module. So it does not prevent to import sphinx.builders.html module from serializinghtml module because it is almost loaded. But some interpreters or environments (settings?) might consider this technique to circular imports...

[progval]

Indeed. This issue seems to be very sensitive to the environment, and many small variations make it work.

I managed to write a dockerfile that reproduces it (with a lot of help from @douardda) , but only within a venv created by Tox:

FROM debian:stretch
 
RUN apt-get update
RUN apt-get install -y python3-pip tox git make
RUN git clone https://github.com/t-b/sphinxcontrib-images.git
WORKDIR /sphinxcontrib-images
RUN git checkout port-to-sphinx-2.x
RUN git submodule update --init
RUN tox -e py35-sphinx21

One of the variations that make it work is to downgrade setuptools, which you can do by adding this before the last line:

RUN tox -e py35-sphinx21 || true
RUN /sphinxcontrib-images/.tox/py35-sphinx21/bin/pip3 install "setuptools<41"

There is also some unexplained way for tox to work outside docker without this manual downgrade, but I'm no longer able to reproduce it.

[tk0miya]

Thank you for information. I also reproduced.

It seems your build process installs sphinxcontrib-images package twice (via tox and via setup.py). And it causes this problem. With removal installation process from tox.ini, it goes succeeded.

commands =
           make -C docs clean
           #./setup.py bdist_wheel
           #./setup.py install
           make -C docs html epub latex

In this case, python interpreter considers sphinxcontrib is an actual package on importing sphinxcontrib.images, not a namespace package.

Note: I also found another condition to reproduce this problem:

FROM python:3.5

RUN git clone https://github.com/t-b/sphinxcontrib-images.git
WORKDIR /sphinxcontrib-images
RUN git checkout port-to-sphinx-2.x
RUN git submodule update --init
RUN pip3 install Sphinx
RUN pip3 install -U 'setuptools<40.0'
RUN pip3 install .
RUN python3 -c 'import sphinxcontrib.images; import sphinx.builders.html'

This will be resolved with upgrading setuptools to 41.0 or later. As an another way, moving to other directories also resolve the error:

RUN cd / && python3 -c 'import sphinxcontrib.images; import sphinx.builders.html'

I don't know what happened in setuptools, but this might be shown for developers.

[costrouc]

This issue is present in nixpkgs. Here is the build log https://hydra.nixos.org/build/109159566/nixlog/1.

Relevant part

ERROR: tests.test_basic.test_build_singlehtml
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nix/store/rajdjnhlp7rkk5ix634qdzfpjbdhffsl-python3.7-sphinx-2.2.2/lib/python3.7/site-packages/sphinx/registry.py", line 475, in load_extension
    mod = import_module(extname)
  File "/nix/store/ga9kam5sgryz01frprfiy0b0kblmy1wi-python3-3.7.6/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/nix/store/rajdjnhlp7rkk5ix634qdzfpjbdhffsl-python3.7-sphinx-2.2.2/lib/python3.7/site-packages/sphinx/builders/epub3.py", line 20, in <module>
    from sphinx.builders import _epub_base
  File "/nix/store/rajdjnhlp7rkk5ix634qdzfpjbdhffsl-python3.7-sphinx-2.2.2/lib/python3.7/site-packages/sphinx/builders/_epub_base.py", line 25, in <module>
    from sphinx.builders.html import BuildInfo, StandaloneHTMLBuilder
  File "/nix/store/rajdjnhlp7rkk5ix634qdzfpjbdhffsl-python3.7-sphinx-2.2.2/lib/python3.7/site-packages/sphinx/builders/html.py", line 1201, in <module>
    import sphinxcontrib.serializinghtml  # NOQA
ModuleNotFoundError: No module named 'sphinxcontrib.serializinghtml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/kwlwn1gfk76x4f57cl7pig7nsg3pgly5-python3.7-nose-1.3.7/lib/python3.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/nix/store/f9040xca26pkvqadjwip3f4pi71264sr-python3.7-sphinx-testing-1.0.1/lib/python3.7/site-packages/sphinx_testing/util.py", line 168, in decorator
    app = TestApp(*self.sphinxargs, **sphinxkwargs)
  File "/nix/store/f9040xca26pkvqadjwip3f4pi71264sr-python3.7-sphinx-testing-1.0.1/lib/python3.7/site-packages/sphinx_testing/util.py", line 102, in __init__
    verbosity, parallel)
  File "/nix/store/rajdjnhlp7rkk5ix634qdzfpjbdhffsl-python3.7-sphinx-2.2.2/lib/python3.7/site-packages/sphinx/application.py", line 239, in __init__
    self.setup_extension(extension)
  File "/nix/store/rajdjnhlp7rkk5ix634qdzfpjbdhffsl-python3.7-sphinx-2.2.2/lib/python3.7/site-packages/sphinx/application.py", line 402, in setup_extension
    self.registry.load_extension(self, extname)
  File "/nix/store/rajdjnhlp7rkk5ix634qdzfpjbdhffsl-python3.7-sphinx-2.2.2/lib/python3.7/site-packages/sphinx/registry.py", line 478, in load_extension
    raise ExtensionError(__('Could not import extension %s') % extname, err)
sphinx.errors.ExtensionError: Could not import extension sphinx.builders.epub3 (exception: No module named 'sphinxcontrib.serializinghtml')