KeyError in parallel build with Sphinx 4.0.1

[pllim]

Describe the bug

Sphinx build fails for astropy when we use Sphinx 4.0.1.

To Reproduce

See log in https://readthedocs.org/projects/astropy/builds/ , for example: https://readthedocs.org/projects/astropy/builds/13727660/

Expected behavior

Build is successful.

Your project

https://github.com/astropy/astropy/

Screenshots

N/A

Environment info

• OS: Linux
• Python version: 3.8
• Sphinx version: 4.0.1
• Sphinx extensions: astropy-sphinx-theme-1.1 numpydoc-1.1.0 sphinx-astropy-1.3 sphinx-automodapi-0.13 sphinx-changelog-1.0.0 sphinx-gallery-0.9.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 (full list in the RTD log above)
• Extra tools: We have parallel build enabled in RTD

Additional context

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/events.py", line 101, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/ext/viewcode.py", line 153, in env_merge_info
    used = env._viewcode_modules[modname][2]  # type: ignore
TypeError: 'bool' object is not subscriptable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/util/parallel.py", line 108, in join
    self._join_one()
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/util/parallel.py", line 130, in _join_one
    self._result_funcs.pop(tid)(self._args.pop(tid), result)
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 451, in merge
    self.env.merge_info_from(docs, env, self.app)
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 310, in merge_info_from
    self.events.emit('env-merge-info', self, docnames, other)
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/events.py", line 109, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function env_merge_info at 0x7f5f85293160> for event 'env-merge-info' threw an exception (exception: 'bool' object is not subscriptable)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/application.py", line 350, in build
    self.builder.build_update()
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 292, in build_update
    self.build(to_build,
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 306, in build
    updated_docnames = set(self.read())
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 411, in read
    self._read_parallel(docnames, nproc=self.app.parallel)
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 462, in _read_parallel
    tasks.join()
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/util/parallel.py", line 111, in join
    self.terminate()
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/util/parallel.py", line 117, in terminate
    self._result_funcs.pop(tid)
KeyError: 12

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/astropy/envs/latest/lib/python3.8/site-packages/sphinx/util/parallel.py", line 117, in terminate
    self._result_funcs.pop(tid)
KeyError: 12
The full traceback has been saved in /tmp/sphinx-err-sfhj_9yk.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

[saimn]

Seems related to #8935.

[astrojuanlu]

In progress: it looks like on the line

    used = env._viewcode_modules[modname][2]  # type: ignore

modname is always astropy.wcs._wcs and env._viewcode_modules[modname] is always False.

[astrojuanlu]

In all the rest of the cases, the entries of env._viewcode_modules are 4-tuples. Looking here:

sphinx/sphinx/ext/viewcode.py

Lines 83 to 103 in 778faff

	entry = env._viewcode_modules.get(modname, None) # type: ignore
	if entry is False:
	return False
	code_tags = app.emit_firstresult('viewcode-find-source', modname)
	if code_tags is None:
	try:
	analyzer = ModuleAnalyzer.for_module(modname)
	analyzer.find_tags()
	except Exception:
	env._viewcode_modules[modname] = False # type: ignore
	return False
	code = analyzer.code
	tags = analyzer.tags
	else:
	code, tags = code_tags
	if entry is None or entry[0] != code:
	entry = code, tags, {}, refname
	env._viewcode_modules[modname] = entry # type: ignore

I'd say that the env_merge_info code should guard for env._viewcode_modules[modname] possibly being False, but I don't understand what it should do instead. In any case, looks like a pretty localized bug.

[astrojuanlu]

This naïve patch fixed the issue:

diff --git a/sphinx/ext/viewcode.py b/sphinx/ext/viewcode.py
index f2a42f33f..fd6b307bc 100644
--- a/sphinx/ext/viewcode.py
+++ b/sphinx/ext/viewcode.py
@@ -150,7 +150,12 @@ def env_merge_info(app: Sphinx, env: BuildEnvironment, docnames: Iterable[str],
         if modname not in env._viewcode_modules:  # type: ignore
             env._viewcode_modules[modname] = entry  # type: ignore
         else:
-            used = env._viewcode_modules[modname][2]  # type: ignore
+            try:
+                used = env._viewcode_modules[modname][2]  # type: ignore
+            except TypeError:
+                # Entry is False, avoid `'bool' object is not subscriptable`
+                # see https://github.com/sphinx-doc/sphinx/issues/9210
+                continue
             for fullname, docname in entry[2].items():
                 if fullname not in used:
                     used[fullname] = docname

[tk0miya]

I posted #9212 to fix this. Can somebody try this?

[astrojuanlu]

On it!

[astrojuanlu]

#9212 fixes this issue, thanks a lot @tk0miya 👍🏽

[tk0miya]

@astrojuanlu Thank you for confirmation. Merging it.

[tk0miya]

I'll ship a bugfix package including #9212 after a while (maybe this weekend). Please wait for a while.
refs: #9214

[pllim]

Thank you!