Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Regression] KeyError on pybind11-generated wrappers with Sphinx 3.2 #8084

Closed
inducer opened this issue Aug 9, 2020 · 5 comments
Closed

[Regression] KeyError on pybind11-generated wrappers with Sphinx 3.2 #8084

inducer opened this issue Aug 9, 2020 · 5 comments
Labels
extensions:autodoc type:bug
Milestone
3.2.1

Comments

@inducer
Copy link

inducer commented Aug 9, 2020

Describe the bug
Described by this error log:

# Sphinx version: 3.2.0
# Python version: 3.8.5 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.2
# Last messages:
#   [new config]
#   8 added, 0 changed, 0 removed
#   reading sources... [ 12%] index
#   reading sources... [ 25%] misc
#   reading sources... [ 37%] ref_ast
#   reading sources... [ 50%] ref_containers
#   reading sources... [ 62%] ref_flow
#   reading sources... [ 75%] ref_fundamental
#   reading sources... [ 87%] ref_schedule
#   reading sources... [100%] reference
# Loaded extensions:
#   sphinx.ext.mathjax (3.2.0) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.type_comment (3.2.0) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (3.2.0) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.intersphinx (3.2.0) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.imgmath (3.2.0) from /var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/imgmath.py
Traceback (most recent call last):
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/application.py", line 348, in build
    self.builder.build_update()
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 297, in build_update
    self.build(to_build,
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 418, in read
    self._read_serial(docnames)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 439, in _read_serial
    self.read_doc(docname)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 479, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/io.py", line 223, in read_doc
    pub.publish()
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/io.py", line 128, in read
    self.parse()
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/parsers.py", line 102, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 170, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
    newabsoffset = self.nested_parse(
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
    newabsoffset = self.nested_parse(
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
    return self.run_directive(
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/directive.py", line 146, in run
    documenter.generate(more_content=self.content)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 1568, in generate
    return super().generate(more_content=more_content,
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 894, in generate
    self.document_members(all_members)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 1559, in document_members
    super().document_members(all_members)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 775, in document_members
    documenter.generate(
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 884, in generate
    self.add_directive_header(sig)
  File "/var/lib/gitlab-runner/builds/0d8732fb/0/inducer/islpy/.env/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 1980, in add_directive_header
    annotations = get_type_hints(self.parent)
  File "/usr/lib/python3.8/typing.py", line 1223, in get_type_hints
    base_globals = sys.modules[base.__module__].__dict__
KeyError: 'pybind11_builtins'

My understanding is that pybind11_builtins is a module referenced by type hints from wrappers based on pybind11 that doesn't actually exist.

See also: https://gitlab.tiker.net/inducer/islpy/-/jobs/148860

To Reproduce
Steps to reproduce the behavior:

git clone https://github.com/inducer/islpy.git
curl -L -O -k https://gitlab.tiker.net/inducer/ci-support/raw/master/build-docs.sh
. ./build-docs.sh

Environment info

  • OS: Debian Linux
  • Python version: (see log)
  • Sphinx version: (see log)
  • Sphinx extensions: see config
@inducer inducer changed the title <what happen when you do on which document project> [Regression] KeyError on pybind11-generated wrappers with Sphinx 3.2 Aug 9, 2020
@inducer
Copy link
Author

inducer commented Aug 9, 2020

Context: Seems to have worked fine on otherwise identical code with Sphinx 3.1.2: https://gitlab.tiker.net/inducer/islpy/-/jobs/141927

@tk0miya
Copy link
Member

tk0miya commented Aug 9, 2020

My understanding is that pybind11_builtins is a module referenced by type hints from wrappers based on pybind11 that doesn't actually exist.

I feel such a module is broken...

@tk0miya tk0miya added this to the 3.2.1 milestone Aug 9, 2020
@tk0miya tk0miya mentioned this issue Aug 9, 2020
Merged
@inducer
Copy link
Author

inducer commented Aug 9, 2020

I feel such a module is broken...

I don't disagree with you, however:

  • pybind11 is a pretty commonly-used tool for wrapping C++ into Python
  • This is a regression for Sphinx 3.1.2.

Thanks for working on a PR to address this!

@tk0miya
Copy link
Member

tk0miya commented Aug 9, 2020

The error was raised inside typing module that is a standard library of python. It says the __module__ member of the some class (maybe a class of pybind11) points invalid and non-existence module. AFAIK, code inspectors like typing module (including autodoc) expect the target object is valid.

@tk0miya
Copy link
Member

tk0miya commented Aug 10, 2020

Note: I reproduced the KeyError without Sphinx. I guess this would be an unintended behavior of typing library.

$ cat test.py
import typing


class Foo:
    __module__ = 'unknown'


typing.get_type_hints(Foo)
$ python -V
Python 3.8.3
$ python test.py
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    typing.get_type_hints(Foo)
  File "/Users/tkomiya/.pyenv/versions/3.8.3/lib/python3.8/typing.py", line 1223, in get_type_hints
    base_globals = sys.modules[base.__module__].__dict__
KeyError: 'unknown'

tk0miya added a commit that referenced this issue Aug 10, 2020
@tk0miya
Merge pull request #8085 from tk0miya/8084_KeyError_for_broken_class
38b868c 
Fix #8084: autodoc: KeyError is raised on documenting a broken attribute
greglandrum added a commit to greglandrum/rdkit that referenced this issue Aug 12, 2020
@greglandrum
lock sphinx version due to problem with 3.2.0
74278fb 
the problem was fixed here: sphinx-doc/sphinx#8084
but that is not yet released
@greglandrum greglandrum mentioned this issue Aug 12, 2020
Merged
greglandrum added a commit to rdkit/rdkit that referenced this issue Aug 12, 2020
@greglandrum
lock sphinx version in CI due to problem with v3.2.0 (#3332)
e95b2b2 
* lock sphinx version due to problem with 3.2.0

the problem was fixed here: sphinx-doc/sphinx#8084
but that is not yet released

* forgot one

* fix bad doc formatting
greglandrum added a commit to rdkit/rdkit that referenced this issue Aug 12, 2020
@greglandrum
lock sphinx version in CI due to problem with v3.2.0 (#3332)
6cd062a 
* lock sphinx version due to problem with 3.2.0

the problem was fixed here: sphinx-doc/sphinx#8084
but that is not yet released

* forgot one

* fix bad doc formatting
@molpopgen molpopgen mentioned this issue Aug 13, 2020
Closed
@tk0miya tk0miya mentioned this issue Aug 14, 2020
Closed
@tirkarthi tirkarthi mentioned this issue Apr 8, 2021
Closed
5 tasks
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
extensions:autodoc type:bug
Projects
None yet
Milestone
3.2.1
Development

No branches or pull requests
2 participants
@inducer @tk0miya