sphinx assumes python-language source for modules

[AstraLuma]

Describe the bug
There exists projects that allow importing non-Python code as a python module (such as xonsh, gqlmod).

sphinx currently chokes on these modules instead of failing gracefully.

To Reproduce
Sorry that I don't have a reduced minimal test case.

1. Use one of these importers
2. Attempt to autodoc against it

From the travis job

  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/sphinx/pycode/__init__.py", line 167, in parse
    raise PycodeError('parsing %r failed: %r' % (self.srcname, exc))
sphinx.errors.PycodeError: parsing '/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/xonsh/lib/os.xsh' failed: SyntaxError('invalid syntax', ('<unknown>', 10, 5, '    ![pushd @(d)]\n'))

Expected behavior
Degrading gracefully, disabling the features that use this or falling back to alternate methods. (I think this is comment-based type annotations and attribute docs?)

Your project
https://github.com/xonsh/xonsh

Environment info

• OS: Linux
• Python version: 3.6, 3.7
• Sphinx version: 2.4.1
• Sphinx extensions: sphinx.ext.autodoc, sphinx.ext.doctest, sphinx.ext.imgmath, sphinx.ext.inheritance_diagram, sphinx.ext.viewcode, numpydoc, cmdhelp",

[tk0miya]

Could you give us a full stacktrace here? I'd like to know where the error happened. BTW, the main target of autodoc is python module. So we can't promise you to work it for non python codes. Anyway I'll try to investigate we can fix it easily.

[AstraLuma]

sphinx-err-i34if66s.log

[AstraLuma]

And at least in the case of xsh, analysis of the runtime objects (inspect and friends) would still be meaningful and useful.

[tk0miya]

Thank you for the log.

I tried to reproduce the error on my local. But I still can't. Instead of the error, I got following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app.build(args.force_all, filenames)
  File "/usr/local/lib/python3.7/site-packages/sphinx/application.py", line 349, in build
    self.builder.build_update()
  File "/usr/local/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 299, in build_update
    len(to_build))
  File "/usr/local/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "/usr/local/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 418, in read
    self._read_serial(docnames)
  File "/usr/local/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 439, in _read_serial
    self.read_doc(docname)
  File "/usr/local/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 479, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/usr/local/lib/python3.7/site-packages/sphinx/io.py", line 316, in read_doc
    pub.publish()
  File "/usr/local/lib/python3.7/site-packages/docutils/core.py", line 218, in publish
    self.settings)
  File "/usr/local/lib/python3.7/site-packages/sphinx/io.py", line 130, in read
    self.parse()
  File "/usr/local/lib/python3.7/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/usr/local/lib/python3.7/site-packages/sphinx/parsers.py", line 93, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 171, in run
    input_source=document['source'])
  File "/usr/local/lib/python3.7/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/usr/local/lib/python3.7/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 3005, in text
    self.section(title.lstrip(), source, style, lineno + 1, messages)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 395, in new_subsection
    node=section_node, match_titles=True)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/local/lib/python3.7/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/usr/local/lib/python3.7/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2097, in directive
    directive_class, match, type_name, option_presets)
  File "/usr/local/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/usr/local/lib/python3.7/site-packages/sphinx/ext/autodoc/directive.py", line 146, in run
    documenter.generate(more_content=self.content)
  File "/usr/local/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py", line 694, in generate
    self.analyzer = ModuleAnalyzer.for_module(self.real_modname)
  File "/usr/local/lib/python3.7/site-packages/sphinx/pycode/__init__.py", line 117, in for_module
    filename, source = cls.get_module_source(modname)
  File "/usr/local/lib/python3.7/site-packages/sphinx/pycode/__init__.py", line 45, in get_module_source
    source = loader.get_source(modname)
  File "/xonsh/xonsh/imphooks.py", line 129, in get_source
    with open(fullpath, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'xonsh.lib.os'

Exception occurred:
  File "/xonsh/xonsh/imphooks.py", line 129, in get_source
    with open(fullpath, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'xonsh.lib.os'
The full traceback has been saved in /tmp/sphinx-err-r5wgfybv.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [Makefile:39: html] Error 2

My Dockerfile is here:

FROM python:3.7-slim

RUN apt update; apt install -y git make build-essential
RUN git clone https://github.com/xonsh/xonsh
WORKDIR /xonsh
RUN pip3 install -e .
RUN pip3 install -r requirements/docs.txt
WORKDIR /xonsh/docs
RUN make html SPHINXOPTS=-T

I guess your error would be fixed by this patch. But I need to reproduce the error to confirm it working fine.

diff --git a/sphinx/ext/viewcode.py b/sphinx/ext/viewcode.py
index 09fcd695f..70fcca90b 100644
--- a/sphinx/ext/viewcode.py
+++ b/sphinx/ext/viewcode.py
@@ -63,11 +63,11 @@ def doctree_read(app: Sphinx, doctree: Node) -> None:
         if code_tags is None:
             try:
                 analyzer = ModuleAnalyzer.for_module(modname)
+                analyzer.find_tags()
             except Exception:
                 env._viewcode_modules[modname] = False  # type: ignore
                 return

-            analyzer.find_tags()
             code = analyzer.code
             tags = analyzer.tags
         else:

[AstraLuma]

Yeah, there's a bug fix for that one open. The above Travis job is from that branch.

[tk0miya]

Succeeded to reproduced now.

FROM python:3.7-slim

RUN apt update; apt install -y git make build-essential
RUN git clone https://github.com/astronouth7303/xonsh
WORKDIR /xonsh
RUN git checkout fix-imphooks-get_source
RUN pip3 install -e .
RUN pip3 install -r requirements/docs.txt
WORKDIR /xonsh/docs
RUN make html SPHINXOPTS=-T

[tk0miya]

Fixed.