Allow unqualified type hint names in generated docs

[alkasm]

It would be great if the current type hinting work would support removing qualification from type names. For some modules, the types may be deeply nested which will more easily break the page layout.

Definitions (by example)

• Qualified name: pkg.module.ClassName
• Unqualified name: ClassName

Example

from typing import Tuple
from numpy.polynomial import polynomial

def multiply_polynomial(
    p: polynomial.Polynomial, value: int
) -> Tuple[polynomial.Polynomial, polynomial.Polynomial]:
    return p, polynomial.Polynomial(*(value * p.coef))

In this code, the type is polynomial.Polynomial but the fully qualified name is numpy.polynomial.polynomial.Polynomial which is what will show up in the docs.

With the following in conf.py

extensions = [
    "sphinx.ext.autodoc",
    "sphinx.ext.autodoc.typehints",
]
autodoc_typehints = "description"

We'd see some generated docs similar to

mypkg.mymodule.multiply_polynomial(p, value)
    Parameters:     * p (numpy.polynomial.polynomial.Polynomial) –
                    * value (int) –
    Return type:    Tuple[numpy.polynomial.polynomial.Polynomial, numpy.polynomial.polynomial.Polynomial]

Instead, with unqualified names, we'd have a much simpler

mypkg.mymodule.multiply_polynomial(p, value)
    Parameters:     * p (Polynomial) –
                    * value (int) –
    Return type:    Tuple[Polynomial, Polynomial]

Rationale

Full qualification inherently gives more context, so it is a sane default. However, intersphinx linkage to the correct types gives even better context, since you can just follow directly to the actual type definition. With both intersphinx linkage and unqualified type names, you can have the best of both worlds: readability and quick disambiguation.

Similar prior work

The external extension https://github.com/agronholm/sphinx-autodoc-typehints collapses the name to the unqualified name by default, unless otherwise opted for, for an example of how this might work.

Variant

One possible issue with the above is that there's still no direct control of the type name---either you get the ClassName, or you get the fully.qualified.ClassName. An alternative may be to use whatever is used in the source code. E.g., if the code has import fully.qualified as qual and uses qual.ClassName in the signature, then Sphinx can respect that. +1 because the docs follow the code, but -1 because no consistency is guaranteed. All three versions could possibly be supported, though, and left up to the maintainer which style they go with.

With the above code example, this variant would turn the generated docs into:

mypkg.mymodule.multiply_polynomial(p, value)
    Parameters:     * p (polynomial.Polynomial) –
                    * value (int) –
    Return type:    Tuple[polynomial.Polynomial, polynomial.Polynomial]

[tk0miya]

+1: Could you give an example please? It would be helpful to me for improvement.

[alkasm]

Added examples to the original post, with another variant as a suggestion.

[tk0miya]

Thank you for update. Indeed, it should be improved more. I'll consider the way to modify its name.

[mquinson]

This would also be particularly precious for the cpp domain, with the namespace being sometimes implicit and given by the context (by the englobing namespace constructs) only.

I am wondering. Is my request about the cpp domain really related to the original request? It's hard to know for me because the answer depends on implementation details that I don't know. If it's another request, I'm happy to open a separate issue.

Thanks for your work,

[jakobandersen]

This would also be particularly precious for the cpp domain, with the namespace being sometimes implicit and given by the context (by the englobing namespace constructs) only.

I am wondering. Is my request about the cpp domain really related to the original request? It's hard to know for me because the answer depends on implementation details that I don't know. If it's another request, I'm happy to open a separate issue.

All domains are almost entirely separate in their implementation, so in general I recommend opening separate issues as default. I'm not sure if the exact details of this issue, so please open a C++ specific issue for that discussion.

[mquinson]

Hello, for the record, I opened #8228 for the specificity of the C++ domain. Sorry for the noise.

[alkasm]

@tk0miya Hey thanks for adding this support! I'm trying to build this example off master but when I try to use the new conf option, I get an exception

Exception occurred:
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/writers/html5.py", line 776, in unknown_visit
    raise NotImplementedError('Unknown node: ' + node.__class__.__name__)
NotImplementedError: Unknown node: pending_xref_condition

full output log

# Sphinx version: 4.0.0+
# Python version: 3.7.7 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.3
# Last messages:
#   
#   looking for now-outdated files...
#   none found
#   pickling environment...
#   done
#   checking consistency...
#   done
#   preparing documents...
#   done
#   writing output... [100%] index
# Loaded extensions:
#   sphinx.ext.mathjax (4.0.0+) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.type_comment (4.0.0+) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (4.0.0+) from /Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py
Traceback (most recent call last):
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/application.py", line 347, in build
    self.builder.build_specific(filenames)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 284, in build_specific
    summary=__('%d source files given on command line') % len(to_write))
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 357, in build
    self.write(docnames, list(updated_docnames), method)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 531, in write
    self._write_serial(sorted(docnames))
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 541, in _write_serial
    self.write_doc(docname, doctree)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/builders/html/__init__.py", line 616, in write_doc
    self.docwriter.write(doctree, destination)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/docutils/writers/__init__.py", line 78, in write
    self.translate()
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/writers/html.py", line 70, in translate
    self.document.walkabout(visitor)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/docutils/nodes.py", line 214, in walkabout
    if child.walkabout(visitor):
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/docutils/nodes.py", line 214, in walkabout
    if child.walkabout(visitor):
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/docutils/nodes.py", line 214, in walkabout
    if child.walkabout(visitor):
  [Previous line repeated 4 more times]
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/docutils/nodes.py", line 206, in walkabout
    visitor.dispatch_visit(self)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/util/docutils.py", line 472, in dispatch_visit
    super().dispatch_visit(node)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/docutils/nodes.py", line 1995, in dispatch_visit
    return method(node)
  File "/Users/alkasm/prog/test-sphinx/venv/lib/python3.7/site-packages/sphinx/writers/html5.py", line 776, in unknown_visit
    raise NotImplementedError('Unknown node: ' + node.__class__.__name__)
NotImplementedError: Unknown node: pending_xref_condition

I am able to build the given test case example in the PR, but not my example in my first post above. For a minimal example, everything works as intended with

# bar.py
class Bar: 
    pass

# foo.py
import bar
def make_bar() -> bar.Bar:
    return bar.Bar()

but if I import an installed module (even something in the stdlib) instead of my local bar.py, like

import typing
def make_tuple() -> typing.Tuple: 
    return ()

I get the above error.

[tk0miya]

@alkasm Sorry, I can't reproduce the error. Could you share the whole of your example?

[alkasm]

@tk0miya sure, here's a full repro for me: https://github.com/alkasm/sphinx-unqual-types. Not sure if I just have something setup incorrectly, or what.

[tk0miya]

Thank you for the example. It's very helpful to me! I believe #8996 will resolve the bug.

[alkasm]

Thank you for the example. It's very helpful to me! I believe #8996 will resolve the bug.

Hey @tk0miya, I tried out your branch and I was able to verify it works! 🎉

However I also tried it out with the numpy example I gave in my OP for this issue and found a slight complication. It does work with default settings, but does not unqualify the types with autodoc_typehints = "description" in conf.py as I have in my OP. I still have the same output as in the OP. However if I remove that config value then I get a nice

multiply_polynomial(p: Polynomial, value: int) -> Tuple[Polynomial, Polynomial]

I updated the repo example to add the numpy method and have a commented line in conf.py to toggle the description type hints if you want to play with that example. Happy to have this discussion in a new issue if that's better for tracking btw, since the original intent of this issue is indeed solved.

[tk0miya]

Thank you for your confirmation. I merged the PR now. And I'll take a look the case of autodoc_typehints = "description" tomorrow. It would be helpful to me if you'll post another issue.
Thank you for your support!

[RuRo]

@tk0miya I am having the same problem with sphinx 4.0.1, which seems to have 8996 already applied. I think, that I've tracked the issue to the sphinx-qt-documentation plugin. This plugin connects itself to missing-reference events like this

def setup(app: Sphinx) -> Dict[str, Any]:
    ... # snip
    app.connect("missing-reference", missing_reference)
    ... # snip

however, their missing_reference implementation doesn't have the

    content = find_pending_xref_condition(node, 'resolved')
    if content:
        contnode = content.children[0]  # type: ignore

snippet, which eventually leads to the above Unknown node: pending_xref_condition error.

I could submit this as a bug report to the author of sphinx-qt-documentation, but I wanted to make sure, that this behaviour is intended. It seems to me like a potential antipattern. For example, the above snippet (with minor variations) is already duplicated internally in

• sphinx/domains/python.py:PythonDomain.resolve_xref
• sphinx/domains/python.py:PythonDomain.resolve_any_xref
• sphinx/domains/python.py:builtin_resolver
• sphinx/ext/intersphinx.py:missing_reference

And any plugin that connects to the missing-reference event must also now add this snippet (which is complicated by the fact that find_pending_xref_condition doesn't seem to be available in sphinx<4). Do you think that maybe the contnode value should be resolved when the missing-reference event is created in sphinx/transforms/post_transforms/__init__.py?

P.S. I am not very familiar with the reference resolution code, so I might be 100% wrong.

[tk0miya]

@RuRo Could you file a new issue, please? I think your topic is surely related to this issue, but a different one

[RuRo]

Sure, #9240.

[sidneycadot]

Hi - I came here by googling because I want autodoc to generate unqualified typenames for my Sphinx documentation, as suggested by the top post, eg. to prevent ugly type signatures like shown in the attachment.

Reading this thread it seems work was done to address this and the issue was closed, but I can't seem to find anything in the current Sphinx documentation that describes this feature, so I don't know how to get this to work (if it is even working).

Could anyone point me in the right direction?

Thanks!

[RuRo]

@sidneycadot The python_use_unqualified_type_names config option is available since Sphinx 4.0.

However, bear in mind, that it's still a little buggy (experimental) and currently only shortens the type name if the reference could be resolved.

[sidneycadot]

@RuRo Interesting! I'll try tomorrow.

Is there a good reason why this is implemented as a top-level configuration variable rather than an auto doc-specific configuration variable? Are there contexts outside of autodoc where this variable could potentially be used?

(Sorry if that's a silly question@RuRu -- I am not very well versed in the innards of Sphinx.)

[RuRo]

(Sorry if that's a silly question -- I am not very well versed in the innards of Sphinx.)

No problem, I also don't know, why was this implemented in the python domain instead of directly in autodoc :^).

[sidneycadot]

Well it doesn't do what I want it to do, unfortunately; but thanks for the suggestion anyway. :)

[RuRo]

Well it doesn't do what I want it to do, unfortunately; but thanks for the suggestion anyway. :)

Really? Why not? From your description, I would've expected python_use_unqualified_type_names = True together with add_module_names = False to remove most of the noise in the signature. If you have some free time, consider putting together a minimal example and submitting it as an issue (either as a bug in python_use_unqualified_type_names or as a feature request).

[sidneycadot]

Really? Why not?

The variables you mention do influence the observed behavior, but they leave some module prefixes in place. See the attached image.

If you have some free time, consider putting together a minimal example and submitting it as an issue

I will consider it. But, frankly; if I were to do that for all sphinx and autodoc issues I have encountered since starting to write documentation in earnest, it would cost way too much time. It has not been a great experience, I am sorry to say.

I am now strongly considering to do a bit of docutils diving to make an extension that will be able to clean up the generated documentation at a late processing stage by simply doing a regexp-based search-and-replace on anything and everything inside the doctree that isn't to my liking. This is a big and rather unsophisticated hammer that could fix the effect of this particular issue, and several more I have come across. I think my time is spent better doing that; chasing all the different bugs on their own is not very enticing and would take a lot longer.

[sidneycadot]

@RuRo I accidentally bumped in the solution.

As you can see in the examples I present I have an enum type with an initializer. Such an initializer is rendered by autodoc using the repr() function, which for Enum types does something nasty: it emits eg <EnumClass.enumvalue: 10>, rather than the more natural EnumClass.enumValue.

This somehow confuses the code in autodoc that is responsible for signature handling. In particular, after overriding the __repr__ function of my enum class to emit a sane string, everything works like a charm and I get what I want:

https://pydwf.readthedocs.io/en/latest/pydwf_api/pydwf_utilities.html

So the root cause of my issue was that the standard library enum.Enum.__repr__ doesn't honor the convention of returning a string that can be eval'ed. As I understand it, there's ongoing discussion on the Python mailing list to fix it:

https://bugs.python.org/issue40066

In the mean time, I have filed a bug report with Sphinx to see if they can implement a special-case workaround for this type:

#9272

So as for this particular issue, I am satisfied now — I implemented custom __repr__ methods, as suggested as a workaround in my bug report, for all my enum classes and all tools are now happily humming along.

Thanks for your assistance. Most specifically, what triggered me to keep digging was your surprise a few posts back that it didn't work.