Improvements for Python API documentation

[jbms]

I have recently implemented an extensive set of additional functionality/customizations to Sphinx for building the documentation for tensorstore:

https://google.github.io/tensorstore/python/api/index.html

I think that many of the changes that I made would also be useful to many other users, and would perhaps be good fits to incorporate into Sphinx itself, but I wanted to find out if there was interest in that.

Because a large amount of monkey patching was required, it would be easier to either add the functionality directly to Sphinx, or at least add necessary hooks such that monkey patching would not be required.

The theme itself is derived from mkdocs-material (see bashtage/sphinx-material#96 for details on that), but some other improvements were inspired by pdoc3.

The specific changes include:

• Inclusion of object descriptions in the table of contents, along with optional "icons" (the icons are actually just single letters inside a box). The "icon" provides a concise indication of the object type. For example, "[C]" for Python class, "[F]" for Python function, "[$]" for environment variable, etc. (See: https://google.github.io/tensorstore/python/api/index.html)
  • Status: Table of contents now includes object descriptions. Icons not implemented.
• Support for "object synopses", a brief description text shown as a tooltip (i.e. HTML title attribute) when hovering over an object (in the Sphinx domain object sense) reference link. Currently there needs to be specific per-domain support for these, e.g. for Python objects it is set from the first paragraph of the doc comment. (Hover over links on the right table of contents here, for example: https://google.github.io/tensorstore/python/api/index.html) These are also added to the search index and shown in the search results.
  • Status: No change
• Support for pybind11 overloaded functions. Each overloaded function has its own signature and docstring, which are extracted from the combined docstring that pybind11 generates. Additionally, I specify a unique identifier for each overload, that can be used for cross-referencing and also for use in the object identifier and page name. See for example the multiple "Constructors" listed here: https://google.github.io/tensorstore/python/api/tensorstore.Context.html and corresponding source code here showing the syntax used for specifying the overload id: https://github.com/google/tensorstore/blob/b14939b1b8fa15a210dc3ba67e4bf9bd3cb3908a/python/tensorstore/context.cc#L414
  • Status: No change
• Support for doxygen-style "Group" names within doc comments, to group members: see for example https://github.com/google/tensorstore/blob/b14939b1b8fa15a210dc3ba67e4bf9bd3cb3908a/python/tensorstore/context.cc#L376
  • Status: No change
• New "autosummary" implementation. Rather than using a table to display the summary, the summary just contains another copy of the object description, but with the signature and description abbreviated, and slightly reduced padding. This provides a more consistent styling. (Also supports the "Group" names and multiple overloads.)
  • Status: No change
• A new "function parameter" object type for the Python domain, along with a :py:param role for referencing it. Function parameters are also added to the table of contents, and linked from the signature as well: https://google.github.io/tensorstore/python/api/tensorstore.open.html
  (Similar to https://github.com/sqlalchemyorg/sphinx-paramlinks)
  • Status: No change

[mhostetter]

I WOULD PAY FOR THIS FEATURE!

Sorry for the scream-text. This is exactly what I've been trying to make Sphinx / autodoc / autosummary do for years. Every attempt has been manual, hacky, and incomplete. I was about to abandon hope, switch to mkdocs, and attempt to write an extension that does exactly this (only because it seems the entry barrier for devs is easier in mkdocs). But I stumbled upon @jbms's PR in sphinx-material and saw his work there.

I genuinely hope this can be generalized and upstreamed into Sphinx, even if some functionality stays in the theme (sphinx-material or a fork of it). 🙏

[jbms]

Great to hear of your interest, @mhostetter. Would you be interested in helping work on this to make that happen? I would also love to see the functionality get generalized and integrated into Sphinx, and for the new autosummary/autodoc and the rest of the theme to be more generalized so that it can be used for other projects, but I unfortunately don't have a ton of time I can devote to that. If it is going to happen, I would need some help.

[mhostetter]

@jbms I know little of the Sphinx internals, however I'm happy to help wherever I can. Could you spell out in a paragraph some tasks that would need completed / you need help with. Maybe from that paragraph I can determine if I can technically help you. But yes, I'd love to see this happen and help however I can.

[jbms]

Thanks. I will work on writing up a description of the tasks that are needed --- there are quite a few. I also knew nothing of Sphinx internals (in fact I had never used Sphinx at all) before I started using it for the tensorstore documentation.

[tk0miya]

Cool! It would be nice if we can merge the features to Sphinx core. I don't still understand all your works. But it must be worthy. I'll take time to check each feature in detail.

[jbms]

One relatively simple thing:
https://github.com/google/tensorstore/blob/master/docs/tensorstore_sphinx_material/sphinx_material/inlinesyntaxhighlight.py

As noted in the comments there, that is heavily based on this existing pull request:
#6916

That would be great to resolve in Sphinx, but there also needs to be support for latex.

Since the pygments pull request (pygments/pygments#1343) by @tk0miya has stalled perhaps we can just work around it in Sphinx by stripping the \begin{..} \end{..} from the pygments output.

[jbms]

@mhostetter I will try to file some separate issues linked to this one for individual action items. See #9641 for example.

[PaulSt]

Concerning the

* Support for pybind11 overloaded functions.  

I am currently seeing my overloaded function displayed nicely, however the docstring formatting is rather off, the parameters are not listed as bullet points anymore. This was also mentioned here. Am I doing something wrong? Is that a bug?

[AA-Turner]

#11072 makes the display of Union, Optional nicer in the Python domain and autodoc.

A