C domain changes of Sphinx 3 prevent to write doc compatible with Sphinx 2 and Sphinx 3

[vstinner]

The Python documentation can be found in the Doc/ directory of https://github.com/python/cpython/ It is quite big: 262k lines of .rst files (and 1300 of extensions written in Python: see Doc/tools/).

It is no longer possible to build the Python documentation with Sphinx 3, many syntax which were allowed previously ("tolerated"?) are now treated as errors: see https://bugs.python.org/issue40204 for details.

According to the Sphinx changelog, it is a deliberate that the C domain is more strict. Extract:

The C domain has been rewritten, with additional directives and roles. The existing ones are now more strict, resulting in new warnings.

The problem is that I fail to find a way to write a documentation which can be built and rendered properly with Sphinx 2 and Sphinx 3. Dropping Sphinx 2 support is not an option for the Python project. Too many Linux distributions currently build the Python documentation with Sphinx 2, and it seems like upgrading to Sphinx 3 will break at least Python, but likely other projects documentation.

It seems like we have the following options for the Python documentation:

• Stay at Sphinx forever
• Backport the ":c:struct:" and ":c:expr:" markups somehow in Sphinx 2 using our CPython Sphinx extensions
• Update Sphinx 3 to tolerate our "incorrect" documentation using :c:type: and :c:type:PyObject\* (invalid C syntax), maybe as an opt-in option
• Update Sphinx 2 to support Sphinx 3 new syntax: something like Python 2.7 which backported a few Python 3 features like b"bytes string" syntax

Example of issue: .. c:struct: MyStruct is required by Sphinx 3 to document a structure, since ``.. c:type: MyStructis now treated as an error. But with Sphinx 2,.. c:struct: MyStruct` section is simply ignored and the whole section miss from the HTML output.

cc @birkenfeld

[birkenfeld]

IMO (but keep in mind, I'm just a watcher at this point): it's fine to introduce more strictness if

• the "corrected" way works in 2 and 3, or
• the new strictness can be switched off or
• made into "harmless" warnings (but keep in mind that some fanatics set -W by default and don't understand compatibility compromises)

So I would vote for a fix here that enables any of these mitigations.

[vstinner]

I am perfectly fine with pushing massive changes to respect stricter Sphinx, but only if there is a way to keep Sphinx 2 support.

[jakobandersen]

There are indeed quite a lot of changes that may be a bit excessive in the amount of breakage. Though, fundamental change were needed to get the domain updated.
As only the newest version of Sphinx is supported I don't think it is realistic to make another v2. As discussed a bit in #7457, I suggest to add a "allow pre-v3 declarations" option to the C domain, where we try to accept as many as possible of the old style declarations, but issue warnings when it happens (basically what @birkenfeld mentioned as well).
If I start a PR with this change it would be great if you, @vstinner, and/or others can help test it so it actually accomplishes the goal.

[vstinner]

If I start a PR with this change it would be great if you, @vstinner, and/or others can help test it so it actually accomplishes the goal.

Sure.

[vstinner]

Thank you! I'm now waiting for Sphinx 3.2 to update Python documentation ;-)