automodule places members under docstring headers

[asmeurer]

Describe the bug

Whenever you use

.. automodule:: mod
   :members:

Sphinx inserts the module docstring, then inserts the members under that docstring. If the docstring contains headers, the functions are all placed under the bottommost header.

This is hard to tell in most themes, because it isn't actually evident what lives under a given header. However, you can tell if you inspect the webpage.

This came up as I was working on creating an extension to add autodoc functions to the toctree (see https://gist.github.com/agoose77/e8f0f8f7d7133e73483ca5c2dd7b907f and #6316). With this behavior, it places the functions under the module headers in the toctree, which is not what I want.

How to Reproduce

I have created a small reproducer project here https://github.com/asmeurer/sphinx-automodule-test. There is a build here https://www.asmeurer.com/sphinx-automodule-test/

You can see if you inspect the page that mod.function is under subheader, and mod.submod.function2 is not.

In practice, this comes up in SymPy in several places, for example on this doc page. With the current behavior and the extension I am working on, all the functions in that module end up under the "authors" header in the module docstring, whereas they ought to be at the top-level.

Expected behavior

I dug into the code and it looks like the problem is that .. automodule with :members: simply generates RST like

.. module:: mod

<module docstring here ...>

Header
======

.. autofunction:: mod.function

Which tricks RST into thinking that the module headers are part of the top-level document.

It would be better if this instead put the module docstring as content of the module directive, like

.. module:: mod
   
   <module docstring here ...>
   
   Header
   ======

.. autofunction:: mod.function

However, py:module is different from every other directive in that it does not allow content. Is there a reason for this? If py:module worked like every other directive and allowed the docstring to be included as content, then something along the lines of

diff --git a/sphinx/ext/autodoc/__init__.py b/sphinx/ext/autodoc/__init__.py
index 931c14049..5037d340e 100644
--- a/sphinx/ext/autodoc/__init__.py
+++ b/sphinx/ext/autodoc/__init__.py
@@ -956,6 +956,12 @@ class ModuleDocumenter(Documenter):
         merge_members_option(self.options)
         self.__all__: Optional[Sequence[str]] = None

+    def add_content(self, more_content: Optional[StringList]) -> None:
+        old_indent = self.indent
+        self.indent += '   '
+        super().add_content(more_content)
+        self.indent = old_indent
+
     @classmethod
     def can_document_member(cls, member: Any, membername: str, isattr: bool, parent: Any
                             ) -> bool:

would fix this (not claiming that is the cleanest fix on the autodoc side, but I believe it would get the job done).

Your project

https://github.com/asmeurer/sphinx-automodule-test

Screenshots

No response

OS

Mac

Python version

3.9

Sphinx version

master branch

Sphinx extensions

sphinx.ext.autodoc

Extra tools

No response

Additional context

No response