[autorefs] Support interlinking to non-object references

[masasin]

Hi

I have two projects (project_a and project_b) whose documentation I want to link together. Sphinx has intersphinx, and I found mkdocstrings which can do the same thing. However, the docs were written already, and there was no autodoc used at all. I would like to link to specific pages/headings in project_a from project_b.

What the mkdocs.yml looked like before:

site_name: Project A
theme:
  name: material

markdown_extensions:
  - pymdownx.highlight:
      use_pygments: true
      linenums: true
  - mkautodoc


plugins:
  - search
  - autolinks
  - mermaid2
  - awesome-pages

Under plugins, I added mkdocstrings, but the inventory file raised a 404. I thought that it would automatically generated if a python handler is used, and that the default handler was python. I added enable_inventory and set it to true. The objects.inv file existed, but was only 131 bytes big (almost empty compared to the size of the docs).

Since autodocs isn't used, I also added autorefs as recommended in the docs. This got me the same result, even when I turned of autolinks in case it conflicts.

site_name: Project A
theme:
  name: material

markdown_extensions:
  - pymdownx.highlight:
      use_pygments: true
      linenums: true
  - mkautodoc


plugins:
  - search
  - autolinks
  - autorefs
  - mermaid2
  - awesome-pages
  - mkdocstrings:
      enable_inventory: true

What else should I change in order to create an inventory file with the right references?

(Side question: How do you find out what the name of the project in mkdocstrings is? e.g., how did we know that inventory was supposed to be inventory and not Inventory or inventory_project etc on the Usage page? Does it have to be defined somewhere?)

[pawamoy]

Hello!

Sorry, it's not clear to me what you're asking.
You're trying to reference objects of project A in project B's documentation, right?

Did you use the recipe for automatically creating a reference page for each module? Or do you manually add autodoc instructions to your Markdown pages?

[masasin]

Hi! Thanks for the reply. I am trying to reference pages from Project A's documentation in Project B's documentation, rather than objects from the API. Project A doesn't have an API reference page. It's similar to linking to a page (or a heading on the page) from the Project A tutorial. Autodocs is not used at all in my projects. Nothing is automatically generated. I don't have a reference page either. I used this section as a reference for what I did: https://mkdocstrings.github.io/usage/#cross-references-to-other-projects-inventories I added autorefs because of this section: https://mkdocstrings.github.io/usage/#cross-references-to-any-markdown-heading I thought that this would allow me to link to any page, even if it's just a file that I manually added to the docs. (The page and headings automatically appear in the navigation in the left and right sidebars respectively.)

…

On Tue, Jan 10, 2023, 22:45 Timothée Mazzucotelli ***@***.***> wrote: Hello! Sorry, it's not clear to me what you're asking. You're trying to reference objects of project A in project B's documentation, right? Did you use the recipe for automatically creating a reference page for each module? Or do you manually add autodoc instructions to your Markdown pages? — Reply to this email directly, view it on GitHub <#508 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABF4GPBQ4KRDDH3NN4444FDWRXKALANCNFSM6AAAAAATW7K6FY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[pawamoy]

I see, thanks, I understand now.

Currently, autorefs does not add non-objects references to the inventory.
We could consider implementing it, though I'm not sure yet if this is even feasible.

[masasin]

That's too bad. Thanks for the confirmation. I'll link the live page directly in the meantime.

[chrysle]

@pawamoy I'd like to try to work on this, but I'm unsure how to do this yet.

Therefore It'd help me to test several parts of the code (outputting contents) simulating that mkdocstrings is used as an extension.
Is there any way to do this?

[pawamoy]

To be clear, you'd like to add support for adding regular headings (not mkdocstrings-generated ones) into the mkdocstrings-generated objects.inv file? This change would happen in the mkdocstrings/autorefs repository. Basically, you'd want to "register" every encountered heading (though I don't think the code is made to return these headings to you) within the std domain (not py). EDIT: see comment below.

To start testing things, you can just work on a regular project which uses mkdocstrings, clone autorefs somewhere and install it in editable mode:

cd project_using_mkdocstrings
# create venv, activate it, install deps, etc.
git clone https://github.com/mkdocstrings/autorefs
pip install -e autorefs

Let me know if that's enough for you to start 🙂

[chrysle]

Thank you for this information, it helps a lot!

To be clear, you'd like to add support for adding regular headings (not mkdocstrings-generated ones) into the mkdocstrings-generated objects.inv file?

I plan to start with adding all the regular documentation pages (in std:doc) to the inventory.
I think this could be achieved here? It should be worth a configuration option.

[pawamoy]

If by "all the documentation pages" you mean mapping headings from all pages, then yeah. autorefs maps heading ids to their absolute location within the site.

[chrysle]

autorefs doesn't yet support handling object inventories, does it?

So I think the first step should be transferring inventory.py there.

[pawamoy]

Ha, I was mistaken actually. Let me reformulate everything.

So. When adding the mkdocstrings plugin to MkDocs, it adds its own extension to the Markdown parser. We'll call it the main, or outer extension. This extension does two things:

• it uses Jinja templates to render objects. These templates use a heading filter that both register the heading with autorefs and add the object to the inventory. The domain is the domain of the handler that took care of the object, and the role is passed using the data-role HTML attribute. The value of this data-role attribute is set within the Jinja templates.
• it adds a second, inner Markdown extension (amongst other), only used when rendering docstrings, that detects subheadings (headings in docstrings), and bubbles them up to the outer extension. The outer extension then register these headings to autorefs.

The reason why docstring subheadings are not added to the inventory, is because they don't have a data-role HTML attribute. To implement the feature you're interested in, you would first need to modify mkdocstrings to also add docstring subheadings to the inventory. Since these headings don't have HTML data, we can assume the domain is std and the role is doc (IIUC, but maybe there's another, more fitting default role we could use). The code in question:

mkdocstrings/src/mkdocstrings/extension.py

Lines 137 to 147 in 91ca3fc

	for heading in headings:
	anchor = heading.attrib["id"]
	self._autorefs.register_anchor(page, anchor)
	if "data-role" in heading.attrib:
	self._handlers.inventory.register(
	name=anchor,
	domain=handler.domain,
	role=heading.attrib["data-role"],
	uri=f"{page}#{anchor}",
	)

An elif clause could be added:

elif option_to_add_std_headings_in_inventory_is_enabled:
     self._handlers.inventory.register( 
         name=anchor, 
         domain=heading.attrib.get("data-domain", "std"), 
         role=heading.attrib.get("data-role", "doc"), 
         uri=f"{page}#{anchor}", 
     ) 

# heading.attrib.get("data-domain", "std") and heading.attrib.get("data-role", "doc")
# would allow docs writers to specify the domain and role themselves thanks to the attr_list extension:
# ## My heading { data-domain="my-domain" data-role="my-role" }

Secondly, doing that would mean we only handle subheadings in docstrings. So to handle all headings, we must find a way to detect these headings in mkdocstrings (adding them to the inventory is then easy, and done as described above). Maybe we could reuse the inner Markdown extension that reports headings, and set it as an outer extension as well. Not sure yet of the impact it could have. Or maybe this is doable with the MkDocs plugin hooks directly.

@oprypin, if you could confirm what I say is correct, and makes sense, that'd be great 🙂

[chrysle]

That must have been a lot of work, thank you!

So. When adding the mkdocstrings plugin to MkDocs, it adds its own extension to the Markdown parser. We'll call it the main, or outer extension. This extension does two things:

• it uses Jinja templates to render objects. These templates use a heading filter that both register the heading with autorefs and add the object to the inventory. The domain is the domain of the handler that took care of the object, and the role is passed using the data-role HTML attribute. The value of this data-role attribute is set within the Jinja templates.
• it adds a second, inner Markdown extension (amongst other), only used when rendering docstrings, that detects subheadings (headings in docstrings), and bubbles them up to the outer extension. The outer extension then register these headings to autorefs.

I guess these extensions are stored in extension.py and rendering.py.

The reason why docstring subheadings are not added to the inventory, is because they don't have a data-role HTML attribute. To implement the feature you're interested in, you would first need to modify mkdocstrings to also add docstring subheadings to the inventory.

I would also be interested in adding the whole page urls to the inventory, without any heading anchors. I'm sure this can be achieved, too.

An example for that can be found in the inventory at https://webknjaz.github.io/intersphinx-untangled/towncrier.rtfd.io/.

Since these headings don't have HTML data, we can assume the domain is std and the role is doc (IIUC, but maybe there's another, more fitting default role we could use).

We can consider using std:label.

So to handle all headings, we must find a way to detect these headings in mkdocstrings (adding them to the inventory is then easy, and done as described above). Maybe we could reuse the inner Markdown extension that reports headings, and set it as an outer extension as well.

Not sure what you mean with that... Can't we just extend the functionality of the inner extension?

[pawamoy]

I would also be interested in adding the whole page urls to the inventory, without any heading anchors. I'm sure this can be achieved, too.

Fine by me 🙂

Not sure what you mean with that... Can't we just extend the functionality of the inner extension?

The inner extension only runs when converting docstrings. It will never see the rest of the headings written in Markdown pages.

[chrysle]

Some technical question while working:

I've installed mkdocstrings in editable mode in a project, and would like to access the logs after the documentation has been built. How can I do that?

[chrysle]

@pawamoy Sorry for pinging, but I could not find anything about the question and need to output the contents of some variables in a particular state, for example using log.debug(). Could you help me out?

Also I'd like to ask if I can modify the inventory configuration, so it would look like this:

plugins:
- mkdocstrings:
    inventory:
      enabled: true
      register_headings: true

I think this would ease configuring the inventory, and also leave room for further options.

[pawamoy]

I think you can use standard logging utilities to configure the mkdocs.plugins.mkdocstrings.templates logger, as to redirect/duplicate logs into a log file somewhere on the file system.

Yes we can support a map of items for inventory as long as it's backward-compatible.