Hook / event handler needed to run after Markdown pre-processing

[vermiculus]

From facelessuser/pymdown-extensions#2074, it seems like on_page_markdown is run too early to see the completed work of snippets – meaning it seems to be impossible to access the content of included files for effect.

A proposal from that thread is a new hook that runs its logic as the last preprocessor before rendering markdown content into HTML (between on_page_markdown and on_page_content).

Assuming this is desirable (since the trivial example in my other report exposes a gap in the standard hooks), what should a PR to address this make sure to consider? Any ideas for a hook name?

[pawamoy]

As discussed in the pymdownx discussion, on_page_markdown does not run too early, it does exactly what it is supposed to do!

I wouldn't say there's a gap in the hooks, since the Markdown rendering process runs "atomically" from the point of view of a plugin. If you want to insert logic before/after specific pre-processors, the way to go is with a Markdown extension of your own.

In other words, instead of trying to insert logic in the Markdown rendering process from a plugin hook, you should try to insert your logic using a Markdown extension. From what I understand, you don't have a requirement of using a plugin hook: it's just that it's more convenient than writing a Markdown extension?

If your extension simply searches and replace contents, it shouldn't be too hard to write. Just make sure to have it run after the snippet pre-processor, then you can include this extension in markdown_extensions, in mkdocs.yml. WDYT?

[oprypin]

Yes- the thing is, if we were to do a search for MkDocs plugins that contain on_page_markdown, we can predict for almost all of them in advance that the solutions they provide are brittle and would much better be implemented as a Markdown extension.

Say there's a plugin that recognizes links that are similar but subtly different from normal Markdown links (ghm).
There's no way for it to know that the snippet isn't inside a code block (ghm). Maybe it could try to prevent detection inside inline backticks but there'd be no end to the edge cases.

If you have a need to access the full power of both MkDocs plugins and Markdown extensions at the same time, the plugin can and should insert its Markdown extension in on_config rather than ever using on_page_markdown.
See how it's done here as an example:
https://github.com/mkdocstrings/autorefs/blob/4a45c3bbc08b6f403f25c513f6553109ae6ae5d2/src/mkdocs_autorefs/plugin.py#L129

The real problem is that this approach is not documented anywhere and as such has been kinda secret knowledge.
Additionally I've been considering making this even easier for hooks - add some API in MkDocs that would allow registering a class as a Markdown preprocessor or treeprocessor without some of the other boilerplate.

[vermiculus]

I'm not saying on_page_markdown isn't doing the appropriate thing (literally what's on the tin); all I'm saying is that there is no hook that works for my use-case :-) I do know what maintaining a large open-source project is like – and what maintaining customer-facing API is like. I'm not trying to say the current implementation is wrong; I'm trying to solicit feedback before attempting to fill in a gap that I've encountered.

From what I understand, you don't have a requirement of using a plugin hook: it's just that it's more convenient than writing a Markdown extension?

You're correct enough – I attempted to write an extension and wasn't able to actually get it to load without attempting to install it locally (which I did not attempt since I do not want this extension installed on my machine). I'm not using any of the standard tools like pyenv or similar that might resolve this specific concern since I don't really work with python outside of this context (and I've already got enough toolchains to be concerned with).

I chose to use a hook since I was able to track the hook's source next to the documentation and have MkDocs respect it.

Any sort of complete, integrated example site demonstrating how to load a 'hello, world' extension from source would be good to reference from the MkDocs documentation.

I'll give implementing a real plugin another shot; it looks like the documentation has been updated since I was last looking at the core approach I've been taking.