Sphinx domain for Rust

[munircontractor]

Hi everyone,

I just published an alpha version of a Sphinx domain for Rust, along with a supporting Rust executable to generate the directives from the code. The project is https://pypi.org/project/sphinxcontrib-rust/. It is still in early stages, but has most basic functionality.

https://sphinxcontrib-rust.readthedocs.io/en/stable/index.html has the sample output from running the project on itself.

This announcement has two purposes:

1. Get help with some of the Sphinx internals. Most of this was done by digging through the Python domain's source code, so if someone wants to help me with things like formatting the text of the signatures, adding links for the types in the function signatures, etc. I'd be much obliged. As you can see from the linked docs, the formatting of the Rust directives needs work still.

2. Get some Rust users to take a look at the extension, give it a spin and provide some feedback on things like the organization of the items and the TOC.

Please let me know if anyone is interested in this. Thanks!

[chrisjsewell]

ahahaha, I wish you'd posted this about 8 hours ago; that's when I started writing a rust domain 😂

So firstly, I'm obviously not as far along as you in hooking everything up, but I did basically use the same parsing approach as you; with syn, except I link my python code to my rust code with py03, and allow for (de)serialisation of the analysed package with serde:

So there are improvements there, and I might be interested to collaborate (I wish you wouldn't be on gitlab 😄)

BUT the big issue that I don't see is solved by you or me (with syn) is meta-programming, i.e. macros:
this will not document "properly" meta-generated API objects.

rustdoc does do this, since it uses an intermediate representation generated by the compiler, i.e. after macros have been "expanded": https://rustc-dev-guide.rust-lang.org/rustdoc.html

It does this exactly here: https://github.com/rust-lang/rust/blob/25087e02b34775520856b6cc15e50f5ed0c35f50/src/librustdoc/lib.rs#L773

You can view a debug of this intermediate representation like so:

rustup run nightly cargo rustc -- -Z unpretty=hir-tree

but, to my knowledge, there is no way to hook into this from a third-party crate, without literally building the compiler from source 😒

So, that's the main question I have for you; do you see this as an issue, and can it be solved?

[chrisjsewell]

thanks for the reponse

I can clone to GitHub if it helps, and you can fork from there.

Perhaps you would be interested in allowing it to be moved to https://github.com/sphinx-extensions2, so we could colab there?
Naturally, I would grant you full access privileges, and also any others that wanted to colab

I have already working there on https://github.com/sphinx-extensions2/sphinx-autodoc2, with the similar goal of good auto-analysis, and allowing for both rst/myst docstrings, so it would also be interesting if there is any synergy there

It's likely easier to share the configuration with it, than using a separate executable

yeh and just easier for users; it would be ideal if users are just able to run the full process via sphinx, rather than an annoying two-step approach.
Again this is a goal of https://github.com/sphinx-extensions2/sphinx-autodoc2; to separate the analysis and build phases, but also allow sphinx to handle both of them

I certainly don't want to fork rustc for this. So, I think I'm ok without meta API being handled.

Oh yeh exactly, It's just an annoying limitation

[munircontractor]

yeh and just easier for users; it would be ideal if users are just able to run the full process via sphinx, rather than an annoying two-step approach.

It's still a one step approach for the user in my case. It starts a subprocess for the doc generation once the Sphinx config is available. Users just install the Sphinx extension and edit Sphinx config as necessary. I'll look more into pyO3 today and see if I can use that to read the config from Rust instead of passing it over the CLI.

I have already working there on https://github.com/sphinx-extensions2/sphinx-autodoc2, with the similar goal of good auto-analysis, and allowing for both rst/myst docstrings, so it would also be interesting if there is any synergy there

How are you handling the mixed docstrings? I wasn't considering on supporting that with Rust. Either all docstrings are rst or all are md.

I'd love to collab as well. Let me see what it'll take for me to move over to Github. We can keep discussing things here or in a more private channel till then.

[munircontractor]

Hey @chrisjsewell are you still interested in collaborating? I've have more thoughts on our current discussion after doing some research, and also made some progress with code.

[chrisjsewell]

Heya, sorry I didn't reply before. I am still interested in collaborating, but...
I did create a whole "technical demo" that goes in a bit of a different direction to yours 😅 https://sphinx-rust.readthedocs.io/en/latest/ (in particular, see the design section)

Obviously happy to hear your thoughts, I had to get this finished for a client, and now kinda waiting for their reply before considering how much more time I devote to it going forward

[munircontractor]

Yeah, I see that. It looks pretty nice overall. For me, it is a hobby project, so if your client wants to proceed, I would be happy to merge some of my code with yours, assuming no licensing issues.

I've done some refactoring on my code, and my Rust structs are very similar to your data models. I'm also about to a merge an MR today with much better support for Myst.

From a user perspective, we largely differ in how we organize and present the output. In my case, I organize everything close to how it is organized in the source code, whereas you are organizing it more like rustdoc, with separate file for each item. This is a sample of my output - https://sphinxcontrib-rust.readthedocs.io/en/latest/docs/sphinx-rustdocgen.html.

The internal mechanics are also different between us, but that shouldn't affect the user much. Overall, we're both driving everything via the Sphinx build and configuration. My main concern with using py03 is all the dependencies that come with. It made my builds really slow and blew up the executable size a lot. Other than that, I don't really have any issue with it. So, if we end up merging things, I'll just adapt the code to work with what you have.