ENH: add new sphinx-external-toc package

[agoose77]

This PR adds a new sphinx-external-toc package. The intention is to replicate the functionality of the sphinx-external-toc Python package, for later plumbing into MyST-CLI and JupyterLab-MyST.

This PR:

• Adds support for upgrading legacy ToCs
• Adds support for book, article, and no-format ToCs
• Implements ToC validation using JSON Schema

[fwkoch]

Looking really nice! Didn't totally parse all the coercion logic, but it looks sensible. And the JSON validation from types is good - easy!

[fwkoch]

Worth adding validate.errors to this message? I think you were saying those errors were sometimes totally undecipherable because of all the complexity with unions, so probably not, I suppose.

[agoose77]

I'm not sure -- as you note, I found that the errors are fairly non-intuitive. I think for now it might be sufficient just to complain about an invalid TOC.

[fwkoch]

I was just wondering if this should be a separate package on executablebooks, since it isn't really MyST related...?

[agoose77]

That's a good question. I think for now we keep it here because it will exclusively be used by myst-cli. Soon it will end up in jupyterlab-myst / jupyter-book, and at that time we can pull it out once the organisational structure is clearer?

[rowanc1]

@agoose77 looks like test are failing here. Are you working on that to get this ready to review?

[agoose77]

agoose77 looks like test are failing here. Are you working on that to get this ready to review?

Typical! I just rebased it, so I'll check out what's regressed. I think it's to do with AJV. c.f. ajv-validator/ajv#2389

[rowanc1]

I understand the TOC round-tripping issues a bit better. Thanks for the call @agoose77.

Plan we discussed:

• merge the first version of sphinx-external-toc when tests pass here (@agoose77 can advise)
• second PR (ENH: use new TOC package in myst-cli #957), which uses it and replaces the current reader is breaking because we are (potentially) round-tripping incorrectly. The current implementation is not as strict as the spec, and the writing can be wrong. We can either leave that for a bit, or have a fallback to the current implementation and a warning.
• We should have a simpler format for internally reasoning about tocs (probably unist based!), that requires more though, especially about numbering. That thinking/proposal can happen any time, maybe @agoose77 can have a go.