New Markdown parser

[willmcgugan]

commonmark is unmaintained and doesn't support tables, and other features.

We should replace it with a new Markdown parser, for the benefit of both Rich and Textual.

markdown-it-py was recommended.

• Check dependancies of whatever we end up using. Ideally there will be no / few dependancies, and pure Python only. Otherwise, we could consider an external package.
• Suggest a similar approach to the current Markdown renderer if possible, with a mechanism to customize blocks.

[kellyjonbrazil]

Markdown table support would be terrific!

[aroundwithalex]

An FYI: markdown-it-py only supports Python 3.7 or above. Consequently, to achieve the requirements, the Python version specified in tools.poetry.dependencies of pyproject.toml may also need to be changed from ^3.6.3 to ^3.7.0.

[willmcgugan]

That may be a step I'm willing to take!

[hukkin]

markdown-it-py v2.1.0 is the only markdown-it-py version without Python 3.6 support. Unless there's a reason that rich must pin to the newest release or later, markdown-it-py dependency will work fine for Python 3.6 users too. A dependency resolver (such as pip's) will simply fetch a supported dependency version. No need to drop Python 3.7 for this IMO (although it may make a lot of sense for other reasons).

[Benjamin-Lee]

Just want to add that MD tables would be a great addition. I'm using rich-click and would love to use tables in my command usage output.

[frostming]

Another parser to recommend: marko.

It supports Python 3.6 and supports the latest commonmark spec as well as GFM. It also has a good interface for custom render to render as rich output. It is a bit slower than markdown-it-py but should be good enough.

[42sol-eu]

@willmcgugan @kellyjonbrazil is there something new on that topic.
Just asking because I was thinking about using / extending rich to understand basic asciidoc - my idea would be to fork the underlying markdown parser (at the moment commonmark but that is not a good idea if fwe will se a new parser in a upcoming release.

• MarkdownIt
• marko

are mntioned here. The first one is referenced from commonmark's reposiftory. But that shoul not be the only reason to use it.

Both look like potent candidates to me. But I guess that should be @willmcgugan call.
From the readme in MarkdownIt I see a nice API. Both claim to are extensible - which should be obvious for commonmark parsers...

Eager to hear your opinions

[kellyjonbrazil]

Hi @42sol-eu! Not an endorsement, but just want to throw in there that I have used Mistune (https://mistune.readthedocs.io/en/latest/) in one of my projects and it seemed to be fairly extendable. I'm not sure how it compares to the others - I didn't have a chance to do a thorough comparison at the time when choosing it. I think one of the factors was its table support. Sorry to just add another option to your plate. :)

Edit: Here is how I'm using mistune in jc-web: https://github.com/kellyjonbrazil/jc-web/blob/master/app.py

[bollwyvl]

This is now perhaps somewhat more pressing, as commonmark -> future now shows up in vulnerability scanners with CVE-2022-40899, and there's been no feedback from maintainers on issues or PRs.

Some observations:

• mistune seems to have settled down after some pretty significant breaking changes from 0.8.0 -> 2.0.0 -> 3.0.0 so may be worth considering (and not trying to code around multiple versions)
  • deps: none 🎉
  • it's "winner take all", inheritance-based extensibility e.g. class MyGrammer(BaseGrammar, SomeOtherGrammar):
• markdown-it-py is also a fine tool, today
  • deps: one, non-stdlib dependency (typing_extensions doesn't really count) , under the control of the same org
  • it uses a functional-style extension mechanism
    • e.g. my_plugin(md); some_other_plugin(md)
  • however there is some risk it will become abandoned by its parent organization, which is writing yet another implementation (long thread)

Over in the jupyter stack, we have stayed with mistune, but haven't gotten fully onboard with mistune 3.

Not sure if it's feasible, but given none of the above are/will be compatible with each other (much less extensible by each other's extensions), another tack might be to leave the door open for a "Bring Your Own Markdown Renderer" API, so rich base package can stay out of the business altogether, aside from defining a very high level API specification. This would leave the door open for major revisions of renderers, e.g. rich-markdown-it@3.x which are directly tied to a specific implementation

[willmcgugan]

We have a branch that uses markdown-it-py for the Markdown parser. It was on-hold because at the time Rich supported Python3.6 and MIP didn't. So in theory we should be able to get that working quite soon.

[darrenburns]

The new parser will be in the next Rich release. It does not add table support - it just gets us to parity with all pre-existing features (although, it does add support for strikethrough text 🙂)

[github-actions]

I hope we solved your problem.

If you like using Rich, you might also enjoy Textual