0.8.4 vs 2.0.0a4 performance

[josephernest]

Congrats @lepture for this project! (I'm doing a benchmark of many Markdown to HTML libraries, and mistune seems to be the best!)

I used timeit with the same Markdown document, and:

• version 0.8.4: mistune.markdown(s) took 4ms on average
• version 2.0.0a4: mistune.markdown(s) took 13ms on average (for the same document)

What could be the reason of 2.0.0a4 be 3x slower than version 0.8.4?

Can I use 0.8.4 for my current project, is it still stable?

s = 'my_sample_document'
import timeit
print(timeit.timeit("markdown(s)", setup="from mistune import markdown;from __main__ import s", number=100)/100)

PS: I also found this article, it's linked to this topic: https://getnikola.com/blog/markdown-can-affect-performance.html

[lepture]

I'm aware of this.

[karlcow]

1. version 2.0.0a4 is a complete different architecture. single file vs modules
2. 4ms to 13ms is probably not relevant enough as an example, it would probably require a bigger test file or test individually for each code constructs.
3. i would be necessary to identify the painful lines with a line profiler

Still it would be very difficult to compare 0.8.4 with 2.0.0a4 because of 1.

[josephernest]

Thanks for your answers @karlcow @lepture.

About 2., I think it is rather relevant: I tried on both my home laptop and two distant servers with different documents (both long and short; but I can't post them here because they contain personal data), and I used timeit, etc. the conclusion was always between 2x - 3x faster in favor of 0.8.4

The fact 0.8.4 has a single file architecture might be one of the reasons: I observed for different projects that it's sometimes faster to have a one-file structure (Bottle web framework is single-file)

[karlcow]

See https://gregoryszorc.com/blog/2019/01/10/what-i%27ve-learned-about-optimizing-python/

Starting a Python interpreter and importing Python modules is relatively slow if you care about milliseconds.

About why I was saying it doesn't matter: the x3 really depends on the use case.

And it also probably depends on cold start vs cached compiled version. But we are talking about milliseconds, here so I guess it starts to matter if you are in the business of processing thousands of documents and have to render them right away.

Now if we compare the document generation to let say rtt, latency, etc in case of a webserver. We start to be in a very different scenario where 4ms or 13ms doesn't matter.

Improving performances is good, but not necessary everywhere.

What would be interesting is to understand how it impacts your own use case, that would make it easier to find the right path of correction or not. It's always easier to work on something concrete.

[josephernest]

Thank you for your answer @karlcow. I am indeed aware of these profiling techniques (you're right: the initial setup / import time should not be taken into consideration, and this can be done well by using timeit properly).

4ms vs 13ms was the timing on my local (quite powerful) computer. When I do it on the distant dedicated server, on real (bigger) documents that I need to serve, I have these numbers:

mistune(0.8.4)      24ms-33ms
mistune(2.0.0a4)    84ms on average

(Once again I tested carefully, on many similar-length documents, and I profiled only the execution of the rendering, I did not take the import/initial setup time in consideration, etc.)

As I serve many requests par second on my server, I will probably use 0.8.4: it seems to work fine, so I'm happy with 0.8.4.
(If I take the whole request on my server, 2.0.0a4 would double the CPU time for each page served to the clients, compared to 0.8.4).

PS: anyway mistune is already 5x times faster than other markdown libraries, so it's an excellent solution!

[lepture]

@josephernest there are some edge cases that 0.x mistune will perform very bad. Are you rendering markdown text on the fly? Here is how I use mistune in Typlog:

1. post table contains: content and html columns
2. on post publish, mistune converts markdown content into html content, markdown text is saved into content column, html text is saved into html column
3. in this case, you don't have to render markdown on every request.

[viniciusban]

As I serve many requests par second on my server, I will probably use 0.8.4: it seems to work fine, so I'm happy with 0.8.4.
(If I take the whole request on my server, 2.0.0a4 would double the CPU time for each page served to the clients, compared to 0.8.4).

Hi @josephernest.

How about saving the parsed document instead of doing it in each request?

And if you already do it, have you already thought about converting the markdown source in a queue, detached from the request?

[josephernest]

@viniciusban Thanks for these suggestions. I'll probably try these later; for now ~25ms with mistune 0.8.4 is fine, I think I'll just keep using this version and it's perfect.

[karlcow]

Could be useful. https://functiontrace.com/

[brezniczky]

Hi @josephernest, I was thinking if anything performance critical happens, might be worth grinding the code down on Cython and see how it performs ... "I believe" 0.8.4 seems to be functioning well in my case (and can't see why it shouldn't), probs will give you some benefit (I expect up to 2x without any modifications to the pure python code just putting it through Cython).

Regretfully I haven't done any particular benchmarking, as this processing step isn't foreseeably becoming a bottleneck in the short/medium run in my case.

(By the way, I've spotted that A4 is recommended in the README: pip install mistune==2.0.0a4, might it be worth increasing it to the most recent a6?)

(Update: sorry just now checking your linked article - and indeed, Cython is one of the tested ones there ... anyway, others, maybe even me might be interested if you have any info about the performance implications.)

[lepture]

Here is our benchmark for v3 on my Macbook. Please check v3 branch code.

mistune (3.0.0a3) - axt: 52.11806297302246ms
mistune (slow) - axt: 51.74398422241211ms
mistune (fast) - axt: 55.53317070007324ms
mistune (full) - axt: 60.16802787780762ms
mistune (0.8.4) - axt: 61.09118461608887ms
markdown (3.3.7) - axt: 190.69504737854004ms
mistletoe (0.8.2) - axt: 96.36688232421875ms
markdown_it - axt: 161.29398345947266ms
mistune (3.0.0a3) - setext: 36.865234375ms
mistune (slow) - setext: 34.182071685791016ms
mistune (fast) - setext: 34.09314155578613ms
mistune (full) - setext: 39.25585746765137ms
mistune (0.8.4) - setext: 33.128976821899414ms
markdown (3.3.7) - setext: 119.94290351867676ms
mistletoe (0.8.2) - setext: 81.09426498413086ms
markdown_it - setext: 101.91178321838379ms
mistune (3.0.0a3) - normal_ul: 232.5129508972168ms
mistune (slow) - normal_ul: 242.85101890563965ms
mistune (fast) - normal_ul: 228.6520004272461ms
mistune (full) - normal_ul: 244.9331283569336ms
mistune (0.8.4) - normal_ul: 154.64091300964355ms
markdown (3.3.7) - normal_ul: 322.45707511901855ms
mistletoe (0.8.2) - normal_ul: 295.5038547515869ms
markdown_it - normal_ul: 376.8129348754883ms
mistune (3.0.0a3) - insane_ul: 407.55295753479004ms
mistune (slow) - insane_ul: 406.4030647277832ms
mistune (fast) - insane_ul: 393.967866897583ms
mistune (full) - insane_ul: 416.0141944885254ms
mistune (0.8.4) - insane_ul: 311.251163482666ms
markdown (3.3.7) - insane_ul: 509.9070072174072ms
mistletoe (0.8.2) - insane_ul: 468.11795234680176ms
markdown_it - insane_ul: 311.6879463195801ms
mistune (3.0.0a3) - normal_ol: 101.10306739807129ms
mistune (slow) - normal_ol: 95.48306465148926ms
mistune (fast) - normal_ol: 95.92795372009277ms
mistune (full) - normal_ol: 104.74610328674316ms
mistune (0.8.4) - normal_ol: 69.27800178527832ms
markdown (3.3.7) - normal_ol: 166.14508628845215ms
mistletoe (0.8.2) - normal_ol: 132.5092315673828ms
markdown_it - normal_ol: 146.73089981079102ms
mistune (3.0.0a3) - insane_ol: 182.69801139831543ms
mistune (slow) - insane_ol: 182.79385566711426ms
mistune (fast) - insane_ol: 178.88998985290527ms
mistune (full) - insane_ol: 185.28366088867188ms
mistune (0.8.4) - insane_ol: 198.95100593566895ms
markdown (3.3.7) - insane_ol: 193.27211380004883ms
mistletoe (0.8.2) - insane_ol: 326.793909072876ms
markdown_it - insane_ol: 306.52713775634766ms
mistune (3.0.0a3) - blockquote: 60.926198959350586ms
mistune (slow) - blockquote: 59.05294418334961ms
mistune (fast) - blockquote: 60.379981994628906ms
mistune (full) - blockquote: 63.011884689331055ms
mistune (0.8.4) - blockquote: 41.768789291381836ms
markdown (3.3.7) - blockquote: 240.58294296264648ms
mistletoe (0.8.2) - blockquote: 117.5680160522461ms
markdown_it - blockquote: 134.1688632965088ms
mistune (3.0.0a3) - blockhtml: 29.725074768066406ms
mistune (slow) - blockhtml: 28.44071388244629ms
mistune (fast) - blockhtml: 29.629945755004883ms
mistune (full) - blockhtml: 34.86800193786621ms
mistune (0.8.4) - blockhtml: 66.39289855957031ms
markdown (3.3.7) - blockhtml: 179.1081428527832ms
mistletoe (0.8.2) - blockhtml: 51.030874252319336ms
markdown_it - blockhtml: 100.23188591003418ms
mistune (3.0.0a3) - fenced: 17.323017120361328ms
mistune (slow) - fenced: 17.605304718017578ms
mistune (fast) - fenced: 18.645048141479492ms
mistune (full) - fenced: 18.368005752563477ms
mistune (0.8.4) - fenced: 23.174285888671875ms
markdown (3.3.7) - fenced: 139.7559642791748ms
mistletoe (0.8.2) - fenced: 39.73197937011719ms
markdown_it - fenced: 54.20064926147461ms
mistune (3.0.0a3) - paragraph: 379.1360855102539ms
mistune (slow) - paragraph: 1751.7082691192627ms
mistune (fast) - paragraph: 380.15103340148926ms
mistune (full) - paragraph: 397.8588581085205ms
mistune (0.8.4) - paragraph: 398.9071846008301ms
markdown (3.3.7) - paragraph: 1154.524326324463ms
mistletoe (0.8.2) - paragraph: 3151.219129562378ms
markdown_it - paragraph: 3263.485908508301ms
mistune (3.0.0a3) - emphasis: 88.96899223327637ms
mistune (slow) - emphasis: 58.29119682312012ms
mistune (fast) - emphasis: 86.65227890014648ms
mistune (full) - emphasis: 92.79203414916992ms
mistune (0.8.4) - emphasis: 80.56378364562988ms
markdown (3.3.7) - emphasis: 293.503999710083ms
mistletoe (0.8.2) - emphasis: 128.4170150756836ms
markdown_it - emphasis: 233.23297500610352ms
mistune (3.0.0a3) - auto_links: 13.403892517089844ms
mistune (slow) - auto_links: 16.28589630126953ms
mistune (fast) - auto_links: 14.261960983276367ms
mistune (full) - auto_links: 18.305063247680664ms
mistune (0.8.4) - auto_links: 19.008874893188477ms
markdown (3.3.7) - auto_links: 99.04813766479492ms
mistletoe (0.8.2) - auto_links: 34.68179702758789ms
markdown_it - auto_links: 76.89023017883301ms
mistune (3.0.0a3) - std_links: 75.40297508239746ms
mistune (slow) - std_links: 68.76993179321289ms
mistune (fast) - std_links: 77.87418365478516ms
mistune (full) - std_links: 81.43281936645508ms
mistune (0.8.4) - std_links: 46.09513282775879ms
markdown (3.3.7) - std_links: 152.02832221984863ms
mistletoe (0.8.2) - std_links: 90.77978134155273ms
markdown_it - std_links: 124.54009056091309ms
mistune (3.0.0a3) - ref_links: 166.98694229125977ms
mistune (slow) - ref_links: 140.70892333984375ms
mistune (fast) - ref_links: 165.57002067565918ms
mistune (full) - ref_links: 185.20402908325195ms
mistune (0.8.4) - ref_links: 130.09381294250488ms
markdown (3.3.7) - ref_links: 333.1298828125ms
mistletoe (0.8.2) - ref_links: 222.78594970703125ms
markdown_it - ref_links: 297.23596572875977ms
mistune (3.0.0a3) - readme: 183.7749481201172ms
mistune (slow) - readme: 213.71102333068848ms
mistune (fast) - readme: 177.66094207763672ms
mistune (full) - readme: 196.81501388549805ms
mistune (0.8.4) - readme: 137.41517066955566ms
markdown (3.3.7) - readme: 570.9152221679688ms
mistletoe (0.8.2) - readme: 349.81465339660645ms
markdown_it - readme: 501.8589496612549ms