Handle inline code spans with multiple backticks

[mgeisler]

Before, options.code_block_token was used for inline code spans. Now we always use a backtick.

As far as I can tell from the reference and from experimentation, code spans should always use one or more backticks and never use something else such as tildes.

The PR also ensures that we insert the necessary number of backticks to quote any backticks found in the code span itself. Before, only the case with a single backtick was handled, now the code can contain arbitrarily many backticks.

This was found using a fuzz test like the one in #55. The difference is that I restricted the input to a single paragraph of Markdown text, meaning that it only fuzzes inputs which match

[Event::Start(Tag::Paragraph), .., Event::End(Tag::Paragraph)]

There seems to be some more corner cases that are not handled already, so I'll try to send more PRs to fix those.

[mgeisler]

One of the issues found by the fuzz test is pulldown-cmark/pulldown-cmark#655. The input "`\n`\n`" is parsed like this:

"`\n`\n`" -> [
  Start(Paragraph)
  Code(Borrowed(""))
  SoftBreak
  Text(Borrowed("`"))
  End(Paragraph)
]

This is wrong since the code span should contain " ", not "".

[Byron]

Thanks a million for your continued work - I feel that you will chip off some misbehaviour with each PR until this crate finally works correctly!

With that said, I have added some assertions for count_consecutive_backticks just to validate that it is supposed to work that way - my intuition is that it should abort early and no test breaks if it does. Maybe there can be another test to pin down the 'non-breaking' behaviour.

Finally, there is a question about whitespace.

Please let me know what you think - even as is it can be merged as I am sure the fuzzer will force all actual issues out already so it's OK to make mistakes on the way and start with fast broad strokes without need for details like the ones I brought up.

[mgeisler]

Thanks a million for your continued work - I feel that you will chip off some misbehaviour with each PR until this crate finally works correctly!

I'm very happy to be able to help!

With that said, I have added some assertions for count_consecutive_backticks just to validate that it is supposed to work that way - my intuition is that it should abort early and no test breaks if it does. Maybe there can be another test to pin down the 'non-breaking' behaviour.

Thanks for adding the tests! I don't think it can abort early since it needs to look for backticks through the entire string. Well... there are two small things one could do:

• Iterate over text.bytes() instead of text.chars() since we are counting an ASCII character (backtick) and since the UTF-8 encoding happens to preserve ASCII characters.
• I guess the function could stop looking when it sees that max_backticks is larger than the remaining number of bytes. So if it has found 10 consequtive backticks and if !in_backticks, then the function could stop iterating 10 bytes before the end of the string. However, I don't think that will be worth it since the vast majority of strings only contain 1 or 2 backticks 😄

[Byron]

This sounds like making these changing to counting backticks goes beyond broad strokes 😁, so let's keep the momentum and make round-tripping possible :).

Thanks again, I can't wait to see more PRs just like this one.

[mgeisler]

Cool, thanks for merging it! I found what looks like another small issue in pulldown-cmark, see pulldown-cmark/pulldown-cmark#657.

[mgeisler]

Cool, thanks for merging it!

I found what looks like another small issue in pulldown-cmark, see pulldown-cmark/pulldown-cmark#657. It seems that combining the two crates like this is a fruitful way to tease out inconsistencies 😄