Add BiDi support

[ahangarha]

Since Markdown doesn't have any specific tag to specify the direction of [a given block of] a text, and since there are many cases in which the text body is mixed with RTL and LTR text, there is a need to detect the direction of the text automatically based on the context.

This feature is extremely need to be implemented in whatever possible way but the most feasible solution (as per my understanding) is to apply bidirection (bidi) to the render engine so it pass the responsibility of dealing with direction to the browser.

[raphlinus]

I'm open to adding BiDi, but very fuzzy on exactly what would be added to pulldown-cmark to support it. Are you suggesting an extension to the CommonMark language to indicate paragraph directionality?

[ahangarha]

I do suggest to add dir="auto" to the blocks of text. It can effectively handle 99% of the cases.

This is completely practical and nondestructive. Anytime it can be replaced with the newer (if any) solution.

As far as I follow discussions, nobody is taking any action to solve this issue. I guess Gitlab is doing the same thing I am proposing here.

[raphlinus]

Yes, I'm interested in seeing what other CommonMark / markdown implementers are doing here. If there's consensus that adding dir ="auto" is a good thing, I don't have any objection. There's a question of whether it would be behind a flag and, if so, what the default is. One of our concerns is being compatible with the cmark reference implementation, and we'd need that for tests.

[ahangarha]

Can you share some resources in this regard?

I am not hopeful anyone come to any conclusion anytime soon. This is not priority for many people. Otherwise they could propose some solution long time ago.

Still I think bidi is practical and not destructive. it doesn't change thing except adding little attributes to rendered html. anytime it can be undone without any harm

[trinity-1686a]

From scrapping the web a bit on it, there seems to be very few actual implementations of markdown supporting rtl languages.
Showdownjs says it does, but it's demo does not actually do anything about it.
Markdig does support it, using dir="rtl" and matching of the first char, because of pore browser support (which seems to mean IE, others appear to support it fine), it's notably the only implementation listed here that does.
Some implementations don't seems to care about supporting rtl, others don't want to move before proper normalisation.

Some people seems to agree that dir="auto" is a good thing, but others would like an additional syntax to make explicit the expected direction, for content in mixed scriptures (some common requests appear to be <--rtl--, <- or waiting for the attribute syntax ({dir=rtl}) to get normalise).

If I may add my grain of salt, pulldown-cmark should add dir="auto" when necessary, it can't harm in any way, and have it gated behind a flag so that it can stay compliant with reference implementation. However I think explicit ways of choosing are a bigger change that should probably be discussed and normalized before an implementation happen.

[ahangarha]

Gitlab does the same: add dir="auto" to block of text so the first character of the paragraph determine the direction of the whole paragraph.

As a Persian speaking person, I have found this method more logical. I prefer this method as it is easy to understand by people, it doesn't need any complex implementation and it doesn't need any additional syntax. I can do all my formatting on Gitlab just because it adding this attribute to each block of text.

[ahangarha]

I think it is better you implement what I am proposing. I am sure you can prove others that this is the right way of doing it.

Markdown should remain as simple as possible. it is not supposed to support complex and rare cases of formatting text. As a Persian speaking person who is dealing with this issue, I am telling you this method is doing the job perfectly. Show others that it works.

[ahangarha]

Hi again

I have made a Firefox add-on called Add Bidi Spport to demonstrate how my approach would work for adding bidi support to websites. Please install and use it in some websites and see the impact.

It is not made to solve the issue on all websites but to show how with a very simple approach, the issue can be efficiently resolved. Share your comments.

You may activate the addon and use it on showdownjs with the following text:

![Showdown][sd-logo]

یک two سه four


# نصب / یک two سه four

## LTR: One دو three چهار

You can download the latest release tarball directly from [releases][releases]

## Bower

    bower install showdown

## npm (server-side)

    نمونه کد فارسی npm install showdown و ادامه متن

## CDN

You can also use one of several CDNs available: 

* rawgit CDN

        https://cdn.rawgit.com/showdownjs/showdown/<version tag>/dist/showdown.min.js

* فارسی cdnjs

        https://cdnjs.cloudflare.com/ajax/libs/showdown/<version tag>/showdown.min.js

---------

# Syntax


## Table of contents

- [مقدمه](#introduction)
- [Paragraphs](#paragraphs)
- [سرآیند](#headings)
    * [Atx Style](#atx-style)
    * [سبک Setext](#setext-style)
    * [Header IDs](#header-ids)
- [Blockquotes](#blockquotes)
- [Bold and Italic](#bold-and-italic)

There are some fixes needed for lists but as you can see, it works perfect.

Share your concerns with me so that I can improve my solution. This issue must be resolved once for ever.

[ahangarha]

Any update in this regard?

Please see my article on Plume which contains both RTL and LTR text.

You can use my add-bidi-support firefox addon to see how effectively it resolves the issue.

This is the comparison between the two (with and without bidi support):

[raphlinus]

I'm not sure whether there's an actionable proposal here, and also not sure whether it's better taken up as a proposed enhancement of CommonMark.

[ahangarha]

The actionable proposal is presented above. my addon is just an attempt to show how it works.

It seems nobody is interested in addressing this issue. I have proposed it on CommonMark forum but nobody replies.

If there is any better way to approach please let me know. There are many projects suffering from this issue.

[mskf1383]

I'm not sure whether there's an actionable proposal here, and also not sure whether it's better taken up as a proposed enhancement of CommonMark.

I talked with CommonMark team to add this to spec, but they sayed it is not an issue for spec, it is about implemention:

Adding a dir="auto" attribute to p elements (and presumably h1, td, li, and other elements that contain text) is really a rendering issue, not a parsing issue, so this is not really an issue for the spec, but rather for implementations.
commonmark/commonmark-spec#722 (comment)

So you can add this to Pulldown-Cmark. It is not incompatible with CommonMark specs, if you add this.

[Martin1887]

This seems like a task for an external HTML renderer, as #637.

[ahangarha]

This seems like a task for an external HTML renderer, as #637.

If you mean the rendered in the browser, I don't think so. Browser will render what we in HTML and CSS define. Even in Android development, such issues have their own directives and styles.

Unless browsers decide to add dit=auto to all elements (which is not a good thing in all cases), we need to handle this while generating the HTML file.

[Martin1887]

No, I mean an external crate generating the HTML from the events of this crate. Note that the purpose of this crate is parsing Markdown, and the HTML renderer is only a basic one that will surely be separated in optional features as proposed in #519 and #526 (it will not probably be removed because it is useful for tests).

That external crate could have many options and for instance reading them from a TOML file, something much more convenient that tweaking arguments in this crate. Also note that the initial implementation of that crate can be very similar to a copy-paste of the html and escaping modules of this project, and from that starting point growing to a full-featured Markdown to HTML converter.

[Martin1887]

#645 has been created to track all requested features that would fit in a new full-featured HTML renderer crate.