Mint Langauge

[gdotdesign]

This PR is for supporting the Mint programming language https://www.mint-lang.com/

[joshgoebel]

Thanks so much for your contribution! Sadly, due to lack of maintainer time we no longer merge additional languages into the core library - but we'd love to have a 3rd party language module if you'd be willing to create one and maintain it over time. It's pretty easy to convert what you've already got into a 3rd party module.

We can host that repository here at the highlightjs organization (and are happy to) or you can host it on your own GitHub. Once you have the repository all setup we will happily accept a PR to add a link to our list of supported languages.

Please read https://github.com/highlightjs/highlight.js/blob/master/docs/language-contribution.rst.

[lukepighetti]

@yyyc514 I put effort into contributing Mint to this project because AFAIK highlight.js is upstream of a bunch of tooling (ie Discord syntax highlighting feature) and none of this tooling will be installing individual language packages. My understanding of the purpose of highlight.js was to be upstream of other tooling.

Can you please correct me if I'm misunderstanding something?

[joshgoebel]

I put effort into contributing Mint to this project

And you still can, via a 3rd party module... the docs clearly state this in several places, if there is something else we could add to make this clearer for people I'd be happy to. I do realize the frustration to think you're doing one thing and then have to do something slightly different.

My understanding of the purpose of highlight.js was to be upstream of other tooling.

I'm not sure that's the "purpose" - we don't have a singular purpose - and if we did I'm not sure that would be it. Many people use the library directly via CDN on their websites/blogs just to highlight code. Many people use it as a dependency in larger projects also, yes.

If those larger projects (and smaller ones also) adding a packaged 3rd party dependency is usually one to two lines of code. But yes, the those larger projects have to decide for themselves which 3rd party modules (or none) they wish to use.

For more background: #2149

The ship has already sailed on this (for now) - the core library maintainers simply don't have time to review/approve/maintain 3rd party languages - and it's hard enough to find time to keep up with the first party ones.

[lukepighetti]

I don't understand why having you create a whole new repo in the highlightjs project is easier for maintenance than clicking the button to accept this PR?

[joshgoebel]

I don't understand why having you create a whole new repo in the highlightjs project is easier for maintenance than clicking the button to accept this PR?

We don't maintain those - you'd have to be willing to. We're merely offering to host it. Long-term hosting them might have proven to be the wrong decision (if it creates an illusion of maintenance), but oh well that's what we did, so here we are. :-)

Sorry if that wasn't clear originally.

[lukepighetti]

I feel like I must still be missing something. So you're saying that there's currently no way to get upstream of tooling that consumes the entire highlight.js core package?

[joshgoebel]

Did you read the thread I linked you to?

So you're saying that there's currently no way to get upstream of tooling that consumes the entire highlight.js core package?

Correct, we are no longer accepting new language PRs.

[lukepighetti]

I did skim the 65 page long thread you linked to me, yes. It just seems unthinkable and completely unbelievable.

[lukepighetti]

If @isagalaev is willing to state that this is inline with the original intent of this project I will accept this stance as being reasonable. My opinion is that if you're not allowing new languages to get upstream of highlight.js consumers then the project is deprecated and no longer being maintained. If that's the case the project should be publicly deprecated so that consumers can migrate to a maintained package such as Prism.

[joshgoebel]

project is deprecated and no longer being maintained.

I'm sorry you feel that way. I'll try VERY hard not to be completely offended at that after all the hours I've poured into the existing first party modules and the parser. :-)

@isagalaev is still around, but isn't so involved in the day to day (other than keeping the website up). This isn't happening without his awareness - we don't have him tied up in a dungeon somewhere against his will.

It just seems unthinkable and completely unbelievable.

It is what it is. It's what happens when you have few maintainers and precious little maintainer time. The project (ongoing development, merges, fixing bugs) was pretty much atrophied for some time before I come along and threw myself into it. I think there were 300-400 stale/neglected issues when I started, 70 defunct PRs destined to never be merged... At least now the project is active again, many have stepped up and chosen to create 3rd party language modules.

I think where we are now was a better outcome than no Highlight.js at all, though sadly it seems perhaps you disagree.

[joshgoebel]

If that's the case the project should be publicly deprecated so that consumers can migrate to a maintained package such as Prism.

3.3 million weekly downloads and I think we're still growing, but hard to say. I think tons of people still find the library super useful - as often I think we try to do MORE than Prism - so definitely feels good to provide people with a choice. You're the first person to push back so strongly with the whole "shut it down" line...

[joshgoebel]

this is inline with the original intent of this project

Just to be clear I'm pretty sure it's not the original intent. I think the original intent was far more open, but over time, things change - sometimes for the better, sometimes for the worse. Again see 65 page thread. :-)

[lukepighetti]

Unfortunately I will have to rely on Prism to get Mint syntax highlight upstream of other tooling even though highlight.js was my original target for this work. This is a non-trivial investment in my time to do so. So perhaps you can understand my frustration when the solution is a single click away for you. As far as I know we have done everything as specified on the highlight.js website. All tests pass. There are no merge conflicts. We have met all requirements.

[joshgoebel]

Unfortunately I will have to rely on Prism to get Mint syntax highlight upstream of other tooling. So perhaps you can understand my frustration when the solution is a single click away for you.

I understand, but it's a matter of time... and it's rarely a single click - that's the deceptive thing... it's future support issues, bug reports, the time to review your language - does it agree with how the core library things about classes, etc. - often a review leads to further changes.

Perhaps you can understand my frustration when you refuse to move a few files around and publish a 3rd party language repository like many others have done - instead choosing to declare our project dead and in need of burial. :-)

As far as I know we have done everything as specified on the highlight.js website.

If the website says somewhere we still merge in new languages please let me know so I can get it changed. The documentation in the project was updated a while back to reflect this.

[lukepighetti]

I don't need a third party library. That is of no value to me. I can do this in my own repo, I would never expect you to host that for me, nor would I want you to. The only value highlight.js has to me personally is to get a language definition upstream of other tooling.

I am reading the docs again and I find them to be misleading. They do not explicitly state that highlight.js is no longer taking submissions. It just says "read this file for more info" and then goes on to tell you how to add a language to the repo.

Even the maintainer's guide says nothing about this. Nor the commit policy. Nor the language contributor checklist. Nor the contributing section of the style guide.

Even the "On requesting new languages" says

Highlight.js doesn’t have a fundamental plan for implementing languages, instead the project works by encouraging development of 3rd party language grammars from contributors. We’re also happy to host 3rd party language grammars at the highlightjs GitHub organization - no matter how obscure or weird.

This means that there’s no point in requesting a new language without providing an implementation for it. If you want to see a particular language included in highlight.js but cannot implement it, the best way to make it happen is to get another developer interested in doing so. Here’s our Language definition guide.

This seems pretty explicit: if you can implement it, you can make a PR.

[joshgoebel]

If you have a suggestion on where/how to make the docs clearer please share or make a documentation PR and I'd be happy to review. If we can avoid this confusion in the future that'd be great to catch it earlier rather than later.

Although again of everyone I've spoken with you're the first person to have such a bad reaction - many just go off an implement a 3rd party module and call it a day. And I assume people do use those 3rd party modules. It seems like you're more interested accomplishing your own goals (getting this upstream) than you are helping the broader community - who perhaps would enjoy a 3rd party Mint module they could use.

I thought the paragraph above was clear:

Highlight.js doesn't have a fundamental plan for implementing languages,
instead the project works by encouraging development of 3rd party language
grammars from contributors.

[joshgoebel]

This seems pretty explicit: if you can implement it, you can make a PR.

Hardly, if anything its implicit. How would you edit it to make it clearer? If you'd like to contribute a documentation fix to prevent anyone else from feeling the same frustration you felt here would truly be greatly appreciated.

[lukepighetti]

So you'll let me spend time contributing documentation but not the language definition I worked on? That's pretty messed up.

[joshgoebel]

So you'll let me spend time contributing documentation but not the language definition I worked on? That's pretty messed up.

Sorry you feel that way.

[lukepighetti]

Thanks for your time.

[joshgoebel]

Do you find this clearer:

    Highlight.js does not have a fundamental plan for implementing new languages
    - i.e., the core team doesn't usually develop new languages. The core team
    instead focuses on parser development, bugs, and supporting the existing
    languages. Core also currently does not have time to review, merge and
    maintain any additional languages (fixing bugs, dealing with issues, etc).

    Instead, the project works by encouraging 3rd party language grammars from
    contributors willing to help develop and maintain them. We're also happy to
    host those 3rd party language grammars at the ``highlightjs`` GitHub
    organization - no matter how obscure or weird. Or you're wlecome to host it
    yourself - we're still happy to link to it.

    This means that *there's no point in requesting a new language without also
    providing a 3rd party implementation* (we'll simply close "Please support
    language Xyz" issues with a link to this explanation). If you'd like to see
    a particular language available but cannot implement it, the best way to
    make it happen is to find another developer interested in doing so.

    For more info on actually developing a language see our :doc:`language-guide`,
    and for information on how to properly package your 3rd party language module
    see :doc:`language-contribution`.

[shellscape]

A few bits that could possibly be a little gentler:

This means that *there's no point in requesting

Please do not request or submit

For more info on actually developing a language

For more info on contributing a third-party language, please

Just my two pennies.

[joshgoebel]

Please do not request or submit

I find that harsher wording, lol.

[shellscape]

Haha fair enough!

[joshgoebel]

For more info on contributing a third-party language, please

The doc is on developing though, developing is not the same as contributing.

[lukepighetti]

Honestly it’s still not clear. It sounds like if you create a language implementation that works you’ll merge it. I would find it to be more clear if you said something like

“Due to limited maintenance resources we are no longer accepting pull requests for new languages into highlight.js. You’re welcome to fork the project and use our guides to implement a standalone language highlighting package for use on your website or blog.”

If it were up to me this would be at the top of the README.md and on the website as well. Every section that discusses contributions would have a similar notice.

[joshgoebel]

You’re welcome to fork the project and use our guides to implement a standalone language highlighting package for use on your website or blog.”

Except you have LOTS of choices other than forking. If someone wants to build a one-off (for their site or project) we've made that VERY easy to do with 3rd party modules... just download the ones you'd like to use and run the main build script. Done. And of course there are tons of different build tools to automate such things in a larger projects.

And if that's too much you can just copy and paste dist modules at the end of the main library and that'll just work too... so the simplest build script would be like cat browser_core.js languages/*.js > everything.js....

That's not even to mention if all you need is 1-2 languages you can just add the script files in your HTML after HLJS and it'll "just work".

If you wanted to create the fabled "community" language package (with every 3rd party language in one bundle) you would merely need to have a large index.js file and then pull in all the 3rd party packages via NPM as well as core. And then have people require highlights-community...

If you wanted to just ship a CDN or browser build of a "community" package, again trivially easy to do with the existing build scripts. A lot of though went into making it easy for people to work with 3rd party modules.

[joshgoebel]

I'll look into adding a small section to the readme though.

[lukepighetti]

Maybe I wasn’t clear why I found getting into highlight.js so critical. There is a lot of tooling (especially Discord’s syntax highlighting feature) that would magically obtain Mint syntax highlighting if we could get it merged into core. The alternative is to reach out to every possible downstream consumer and asking them to add us explicitly. Discord has every language from core enabled to the best of my understanding. This is my context so anything that doesn’t resolve that issue feels like the same thing. Ie, a plug-in, or forking, or having you host a package I can host myself. Hope that helps explain where I’m coming from and I apologize for being hot headed earlier. I should be clear that my comments do not represent the Mint project they are just my own opinions for something that was personally important to myself emotionally.

[joshgoebel]

No, I fully understand where you're coming from. Long, long ago I was on your side of the argument. I still truly am in spirit, but not in actuality because I'm a pragmatist. This was hashed out to death in the super long thread. If someone wants to solve this they either:

• Need to invest a non-trivial amount of time over time and become an actual maintainer here, slowly build some clout, try to change this policy from the inside, etc. I'm pretty sure it would take more than one person though.
• Find the time (or a team) that can create and maintain a community repo that bundles all the languages - and get upstreams to use that instead. If those people existed though (or had time) I don't know why they wouldn't already be helping with core - but they weren't/aren't. I'd really love to see someone try this though.
• Or if the concern is more Discord directly reach out to their support. I'd love to talk to someone higher up inside Discord about their concern or willingness (or not) to use 3rd party modules. Are they even aware of our stance? I dunno. I can imagine several real and valid concerns though, but it'd be nice to have a chance to talk about how to possibly resolve them.

Hope that helps explain where I’m coming from and I apologize for being hot headed earlier.

Accepted, glad you've cooled down.

[lukepighetti]

I’ve already reached out to support asking if Prism is upstream since they have it listed as a dependency. If I can add Mint to Discord via Prism then I’m happy. Thanks for your time.

[jaredlll08]

Find the time (or a team) that can create and maintain a community repo that bundles all the languages - and get upstreams to use that instead.

Figured I would chime in since I have reached out to a Discord developer.
Their reply at the time (and as far as I'm aware, still is) was to get the language into highlightJS, they aren't really interested in depending on other packages to add support for highlighting.

As for prism, I have no idea where you're getting prism from, I can't find any reference to it anywhere in the context of Discord. Discord very much uses HLJS for highlighting as shown below, would be kinda hard to swap between them I would think:

[joshgoebel]

Their reply at the time (and as far as I'm aware, still is) was to get the language into highlightJS, they aren't really interested in depending on other packages to add support for highlighting.

Probably because of support/maintenance issues... if their highlighting is broken they want to know the one place they can go (or send the user) - to just file an upstream issue and then let us deal with it. Just a guess - I'm sure that's at least part of it.

I'd still like to actually talk to someone on the team and I've put out feelers thru other channels.

Discord very much uses HLJS for highlighting

Yep, although it's not a global - I'd wager they are using it in inside a web worker or something - probably good for security also.

[lukepighetti]

I never heard back from them about how to get upstream of their syntax highlighting feature. I thought they had PrismJS listed in their acknowledgements at some point but I looked just now and I'm unable to find it. To the best of my knowledge they use HLJS exclusively and have no interest in implementing any languages outside of core, but they update from HLJS periodically. They also implement every language in core from what I can tell.

TLDR: As far as I can tell the only way to add a language to Discord is to add it to HLJS core.

[joshgoebel]

As far as I can tell the only way to add a language to Discord is to add it to HLJS core.

I wonder if they know that doesn't happen anymore - or if they did if they would care. That's what it would be nice to discuss.

[lukepighetti]

I would wager that they chose HLJS so that they wouldn't have to have this discussion with anyone, ever.

[Lothrazar]

" Sadly, due to lack of maintainer time we no longer merge additional languages into the core library"

So if its not accepting new languages, then highlight.js is dead.

Sounds like we need to convince Discord to drop it completely and pick up a new community maintained fork or something, one that isnt so closed off to progress

[isagalaev]

So if its not accepting new languages, then highlight.js is dead.

I don't know, it's been working for me for the past 14 years just fine.

Sounds like we need to convince Discord

An option to do anything yourself didn't cross your mind? Like, fork highlight.js, add whatever languages you want in it and offer that fork to Discord? Why is it that "community" is always someone else?

Highlight.js is not a product. Please go away, this issue is closed.

[lukepighetti]

Did you guys ever consider using language submissions to get more contributors? Ie, "we'll allow new language submissions if people help contribute to bringing in the next batch of new languages." It's been my experience that open source contribution is like a burst of flame, it's usually pretty easy to get people to help for like a month or so, especially if they get something out of it. Getting syntax highlighting into Discord is a much bigger prize than most open source projects can offer...

I don't see anything particularly grueling about pulling in new languages, but I understand if you guys got burnt out on it. It used to be if it met the guidelines on the site and tests passed it was good enough, right? I just went thru the process so I see no reason why I couldn't handle the next five languages or something, although I don't think anyone actually confirmed if my submission met the old guidelines.

[joshgoebel]

@lukepighetti Language contributors typically are not language or core maintainers. Maintainers have to spend our time fixing annoying bugs that may require detailed research, a large investment of time - for a small payoff, thinking about complex inter-language interactions with our "autodetect" system , sometimes adding complex new features to the core parser. Or just leaving 'impossible right now' issues hang open (which is always annoying and stressful to a small degree).

People contributing their own languages aren't usually in a rush to fix complex issues in other languages (or the parser) that they aren't familiar with.

I don't see anything particularly grueling about pulling in new languages

I won't go into it again because the why is laid out already in the other thread. Suffice it to say that often merging is often only the tip of the iceberg on long-term maintenance of a language. The more popular the language, the deeper the iceberg. Making the lowest maintenance languages the super obscure ones no one is highlighting - and therefore little use in the first place.

Look at the issues we still have open (some flagged beginner friendly) for JS/TS/C++, etc... super popular languages...

It used to be if it met the guidelines on the site and tests passed it was good enough, right?

Yes, although I'm sure it was always subject to detailed review also and unwritten rules about style, grammar design, etc - the same things I keep in mind when reviewing PRs to core languages today.

Although again tests passing today mean tests failing tomorrow due to complex inter-language dependencies with auto-detection. Every new language added makes it harder to add the next one - unless we just started turning off auto-detect. Or the whole system was reimagined.

[joshgoebel]

I'm actually open to a REAL good idea here... something I and @egor-rogov and @isagalaev could immediately actually perhaps imagine working. But so far all I've heard is "please go back to the way it was before" or "I promise to maintain my own language" - sadly neither of which is good enough. People burn out, move on, get hit by busses. Really we'd need new language submitters to maintain several languages (in addition to their own) to make the math actually work out long term, etc.

Did you guys ever consider using language submissions to get more contributors?

This is the closest thing yet I've heard to some sort of actual "proposal"... and there is some truth in it even if core remains "closed". Since we documented the new policy we have had MANY 3rd party language modules created. I assume some amount of people are using them and quite happy with them.

The real problem seems to be largely Discord and people who magically want to slip into the "upstream"... perhaps that should be dealt with as a separate problem. I'd still love to talk to Discord (and other large users) about their perspective on this if anyone had someone on the inside they could put me in touch with.

[joshgoebel]

Also large parser improvements can require sweeping updates to built-in languages - since there is an aggregate advantage to core languages working the same, using the best features, etc. When we added the new keywords:$pattern stuff the other day I had to review all 184 of the core languages myself (and update many of them). The larger the number of core languages, the larger this burden becomes.