Adding new languages?

[jaredlll08]

I'm not sure if there message board / other place to ask questions, so I hope this Github issue is fine.

From what I understand, new languages are not being accepted into the main repo, instead they are being put into their own git repo, like Structured Text and Solidity, and then from there being pushed to their own NPM packages.

So my question is:

Is there any benefit to this compared to lets say, me owning the git repo and publishing to NPM myself?

[marcoscaceres]

no big advantage... just makes it easier to find new languages at the organization level.

[jaredlll08]

If that's the case, I have already made a repo for the language support and published to NPM, should I transfer the repo to the org or should I keep it in my own org, what would you prefer?

[joshgoebel]

I think if you'd like it here to make it easier to find, we can oblige. Thoughts?

[jaredlll08]

I appreciate the offer, however I don't specifically see the benefit in having the repo be under the highlightJS org.

I've spoken to many developers and project teams about supporting my language highlighter. They are more than happy to add the feature but their dev teams just want to simply pull in the official library and have all of these languages supported out of the box. They won't even consider adding additional packages and have directed myself and others to add support in the main library. Languages added by adjacent repositories in this organization do not seem to be included in the main download or packaged with the NPM package, which defeats the purpose for many of these projects.

Are there any official policies, criteria, or processes for having a language be added into the main repo?

[joshgoebel]

Languages added by adjacent repositories in this organization do not seem to be included in the main download or packaged with the NPM package, which defeats the purpose for many of these projects.

That's something we need to find a solution for. There should be a simple way to build a package that includes external repo grammars. Long-term I'd like to see the web form doing that... right now you could probably setup a build pipeline with something like Webpack, but I don't find that super intuitive (or I'd prefer the tooling to already be in place so people could invoke it with just a simple command, like the existing tooling).

Are there any official policies, criteria, or processes for having a language be added into the main repo?

We're not accepting any new languages into the core repo now - and long-term I think the grammars will likely move out of core, into their own repos. There isn't time for us the maintain the grammars we already have. If you have time to assist and could perhaps help with the tooling around connecting all this stuff together that'd be awesome. :-) Right now there is too much demand and not enough volunteers.

You can see some discussion related here:
#2149

[jaredlll08]

So I'm not primarily a JavaScript developer, so I'm unsure about "best practices", but I am pretty well versed in "get this working" practices, and have a few ideas (possibly very naive ones) that could solve a lot of these issues, I'll continue the convo on the linked issue.

[joshgoebel]

@jaredlll08 Do you have a specific language you're wanting to see added?

[jaredlll08]

So I would like ZenScript to be added.

ZenScript is a scripting language for Java. It is primarily used by CraftTweaker(Github, Project Page) which is a Minecraft mod that allows users to change many aspects of the game using scripts.

In terms of usage of the language, I have links in my GitHub Linguist PR about open source repositories on GitHub:
github-linguist/linguist#4471

As discussed, it is rather hard to get closed source statistics (but not impossible, I can get them if need be, it would just take some time).

For what it is worth, the GitHub Linguist PR was merged and is currently live on GitHub itself, not sure if this matters.

[joshgoebel]

And is there some service or site using Hightlight.js that's "stuck" waiting on an upstream build (that's unwilling to build a custom package) or you'd just like to see it in Highlight.js just to improve the variety of our offerings in general?

[jaredlll08]

I realize this is what most language requests sound like, but honestly I would be lying if I said anything besides that I would like the language in highlight.js because it is one of the most used Syntax highlighters in the web, and I get a good amount of issues where people upload their script files to sites like hastebin (or self hosted hastebins) which use highlight.js; and people pasting script snippets into DiscordApp, which also uses highlight.js, I did find a work around for discord though, by using the "css" highlighter, but as you can see here: https://twitter.com/jaredlll08/status/1181754530427543558?s=19
It is very hit or miss on what is actually highlights, so a proper solution would be ideal.

In terms of Discord, I did make a feature request on their side:
https://support.discordapp.com/hc/en-us/community/posts/360054118152-Syntax-highlighting-for-ZenScript

After making that request though, someone who works at discord (and is also a fellow modder in the modded minecraft community), reached out and said that they think I should try and get it into highlight.js itself as the language is actually used and then it would be supported everywhere. (paraphrased).

Like I said, I guess this is more or less what all your language feature requests are like, but I'm hoping that through me trying to help the project with discussions and finding solutions it shows that I'm not planning in just getting the language in and running away, it will be something that I would support when issues arise (if issues arise).

[jaredlll08]

I did not mean to do that... Thanks github mobile view

[jaredlll08]

I would also like to add on that naturally adding the language would improve the variety of highlight.js's offerings, but at the same time it would also show that the project isn't stagnant (in terms of adding new content) and is still striving, just looking for more people to help out and contribute.

[joshgoebel]

And do you know what discord app is using in their build pipeline? HighlightJS from CDN, npm, etc?

[jaredlll08]

I believe they use NPM, they use react/react native and viewing source files with inspect element does show node_modules folders.

Also when trying to find the best highlighting for zenscript that discord supported, all of the languages on the highlight.js demo site worked, so I assume it is NPM with all the packages.

I can ask around to make sure that is where they are getting highlight.js from, but all the evidence points to npm.

[joshgoebel]

See I'm thinking (whether npm or not)... @marcoscaceres thought npm was the way to go. Something like:

git clone highlightjs (or npm install it?)
cd highlightjs
npm -i
npm -i --save-dev-whatever zenscript_highlightjs
npm -i --save-dev-whatever highlightjs-robots-txt
npm -i --save-dev-whatever highlightjs-best-of-2019
./tools/build --all --for:web

That seems a pretty easy thing to me... and if someone is doing it as part of a larger project there are probably even easier ways to drop it in the pipeline.

[jaredlll08]

I'm not quite sure I follow, would this be for highlight.js building the package for npm distribution, or is it rather how discord could build a custom highlight.js using npm for their own needs?

[joshgoebel]

How anyone could use it to build packages... (or use the web builder, but that's harder to automate I imagine)...

[joshgoebel]

Pruning issues. Closing this out in favor of continued discussion on #2149.