Idea: Make it easy to plug-in Prism or VS Code/TextMate grammars. #2212
Comments
|
Check out this addon https://github.com/foo123/highlightjs-grammar which works for all syntax highlighters and enables you to define a grammar specification for a language and use it as is in every highlighter of your choice (including highlightjs and prism) |
I think that is the hard part unless you plan to publish and maintain a huge repository of grammars yourself. This is why (most) people come to a library I think, for the supported languages. :-) For most people saying "you just have to write your own grammar" is almost as bad as "you just have to write your own parser". :-) What we'd gain (if say we supported Prism grammars) is instant support for all the languages that Prism supports that we do not. The question is might that hurt us by causing fewer people to want to developer Highlight.js grammars in the first place... |
|
@yyyc514 of course you have a point in that grammars need to be written themselves, else it would be good if a repository of grammars exists. I dont mind having such a repository (maybe in the future) and the repos themselves have already made grammars for some popular languages (eg php, python, css, javascript, html, etc.) as examples on how to create fully-functional grammars. But again the point of the grammar addons started through my own use cases where I had to highlight custom languages or dsl's that were not supported by any known language. And it would be a pain in the neck if I had to create parsers all the time, especialy if the parser was tied to a specific highlighter and had to make a new one for another highlighter. So highlighting custom languages was the starting point of the grammar addons. Of course creating a grammar requires a minimal knowledge (eg BNF specifications) but again it is very minimal knowledge than writing a full parser and having to delve into the internals of a syntax highlight framework. So I think it is a win-win situtation. Again all these are optional, only for those who benefit from such an approach, the others can use what exists already as before. This is only an addon, it is not necesarily a replacer of existing frameworks or languages. |
|
BNF spec... how deep does that get you? Are you really just lexing mostly or is it contextual enough to know you're inside a class, that this is a function call, not a variable definition (they can look the same in C, etc), etc? I'd be curious to see a BNF for JSX or something complicated and see how well it worked compared to say what we currently have. Probably things we could learn. |
|
The parsers created by the grammar addons are similar to PEGs which are mildly stronger than context-free parsers. But there are other features (like grammar Take a look at the xml grammar which not only highlights xml properly but detects errors, duplicate attributes, duplicate ids and so on.. The grammar reference is here It is highly versatile, and eases the burden a lot in creating highly functional and feature-free parsers with only minimal knowledge (and in fact knowledge about a language that is needed anyway, eg what is a keyword, an identifier, a comment for that language and so on..). Of course someone may need a feature not supported by grammar specification or even not possible at all by a grammar specification (and in that case would need to write his own highly detailed parser), but I doubt one will be found. |
Yeah, but that's the easy stuff (in my book). We could support the same if we just had callbacks inside a mode that let you do things like save a little state and dynamically apply the rule or skip it, etc... I'm talking about harder stuff like: |
|
If it can be modelled in a PEG grammar it can be done by the addon.
Yes you can know if you are inside a class or function definition and so on, through grammar actions which create contexts. See for example the scoped grammar (further down the page) at this repo |
joshgoebel
changed the title
joshgoebel
changed the title
|
@foo123 Have you written a tokenizer/parser yourself for your grammar files? Or is all you have a bunch of conversation scripts to convert them into various grammar files recognized by other engines? |
|
@yyyc514 My grammar addons have a full-blown PEG-like parser of their own. |
If that's true then you don't really need 99% of HLJS. You could use merely the new TokenTree plus HTMLRender... or perhaps only a slightly modified HTMLRender... that's only 42 lines of code. First turning your PEG grammars into valid HLJS modes seems like a lot of hard work with very little upside... why not just use a plugin instead that replaces our parser with your own? You abort the actual parsing with a before callback and then in your after callback you'd have access to the emitter and could use it's simple API paired with your parsers output stream. I'm not 100% sure all the pieces are in place to support that, but I'd love to see it tried. I'm very curious to see what a 3rd party plugin/grammar would look like that wasn't actually native HLJS based. |
|
@yyyc514 this is exactly what I was doing in the highlightjs-grammar addon, I tried to bypass default hjs tokenizing and replace with the parser of the addon. But (continouing from the other issue), that was a bit hacky so that is why I asked if a better more modular way exists which is not so hacky. The grammar addons since they sport a full blown parser and support defining a grammar through BNF specifications are really very transferable from one editor/highlighter to another. Grammar remains the same only the editor custom stying tags change. You define the grammar once and use everywhere kind of addons |
And the new way is using plugins. Shouldn’t need to touch the library source at all. |
|
Closing this issue - it never really received a lot of discussion. The new API callbacks for |
|
Hello @foo123, @joshgoebel, @HKalbasi, and @RunDevelopment! I was reading this conversation and also PrismJS/prism#2848. I was considering building new third-party languages that are missed in Highligjt.js, @jgm's Pandic and Prism.js (see the table below) because my enterprise's most of the colleagues to use them on their documentation sites I was also considering using @foo123's highlightjs-grammar and prism-grammar to convert VSCode's third-party JSON syntaxes (converted from other types by @pedro-w's TextMate Languages), as:
because I was planning to include some of these languages, that most of the enterprise colleague use more, for my VSCode Markdown theme project I am writing (using Highlight.js and Pandoc) and for the developers' Angular and React project (using Prism.js). I was working on changing Highlight.js and Pandoc CSS colours to match the Prism.js colours, based on the developers' ANgular project, for Markdown project.I noticed the missed language in comparison between syntax highlighters. @jgm's Pandoc missed more languages than both Highlight.js and Prism.js, and Highlight.js missed more than Prism.js. You can analyse the table: Table of comparison between syntax highlighters supporting languages🚫 - Not available
Missed languages in all syntax highlighters
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Interesting tool... though personally I'm far more interested in integration at runtime, like if perhaps we could all agree on some type of loose intermediate representation or API... such that you could just use "all the highlighters" - and pick your output type/theme (prism/highlightjs/etc) but then not have to care about which engine is doing the actual parsing. I'd be curious to know if @foo123 just invented this JSON spec from scratch or if it's some type of standard? It would seem far easier to agree on a simple output API here (token, type, etc) than "one true" unified parser spec... |
|
Would you have any thoughts on this? Such as if Prism.js [for example] implemented our own The rough API: https://github.com/highlightjs/highlight.js/blob/main/src/lib/token_tree.js#L104 |
@joshgoebel The answer is both. As explained here the format used BNF+JSON, combines two standards: BNF format for grammar specification and JSON format (that includes the BNF format) as a versatile and ubiquitus data format. The details of the JSON format (ie names and the meaning of fields) was made up by me based on conceptual separation of functionalities (ie lex model, syntax model, style model, extra settings), but can be adjusted easily. |
Honestly, yes. I've already seen multiple projects having to make multiple grammars for different syntax highlighters. I think it would be a win for the wider ecosystem if the two largest JS syntax highlighter supported the 3 most common grammar formats (TextMate grammars, HighlightJS grammars, PrismJS grammars). We are currently working on Prism v2 and that includes a new custom tokenization API. This API is very simple. It's essentially just a function Themes are a different beast, though. Both HighlightJS and TextMate use scopes to assign colors. Prism does not. So while it should be possible to approximately map HighlightJS/TextMate scopes to Prism class names, the opposite direction will be harder. However, it's not impossible. Prism has standard tokens with clearly defined semantics. So just mapping these standard tokens to scopes will do 80% of the job. Now for the remaining 20%. While tokens can have any number of class names, I would map these class names to scopes as follows:
Not a perfect approach, but it should be a good start. |
Is this API still in development, is feedback welcome? I already shared our API we use for this (internally), would love to see what you're idea is looking like (interface wise). On our side implementing this means tying these two interfaces together. IE, to produce "prism" brand highlighting we'd have a custom Emitter that built a TokenStream... then it's |
Still under development (current API) and yes :) The basic idea is that Prism's tokens, token streams, However, the user-facing API remains the same. Users still just call |
|
@RunDevelopment Is your full list of scopes/class names documented anywhere now? I think we'd just need to work on a mapping table of our scopes to your class names.
And the TokenStream API stable then? If so (from our side) I'm not sure we'd even need the per-grammar support... I imagine we could provide a wrapper on our end: let javaPrism = hljs.wrapPrism(prism.languages.java)
hljs.registerLanguage("java", javaPrism)And to our own grammars we'd add some sort of And internally we'd do something like: // call Prism.tokenize
// create an empty HLJS TokenTree
// loop over the Prism TokenStream, feeding it into our own TokenTree
// call `to_html` on our own token treeAll the magic happening inside the loop where we'd have to deal with conversation of your class names into our scopes. Does that sound about right? |
Could you possibly provide some actual examples for your 20% edge cases? When would you have multiple or non-standard tokens? Is this just because you'd allowed your grammars to get a little wild with what they are tagging things? |
|
@RunDevelopment Any chance we could talk you into adding some actual metadata to your own grammars in v2 like languages name, common extensions, etc? :-) Otherwise it's probably going to look more like so we can add back critical metadata: let p = hljs.wrapPrism(prism.languages.csharp, {
name: ".NET C#",
aliases: ["cs"]
})
hljs.registerLanguage("csharp", p)It'd be nice if you just pre-packaged it. :) |
Yes, and no. We have documented our standard tokens that are widely supported by themes and used in our languages, but there are also non-standard tokens.
Prism themes work using CSS specificity. So by adding more non-standard tokens, you gain more fine control over your highlighting. One example would be keywords. Many modern IDEs highlight control-flow keywords differently. In Prism, you can just give control flow keywords an additional Even more fine-grained control can be obtained by adding more non-standard token, but most languages don't do that because even one non-standard token is already very specific.
Yes. I believe the API has been stable since v1.0 and there are no plans to change it in v2.
That's also what I was thinking for Prism. We would just need a function that wraps HLJS grammars into a custom tokenizer and that's it. Ideally, that function would even be another package completely independent of Prism. Instead of just adding support for a few grammar formats as part of the core library, I'd rather have independent packages that use Prism's API to support different parsers. This would enable people to write third party adapters for their own parsers using the same APIs.
Yup, that sounds like it'll work :)
We will ship aliases in v2 (example), but no titles. The thing with titles is that they aren't that useful. We only have one plugin that needs titles, and it just has a generated mapping with all titles. What does HLJS use titles for anyway? |
User convenience and completeness? LOL. I think they got added because I wanted to use them in the UI of a project.
You're not wrong. 🤔 |
|
@RunDevelopment And it works. :-) About 30 lines of code without a proper style name mapper. 2-3 line patch to HLJS itself to allow grammars to provider their own tokenizers. Is there some good way to enumerate all the languages? Just |
|
See also #3620. Laying out all the pieces for anyone who wants to play with this. |
Not yet. Prism v2 will have a However, that only gets you all currently loaded components (=languages + plugins) of that Prism instance. So there might be other instances with more. We could expose our auto-generated list of all languages, but I'd rather not because that doesn't include third-party components. It's also highly version specific. |
|
I guess i was imagining perhaps a "simplest thing possible" API like: importToHLJS(Prism, hljs)Where that just asked Prism for the "supported/loaded/installed" languages and then loop, wrap them, and inject them into HLJS. |
|
Ah, in that case, enumerating all languages should be enough. |
Brilliant idea or terrible idea? Would this enhance the ecosystem or destroy it?
The text was updated successfully, but these errors were encountered: