(feat) include ESM in CDN build

[aduh95]

Currently, this PR doesn't include an ES module pre-compiled with common languages. ESM user need to load both core and each individual language module. Should I change this?

Fixes: #3199

Changes

Checklist

• Added markup tests, or they don't apply here because...
• Updated the changelog at CHANGES.md

[joshgoebel]

Why would this matter for web usage?

[joshgoebel]

I see now that the npm exports has been drug along into the v11 CDN package unintentionally, but I'm not sure this is necessary at all. There are two common uses cases for CDN:

• Purely a delivery system to get our files onto CDNs, where they are directly loaded/imported (most common)
• Via npm when someone really wants JUST the build assets to directly copy into their project

I'm not sure exports is necessary for either of these cases.

Thoughts?

[aduh95]

I don't think it's necessary indeed, but I thought it wouldn't hurt to keep it I think. It could be helpful for an hypothetical future tool that can generate an import map from a package.json "exports", and humans can also read it to understand the structure of the files to understand which url they should import.

[joshgoebel]

Currently, this PR doesn't include an ES module pre-compiled with common languages.

I believe we'd still want a common build - and I think it would be a monolith (ie, languages actually compiled in)... someone just wanting the "default it just works" setup but also wanting ESM should still be able to have a single source to import to get started, etc...

[joshgoebel]

Also currently failing tests.

[joshgoebel]

Doesn't look like there is any way to test ESM web builds with jsdom?

jsdom/jsdom#2475

[joshgoebel]

@oscarotero Could you pull this, do a cdn build and see if Deno like the resulting build files in build/es?

[aduh95]

@joshgoebel I've added tests for Deno to test the ESM build. I'm not suggesting we should merge it, but I thought it could be interesting as a POC. wdyt?

[oscarotero]

@joshgoebel
I've tested and seems to work fine!

For convenience for Deno developers, I recomend not to minify the code, so it's more easy to see the file and line of possible errors.
To improve the discoverability, maybe add the package in the Deno register (https://deno.land/x). It's not mandatory but it's really easy to do (it works automatically with git tags, so you don't need to do any extra work). I can help you if you want.

[joshgoebel]

My question about testing was specifically related to a browser-like environment, not Deno per se. Yes, it's obvious we could perhaps add a few Deno tests... but let's have that discussion elsewhere (perhaps in the new issue I just created).

Could you please open a second PR with those last few changes and link it to the new issue?

[aduh95]

Could you please open a second PR with those last few changes and link it to the new issue?

I actually need the changes from this PR to run the Deno test (because it needs plain ESM to work), should I remove it from this PR and open a new one when/if this one lands, or should I leave it in (after, it tests that the ESM output is somewhat valid so I suppose that's valuable).

[joshgoebel]

I actually need the changes from this PR to run the Deno test (because it needs plain ESM to work)

Yes, you'd just push your changes as-they-were to another branch. I wasn't suggesting start form scratch on main or anything... then when this is merged that branch could get rebased - and at that point only your test changes would remain.

I don't think we need to do this any longer though... as I don't think these are good baseline tests for Deno as they don't prove anything truly useful. They might serve as regression tests for the API, but they don't provide any real confidence that Deno has the same behavior as say Node.js since they don't exhaustively test the engine (or really test it much at all). If Deno is compatible in the least then we'd expect basic function dispatch to work.

it tests that the ESM output is somewhat valid so I suppose that's valuable).

I don't believe these Deno tests are a desirable way to "test ESM output in general" either... that would be better done by trying a sample bundle/build with ESM during our CI as we do already to test some bundler edge cases. The only reason to add Deno specific tests at this juncture is if they provide value in pushing us towards perhaps officially supporting Deno in the future.

As I mentioned on the other thread if the desire is to move towards "Deno is officially supported" then we should start with porting test/markup/index.js to Deno... that is the most bang for the buck we could get and would provide a high confidence that our engine has the same overall behavior running in Node, in the browser, or in Deno.

[joshgoebel]

To be clear there's nothing wrong with the tests per se, I just don't feel they have value on their own right now... if you wanted to go a step further and add the markup tests as well then the API tests would definitely be great to have in addition... as a regression check for the API on Deno... but just not as our only Deno tests.

[aduh95]

Thanks for the clarification, I'll open a draft PR with the Deno tests after this one lands.

[joshgoebel]

This still seems broken when a language uses module.exports...

hljs.registerLanguage("cshtml-razor",(()=>{"use strict";module.exports=n=>{
var e="built_in",s={},a={begin:"}",className:e,endsParent:!0},i={begin:"{",

It also breaks the ESM build since it now includes module.exports which only works on node, not in the browser...

Previously this was dealt with since we compiled to IIFE first (which had the effect of normalizing this)... now that you are directly compiling to ES it seems to just leave module.exports rather than rewriting it to export default... is there perhaps some Rollup flag to require the exports to the simpler syntax?

[aduh95]

@joshgoebel where do you see this module.exports? I'm unable to reproduce:

$ yarn build-cdn
✨  Done in 8.57s.
$ grep -r 'module.exports' build/   
build//highlight.js:if (typeof exports === 'object' && typeof module !== 'undefined') { module.exports = hljs; }
build//highlight.min.js:;"object"==typeof exports&&"undefined"!=typeof module&&(module.exports=hljs);

[joshgoebel]

Just add a 3rd party module that uses module.exports in extra, such as cshtml-razor in my example.

[aduh95]

I didn't know about extra languages, I'm surprised this didn't come up in earlier discussions about switching to ESM. In a separate discussion, we might want to discuss deprecating the use of CJS for those maybe?

In any case, I've added support for detection of the "type": "module", if they are written as CJS I'm adding back the Rollup CJS plugin which fixes the issue.

I've also added support for "exports" field, that might be a bit off-topic, tell if you prefer I open another PR for this – or if you are simply not interested in supporting it for now.

[joshgoebel]

I didn't know about extra languages, I'm surprised this didn't come up in earlier discussions about switching to ESM.

Didn't think of it. Of course 3rd party languages go straight into the build pipeline just like 1st party ones so it should have "just worked" had you not also changed the input processing in what I presume was an effort to improve it.

In a separate discussion, we might want to discuss deprecating the use of CJS for those maybe?

I'm in no hurry to deprecate CJS - and still think it's the simplest way to get started for many of our users, but I've added an .es.min.js build to dist now as well. If we can continue to support CJS by simply using one additional plugin with Rollup then I feel need to pressure our 3rd party grammar authors to do extra work. If they want to switch to ESM, they can - we've always supported authoring using ESM for 3rd party modules.

I've added support for detection of the "type": "module" ... I've also added support for "exports" field

This is complexity we do not need at the present time and also I don't think guaranteed to be backwards compatible. module is not a great heuristic since some grammars have a package.json, but aren't even real NPM packages (or following the overall NPM spec). Expecting grammars authors to get this 100% right is something we could decide to enforce in the future, but right now it could be a breaking change.

We also have to support support grammars with no package.json at all just by finding their files in the typical src/languages/*.js location. Those files could be CJS or ESM. So the input pipeline needs to be agnostic.

I've reverted back to the much simpler IIFE build process we used before. ESM modules are just IIFE + added export at the end. Since variables should be file local I think this works, am I mistaken?

 const esm = `${data};\nexport default hljsGrammar;`;

[joshgoebel]

I've also added support for "exports" field,

I'm also not sure becoming further tangled with NPM in this way is a step in the right direction. It would be much simpler for us to drop package.json scanning altogether in favor of "one true way" :

• grammar files go in src/languages, CJS or ESM.

I don't find this an onerous restriction, and of course someone is always free to use their own build process. BUT since what we do now (supporting src/languages or package.json) hasn't been a burden (yet), I left it in place with v11. I'm not in any rush to add a lot more variations.

[aduh95]

Good work on simplifying the IIFE build process, I think I had some kind of misconception about how the CJS plugin was working, I ended up doing much more work than necessary, oh well.

I'm also not sure becoming further tangled with NPM in this way is a step in the right direction. It would be much simpler for us to drop package.json scanning altogether in favor of "one true way"

Fair enough, I'd argue that package.json is no longer an NPM-only thing – Node.js itself now uses it, as well as most modern bundlers – but I agree that there is no need to jump on board.

Let me know if there's something more for me to do in this PR.

[aduh95]

Out of curiosity, what's the reason for using async/await here?

[joshgoebel]

Uncertainty about how async works? :-) Are you going to tell me it's effectively the same either way?

[aduh95]

Sorry for the back and forth, but I think it's more efficient to ask rollup to output an ES module here:

Before:

var hljsGrammar=(()=>{"use strict";return e=>{
/*...*/}})();export default hljsGrammar;

After:

function ada(e){
/*...*/}export default ada;

It saves a few bytes (which is always nice for a CDN build), and produces the exact same output for the IIFE files.

[joshgoebel]

but I think it's more efficient to ask rollup to output an ES module here:

Maybe but I've kind of hand verified things the old way (before your last commit) and getting kind of burnt out on this PR... so we'll go with it as it is and then we can circle back in the future.

What you're suggesting though is the REAL problem before was the use/lack of use of plugins when you switched to ESM, not the actual switch from IIFE to ESM that was breaking the default.exports?

[aduh95]

What you're suggesting though is the REAL problem before was the use/lack of use of plugins when you switched to ESM, not the actual switch from IIFE to ESM that was breaking the default.exports?

Yes that's exactly it, I had removed the plugins because I didn't know about extra languages written in CJS.

Maybe but I've kind of hand verified things the old way (before your last commit) and getting kind of burnt out on this PR... so we'll go with it as it is and then we can circle back in the future.

No problem, take care!

[joshgoebel]

Thanks for all your work on the PR and providing momentum to help get this done. 👍 I plan to push a release this week I think.