11.0.0-beta1 is available for testing.

[joshgoebel]

How to get it:

• npm install highlight.js@11.0.0-beta1
• build it from source (fetch the 11.0.0-beta1 tag)
• fetch it via CDN.

Please open a new issue if you have bugs or issues to report.

[wooorm]

Testing this out right now. Just writing down my notes upgrading:

1. Less languages supported, interesting
2. types changed; nice that they’re no longer global!
3. result.emitter no longer exists? it’s now result._emitter but the types haven’t updated edit: my mistake
4. I’m seeing dots in classes now? e.g., title.function — those are supposed to be two classes I guess? And with a prefix they’d turn into hljs-title and hljs-function?
5. Works well!

Do you have an idea about when it’d be ready?

[joshgoebel]

Less languages supported, interesting

What? You just mean common? We try to keep that to our idea of what's most popular/useful. The total number of supported languages in core actually has gone up (by 2).

types changed; nice that they’re no longer global!

Well, now they no longer work seamlessly in VS Code (so all my linting is broken) and I'm super frustrated, but other than that...

it’s now result._emitter

Yeah, bunch of renaming to indicate what is private vs public.

title.function

This is a single scope that will be split into multiple classes when rendered as HTML. (this is still in flux) But if you're just talking with the emitter then you're dealing with just the scope name. It's not converted to actual classes until parse tree is converted to HTML at the very end.

Do you have an idea about when it’d be ready?

Could be a little while, I'm finding the window of being free to break things to allow for a lot of progress and innovation. I'd like to cut at least one more alpha before switching to betas... but if we have a few (very solid I hope) betas then I think people can run v11 as soon as they like just so long as they are prepared for a few bumps between betas.

[wooorm]

What? You just mean common? We try to keep that to our idea of what's most popular/useful. The total number of supported languages in core actually has gone up (by 2).

For lowlight I detect that based on hljs.registerLanguage calls in highlight.js/lib/index.js — that number went down (but it seems those weren’t “real” languages).

Well, now they no longer work seamlessly in VS Code (so all my linting is broken) and I'm super frustrated, but other than that...

Sounds like something is configured incorrectly? Now they work more like types in other projects from a user perspective.

---

Sounds good on the time. I’m rewriting some of the outer API of lowlight too, so would be nice to get both in at the same time (although: lowlight isn’t exposing hljs APIs, so a major for hljs doesn’t need to mean a major for ll)

[joshgoebel]

(but it seems those weren’t “real” languages).

Ah, indeed some stubs and older things got thrown out...

Sounds like something is configured incorrectly?

Well if you could help me fix it I'd be grateful. Before it just worked seamlessly. I've asked a bunch of people but so far it seems you can't have your cake and eat it too.

[wooorm]

I’m guessing it’s that you don’t import types at all! They used to be global state, which meant they could be accessed from everywhere.

I use TS through JSDoc. An import there looks like this: https://github.com/wooorm/lowlight/blob/6d531acc189a414c76525d5e9a3d809d798ffd69/lib/core.js#L4.
Here’s my commit adding types to lowlight

[joshgoebel]

Yeah importing types everywhere was something I was hoping to avoid. Plus a bunch of our types are private and not exported on purpose. How does one import private types that we never export?

[wooorm]

Similar to how you do it with code: not completely (until export maps), but put it in a file that’s not exported.

[joshgoebel]

/**
@typedef {import('highlight.js').Mode} Mode
@typedef {import('highlight.js').CompiledMode} CompiledMode
@typedef {import('highlight.js').Language} Language
@typedef {import('highlight.js').HLJSApi} HLJSApi
@typedef {import('highlight.js').HLJSPlugin} HLJSPlugin
@typedef {import('highlight.js').PluginEvent} PluginEvent
@typedef {import('highlight.js').HLJSOptions} HLJSOptions
@typedef {import('highlight.js').LanguageFn} LanguageFn
@typedef {import('highlight.js').HighlightedHTMLElement} HighlightedHTMLElement
@typedef {import('highlight.js').BeforeHighlightContext} BeforeHighlightContext
@typedef {import('highlight.js/private').MatchType} MatchType
@typedef {import('highlight.js/private').KeywordData} KeywordData
@typedef {import('highlight.js/private').EnhancedMatch} EnhancedMatch
@typedef {import('highlight.js/private').AnnotatedError} AnnotatedError
@typedef {import('highlight.js').AutoHighlightResult} AutoHighlightResult
@typedef {import('highlight.js').HighlightOptions} HighlightOptions
@typedef {import('highlight.js').HighlightResult} HighlightResult
*/

Any way to beautify that? It's a bit ugly/verbose.

[wooorm]

What’s the context? Where are you importing things? These types could be in the core lib instead maybe?
For more info, see https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html

[joshgoebel]

What’s the context?

Getting access to the types in my main source. My source is all JS, it exports no types... all the type exports are ambient modules.

Where are you importing things?

highlight.js

joshgoebel@5c65d22

These types could be in the core lib instead maybe?

I dunno what that means? You mean declare them in highlight.js itself and then export them, import them into index.d.ts and then reexport them? I'm not sure I know how to do that if that's possible - or advisable?

For more info, see

I've read all that in the past. If there is a specific section you were pointing me too, please advise.

[wooorm]

Getting access to the types in my main source. My source is all JS, it exports no types... all the type exports are ambient modules.

I’m proposing the inverse: define your types in your code.

I dunno what that means? You mean declare them in highlight.js itself and then export them, import them into index.d.ts and then reexport them? I'm not sure I know how to do that if that's possible - or advisable?

Indeed: write them in JavaScript. No need to add a manual index.d.ts: typescript can compile javascript and generate that index.d.ts.

Here’s a smaller example: words/levenshtein-edit-distance@2c3d890.
Lowlight is a bit more involved.
A bit more complex is unist-util-select: syntax-tree/unist-util-select@4c1b02e.
And quite complex is xdm: https://github.com/wooorm/xdm.

I am not a fan of typescript. I personally don’t really love types either. But they are somewhat useful — especially for newcomers to a giant ecosystem (unified).
I think types through jsdoc in javascript is rather nice: code is directly runnable, no building needed. But everything is still type checked by typescript (and I like type-coverage to ensure everything is completely typed).
It ensures .d.ts are kept up to date and work

Some more info here: https://github.com/voxpelli/types-in-js

[joshgoebel]

typescript can compile javascript and generate that index.d.ts.

Ah, I hadn't even considered that that because I'm not really interested in adding TS to our build pipeline (in my prior experience it's ridiculous slow). Something to consider for the future perhaps. I just got the ambient module stuff working and not interested in burning more time on this now.

I will save all those references for future reference though.

[joshgoebel]

I am not a fan of typescript.

I am, but not sure it's right for our codebase since it further limits who can contribute and I think we get many of the benefits with JS doctags alone.

I personally don’t really love types either.

I find they make me a lot more thoughtful/disciplined about the API of functions and shape of objects. This has been helpful.

[wooorm]

Having some problems as I’m no longer allowed to use extensions. https://github.com/xojs/xo is configured to enforce them. I quite like being explicit instead of magic, and would prefer to be allowed to use actual paths.

[joshgoebel]

At the back of my mind something is saying "This is the new standard, people will need to update their linters" , but I'm not sure where I'm getting that from. All the Node.js examples with conditional exports seem to show lack of extensions IIRC, so I guess perhaps that's at least one place that left me with that impression.

I do think what your asking is at odds with the whole IDEA of conditional exports though... you're not longer specifying REAL paths OR files at all, just identifiers (that the package routes however it chooses)... so to me the "spirit of the law" behavior seems to remove the extensions. No point pretending to be explicit if it isn't really. I mean it's actually explicit all the way down, just not the way you're used to...

@aduh95 Any thoughts?

---

Of course if we end up walking back conditional exports before release this will cease to matter as you'll still just be able to import anything you wish...

[wooorm]

This is the new standard, [...]

I’d say we used to do extensionless — now we do explicit extensions.
This is closer in line with how ESM works in the browser, or in deno.

[joshgoebel]

I’d say we used to do extensionless — now we do explicit extensions.

I can't keep track because I feel it's changed a few times, I think TS had different ideas than JS for a while, and now conditional exports seems to be hinting at extension-less behavior I thought... I'm curious to see how other packages approach this.

The browser can only deal with files of course (without forcing the resolution behavior back onto the web server itself) so I'd say it was "stuck" without much of a choice...

If found necessary I think the extension based paths could just be added to the exports table beside the path based, could you confirm?

[wooorm]

Also: for xo and Sindre and me: this is folks who used to do extensionless, and are now switching to extensions.

I also don’t like export maps much. But it indeed seems that that prefers no extensions. So I don’t know 🤷‍♂️

[joshgoebel]

I also don’t like export maps much.

I dunno what to think yet... the promise is that ESM and CJS "just work" and no one has to think about highlight.js/cjs vs highlight.js/esm, which seems a nice idea?... whether that's worth all the magic or not... I dunno. :-) Seems a lot of packagers early on had no idea what to make of all this new stuff... which is why it's still a little up in the air as to whether v11 with have conditional exports or explicit paths (the es folder).

[joshgoebel]

Oh and preventing people from reaching into pieces of your library that are intended to be private is a nice benefit - though it doesn't really apply to us at the moment since we have a build process and if we don't want to expose it we just don't build it separately.

[aduh95]

I do think what your asking is at odds with the whole IDEA of conditional exports though... you're not longer specifying REAL paths or file at all, just identifiers... so to me the "spirit of the law" behavior seems to remove the extensions. No point pretending to be explicit if it isn't really.

Yes, exactly, the goal of conditional exports is to hide where the files happen to be in the npm archive, as it's considered an implementation detail. The idea is that hightlight.js/ada could move from ./lib/commonjs/ada.js to ./lib/wasm/ada.wasm in a semver-minor without the users having to care about what's actually inside the package.
It has also been introduced to Node.js with import maps in mind, which should work in the browser (still behind a flag in Chrome today).

[wooorm]

I guess most of your users are using generated bundles though, where it doesn’t matter.

I’d suggest to push for folks who are depending on the files to upgrade to ESM. Dual publishing is complex! ESM is closer to having files directly usable in browsers too.

the promise is that ESM and CJS "just work"

And I think that’s the problem. That promise seems to fail all the time 😬

[joshgoebel]

where it doesn’t matter.

What is the "it" this is referring to?

I’d suggest to push for folks who are depending on the files to upgrade to ESM.

Huh, I'm not following? Which files and what do you mean "upgrade to ESM"?

That promise seems to fail all the time

Well the v11 release isn't necessarily going to happen until post Node v10 sunsetting, so there is still time for the situation to improve.

Dual publishing is complex!

Indeed, but we're not shipping two separate packages and I'm not a fan of this fervor to immediately "switch" the whole ecosystem to ESM, so that means we either ignore ESM for another release cycle or ship dual... and shipping dual seemed the better of those two choices to me.

[wooorm]

What is the "it" this is referring to?

“It” here refers to module format (ESM vs CJS vs dual) — which doesn’t matter to bundle/CDN users

Huh, I'm not following? Which files and what do you mean "upgrade to ESM"?

I suggest that package maintainers start switching to ESM only.

Well the v11 release isn't necessarily going to happen until post Node v10 sunsetting

That’s in 7 days already!

[joshgoebel]

I suggest that package maintainers start switching to ESM only.

Most of the reading I've seen on this is based on "just keep using our prior version if you still need CJS, easy!" which is NOT a philosophy we want to embrace. We do not want tons of people on v10 forever anymore than we wanted tons of people on v9 forever after v10 was released. The best experience is always on the latest and we don't want to give people reasons NOT to upgrade.

which doesn’t matter to bundle/CDN users

I agree for a whole lot of people just happily using our CDN module that this is an entire non-issue. And our browser/CDN builds aren't changing.

That’s in 7 days already!

Oh, LOL..... well then it'll be after Node v10 I guess... I'm really looking for some actual feedback with bundlers (or something) and such things where people literally can't use the library because something is broken broken. So far all we have is your wishing the paths could have extensions. I assume if you removed the extensions it worked fine (though you haven't said).

Also, the ~ official support window for Highlight.js v10 is ~ 3 months or so... so if a bundler doesn't support conditional exports yet but plans to soon that's also OK I think.

[wooorm]

which is NOT a philosophy we want to embrace

I’m not suggesting users use CJS. I’m suggesting they use ESM

I'm really looking for some actual feedback with bundlers (or something) and such things where people literally can't use the library because something is broken broken

Some more info here: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c.
Jest, webpack@4 (CRA), Electron, in some cases Next, are mostly where the trouble is. But they can either be configured to work, or plan on landing this when Node@10 is EOL (so in ±6 days).

I don’t know about conditional exports support in bundlers, as I don‘t use that myself.

I assume if you removed the extensions it worked fine (though you haven't said).

Yup, it works

[joshgoebel]

I’m not suggesting users use CJS. I’m suggesting they use ESM

Right, but ESM only is a reason for someone not to upgrade if their codebase is "stuck" on CJS. I don't want to do anything to encourage "well we'll just keep using v10, it works".

[wooorm]

ESM only is a reason for someone not to upgrade if their codebase is "stuck" on CJS

Definitely.
On the other hand, it’s going to happen anyway: some maintainers will use ESM. Especially for new projects. So users will notice it at some point in time.
I think there’s something to say for collective action from maintainers. That way it won’t be only one package that’s a version behind, but more and more. It’s not going to be the most of fun for users, but slowly moving over 2 years isn’t fun either, with collective action at least it’s swift.

highlight.js is relatively “high-level” project, with many beginners, compared to most of Sindre’s or my stuff. And it’s relatively unique at that for having no dependencies.

Generally I think these changes “bubble upwards” through the ecosystem, from low-level one-liners, through more user-facing projects, to users.

I think you have some options and that choosing to not be one of the first to switch is a fine strategy. On the other hand, most of your users won’t notice and the folks that do depend on it through npm are (presumably) aware of the CJS -> ESM move, and it might cause them to switch faster!

[joshgoebel]

Yeah, I just wish I had more stats. I just feel like v11 already has enough breaking changes and we already "force" our users to do enough (stay up-to-date with releases, including hopefully major ones)... all of this ultimately to serve their interests though - since we can only maintain a single version - the benefit they receive is new features, better grammars, bug fixes, etc... so there is a [hopefully] clear benefit to offset any annoyance.

At this point I just can't see the direct payoff for forcing users to upgrade to ESM if they don't wish to. Our codebase is ESM already. The final format of our build assets (ESM or CJS) honestly isn't very interesting to me. Now we'll just have 3 output formats instead of 2 and users can use whichever works for their usage. If that proves to be some sort of maintenance headache we'll see, but I'm not imagining it will be.