Migration to ES Modules

[Josh-Cena]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Motivation

I did a very naïve attempt to migrate to ESM in #5816, but I far underestimated the difficulty in pushing it through. After some pondering, I think this should be rolled out progressively.

Status of ESM

ESM is a new type of Node modules system, replacing the old common JS system (require + module.exports). For the engine, the parsing goal is different ("module" or "script"), so a file needs to be determined as ESM or CJS before executing it, either through the .mjs extension or through the "type": "module" entry in the nearest package.json.

If a file is ESM, it can import from other ES modules. However, it may not be able to import all CJS modules, depending on how the CJS module is structured, because exported symbols, per the ES spec, need to be lexically determined, while CJS can be far more dynamic.

If a file is CJS, it can't import ESM modules, unless the await import() dynamic import is used. However, this is against the norm of how most modules are imported. This effectively means as soon as a considerable number of the dependencies are ESM, we have to migrate ourselves.

Many packages on NPM are now ESM-only, most notably the packages by Sindre Sorhus, and MDX v2, which is the pillarpost of our architecture.

Benefits

1. Unlock future dependency upgrades. Many popular libraries are seeking to upgrade to ESM; if we can be ESM, we can interoperate with them. For example, MDX v2, chalk...
2. A similar transpilation target for client and server code; no need for multiple tsconfigs
3. Permit top-level awaits

Blockers

1. TypeScript has very lame support of ESM transpilation so far. It seems their deferred ESM support still won't land in 4.6, so in the meantime we may have to run all our Node code with --experimental-specifier-resolution=node and keep the old resolution
2. Jest doesn't seem to like importing ESM dependencies
3. We use import-fresh to bypass cache and always import fresh modules for hot reloading, etc. But ES Modules don't expose caching manipulation yet
4. Community dividing: although the ESM migration will surely be a major version, it means in the foreseeable future after that, some plugins will not be compatible with the new version (especially those that import utils and logger)

Actions needed

1. Removing __filename and __dirname. These globals don't exist in the ESM scope, replaced by the import.meta.url
2. Setting target: 'nodenext' in the tsconfig.
3. Changing our import paths. ESM requires the index.js name and the .js extension to be explicit.
4. Setting "type": "module" in our package.json.
5. Tweaking the configuration for related tools?

Plan

1. Config files: JS config files like docusaurus.config.js and sidebars.js can be allowed in ESM once we figure out how to bypass cache and fresh-import ESM
2. Utils: this includes utils, utils-validation, logger. They should always be distributed as dual-package because plugin authors are likely to import them as CJS.
3. Core: the biggest blocker is still the extensive use of import-fresh. Migration to ESM means we can import both CJS and ESM plugin modules with await import. However, because users only interact with the core through its own CLI, we don't have to care about others importing this.
4. Plugins: migration of plugins can only happen after migration of core (or at least solving import-fresh in the core), because they have to be imported as ESM.

Related issues/PRs

If an issue or PR is related to the ESM migration process, please link to this meta-issue and we will add it to the list below for tracking purposes.

• Support ESM config file #5379
• misc: migrate all packages to ESM-only #5816
• misc: convert all internal scripts to ESM #6286
• refactor: mark all functions that import external modules as async #6521
• refactor: convert CLI entry points to ESM; migrate create-docusaurus to ESM #6661
• refactor: ensure lodash is default-imported #6716
• refactor: import jest as global; unify import style of some modules #6898
• refactor(theme-common): mark package as ESM #6899
• feat(core): allow plugin lifecycles to return relative paths #6921
• feat(core): support docusaurus.config.cjs as default file name #7371
• fix(core): make generated files have unambiguous module types #7379

[iPurpl3x]

Yes !!!! +1 !
Finally I stumbled on this issue, I had updated @mdx-js/react and had errors and it took me a while to understand what happened...

Error was something like

_mdx_js_react__WEBPACK_IMPORTED_MODULE_2__.mdx is not a function

and

export 'mdx' (imported as 'mdx') was not found in '@mdx-js/react' (possible exports: MDXContext, MDXProvider, useMDXComponents, withMDXComponents)

[Josh-Cena]

@iPurpl3x Your problem is probably not because of ESM, because @mdx-js/react is used on client-side, where Webpack can take care of ESM syntax. Instead, it's because of the API changes in v2. MDX will be compiled to a JSX file containing a line import { mdx } from '@mdx-js/react'; which doesn't exist in the latest MDX version.

[Josh-Cena]

I ran yarn outdated and below are all dependencies that are ESM and we can't upgrade:

Outdated dependencies

Package                          Current Wanted Latest
@mdx-js/mdx                      1.6.22  1.6.22 2.0.0
@mdx-js/react                    1.6.22  1.6.22 2.0.0
boxen                            5.1.2   5.1.2  6.2.1
chalk                            4.1.2   4.1.2  5.0.0
escape-string-regexp             4.0.0   4.0.0  5.0.0
globby                           11.1.0  11.1.0 13.1.1
hast-util-to-string              1.0.4   1.0.4  2.0.0
is-root                          2.1.0   2.1.0  3.0.0
leven                            3.1.0   3.1.0  4.0.0
mdast-util-to-string             2.0.0   2.0.0  3.1.0
rehype-parse                     7.0.1   7.0.1  8.0.4
remark                           12.0.1  12.0.1 14.0.2
remark-emoji                     2.2.0   2.2.0  3.0.2
remark-math                      3.0.1   3.0.1  5.1.1
remark-mdx                       1.6.22  1.6.22 2.0.0
remark-parse                     8.0.3   8.0.3  10.0.1
remark-stringify                 8.1.1   8.1.1  10.0.2
stringify-object                 3.3.0   3.3.0  4.0.1
to-vfile                         6.1.0   6.1.0  7.2.3
unified                          9.2.2   9.2.2  10.1.1
unist-builder                    2.0.3   2.0.3  3.0.0
unist-util-remove-position       3.0.0   3.0.0  4.0.1
unist-util-visit                 2.0.3   2.0.3  4.1.0

Mostly Sindre Sorhus packages and MDX ones

[sachaw]

This is just going to get worse with time, @Josh-Cena Thanks for the great work, hopefully we can get Docusaurus using ESM soon.

[sambacha]

sindre is working on converting his packages to pure ESM, see https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

also, the latest @docusaurus/migrate package does not work, the index.mjs is re-written as index.js causing it to fail. Renaming the file extension fixes it

[slorber]

also, the latest @docusaurus/migrate package does not work, the index.mjs is re-written as index.js causing it to fail. Renaming the file extension fixes it

What do you mean by "does not work"? How do you run it and how do you see it not working exactly? It seems to work for me

[Josh-Cena]

@slorber It was a mistake made in the last publish😅 The file is called bin/index.mjs but in package.json it's referred to as bin/index.js. You probably need to run it outside our workspace. See #6897

[Josh-Cena]

TS 4.7 seems promising: https://devblogs.microsoft.com/typescript/announcing-typescript-4-7-beta/#ecmascript-module-support-in-node-js We should be able to investigate after that

[slorber]

Yes, also great to have "exports" field support!

[mrienstra]

yarnpkg/berry#3843 seems to have been fixed yesterday!

Nice! Looks like that change is available in 4.0.0-rc.22. Run yarn set version canary if you want to try it. v4 release is still a little ways off ("may take a couple more months"), but:

[...] what's in master is stable, and I'd recommend you to try it. The only notable difference with stable is that we reserve the right to land a couple more breaking changes in future RCs, but in terms of stability it's almost always better to use RCs than stable.

... according to yarnpkg/berry#4895

Edit: See yarnpkg/berry#3591 for v4 breaking changes. I've seen two mentions of people stumbling over enableGlobalCache default changing from false to true. (Google search: site:yarnpkg.com "enableGlobalCache")

[mrienstra]

Update: the fix that adds support for the package.json includes field is also in the Yarn 3.2.4 release.

[samos123]

I hit an error after migrating to ES modules due to generation of client-modules.js which uses require:

docusaurus/packages/docusaurus/src/server/index.ts

Line 182 in 7dcad0c

.map((clientModule) => ` require("${escapePath(clientModule)}"),`)

The V3 announcement made me think I should be switching to modules. I might be doing something silly, not a JS expert.

[slorber]

@samos123 this issue is about running ES modules natively inside the Node.js runtime (or publishing ESM package), not about public-facing features supporting ESM.

We already support ESM syntax in client modules (our own site use it for a while if you look for an example), and v3 brings ESM support for site config, sidebars etc...

If you encounter a bug, please open a dedicated issue. "I hit an error" without any extra information is not going to help us in any way help you.

[Zamiell]

homotechsual from the Discord server told me to post this bug report here, sorry if this is off-topic.

Docusaurus does not seem to support the "type" field in the "package.json" file. Subsequently, it seems impossible to use ESM with Docusaurus. (To be clear, you can have an ESM config file, but the project itself can obviously not be ESM.)

Here are steps to showcase the problem:

# Create a new Docusaurus website. This line is copy-pasted from the documentation here:
# https://docusaurus.io/docs/typescript-support
npx create-docusaurus@latest my-website classic --typescript

# Go into the website.
cd my-website

# Build the website and watch it succeed.
npm run build

# Edit the "package.json" file and add: `"type": "module",`
vim package.json

# Build the website and watch it fail.
npm run build

# Edit the "package.json" file and change "module" to "commonjs".
vim package.json

# Build the website and watch it fail again.
npm run build

This seems very unexpected, as it was my understanding that the type field defaults to "commonjs", so omitting it entirely should be equivalent to putting "type": "commonjs", but I guess that isn't the case here.

1. Should Docusaurus be fixed such that "type": "commonjs" is equivalent to omitting it entirely?
2. Is it intended that ESM is not supported in the latest version of Docusaurus?
3. Is this issue about updating the Docusaurus monorepo itself to ESM, or is it about end-user projects using ESM, or both?

[Josh-Cena]

@Zamiell type: commonjs is not equivalent for Webpack. When type is absent, Webpack allows both ESM and CJS syntax in the same file; when the module type is unambiguous, it only allows one.