Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support --require of ESM; closes #4281 #4304

Merged
merged 7 commits into from Jun 1, 2020
Merged

Conversation

JacobLey
Copy link
Contributor

@JacobLey JacobLey commented May 27, 2020

Allow files/modules specified in --require to be ESM.

CommonJS loading is still supported and the default.

closes #4281

Description of the Change

Support --requireing ESM files in config. Current behavior is to fail on ERR_REQUIRE_ESM.

Re-use existing "require or import" logic for loading test files to load required files.

Alternate Designs

Pipe requiring the ESM file through something like esm or ts-node. Although that seems much more complicated and overkill.

Why should this be in core?

Requiring JS files before the test suite is a core part of Mocha's functionality. This extends that support to ESM-first codebases, which is natively supported via "type":"module", or --exerimental-module.

Benefits

Allows developers more freedom in choosing to write ESM or CommonJS.

Possible Drawbacks

Native ESM in NodeJS is still labeled as experimental (although an --experimental flag is no longer needed). It is possible the usage will change, but that is justification for re-using existing "require or import" functionality.

Applicable issues

This should be an "enhancement" (minor release). --requireing CommonJS is still the default and fully supported, this will just allow additional support for ESM.

Allow files/modules specified in `--require` to be ESM.

CommonJS loading is still supported and the default.
@jsf-clabot
Copy link

jsf-clabot commented May 27, 2020

CLA assistant check
All committers have signed the CLA.

@JacobLey
Copy link
Contributor Author

No tests included in this first pass.

I don't see any existing tests for this... so not exactly sure where to put them.

@@ -15,7 +12,7 @@ const requireOrImport = async file => {
return require(file);
} catch (err) {
if (err.code === 'ERR_REQUIRE_ESM') {
return import(url.pathToFileURL(file));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what url.pathToFileURL was achieving?
import() allows a plain string, and this way the required file can still be resolvable via node's logic (e.g. loading from node_modules)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it was required for Windows.
Gosh I hate windows

const path = require('path');

const requireOrImport = async file => {
file = path.resolve(file);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved resolve() to loadFilesAsync so the file doesn't have to be a fully-resolved name, but can be a package from node_modules as well.

@coveralls
Copy link

coveralls commented May 27, 2020

Coverage Status

Coverage increased (+0.01%) to 93.445% when pulling c411c30 on JacobLey:requireESM into 6d60eb0 on mochajs:master.

Copy link
Member

@boneskull boneskull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Great work. Can you please add an integration test in test/integration/options/require.spec.js which loads an ESM module fixture via --require?

@boneskull boneskull added type: feature enhancement proposal semver-minor implementation requires increase of "minor" version number; "features" labels May 28, 2020
@boneskull
Copy link
Member

apologies, the organization of test is often not obvious. probably need to add more info to test/README.md

As both .mjs and type=module (combined with cjs for good measure).

Updated linter to allow tests to use spread operator (ecmaVersion 2018)

Allow --require'd module to be an object, or "module"
],
{
env: {
...process.env,
Copy link
Contributor Author

@JacobLey JacobLey May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I updated the parser options (parserOptions: 2018) in eslintrc for.

Considering mocha is documented with:

As of v7.0.0, Mocha requires Node.js v8.0.0 or newer.

This seems safe enough for tests at least.

If there is an easier way to pass --experimental-modules to the mocha (instead of calling node directly) happy to use that instead...

Copy link
Member

@boneskull boneskull May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I apologize for the state of things (we're working on it; see #4293), but...

I wanted to get a subset of the integration tests (test/integration/) running in a browser, but haven't completed this work. Stuff in test/node-unit/ (and a few other places) is allowed to use ES2018 syntax, but basically everything else under test/ is limited to ES5. It's probably OK to allow ES2018 in test/integration/, if we expect #4293 to get merged in the near future, since we'll be able to transpile everything down to where it needs to be. I don't envision browser-based integration tests landing before then, so IMO it's fine. Others may disagree (@mochajs/core ?)

However, stuff in test/unit/ and test/browser cannot, at this time, be ES2018, because these tests run in IE11. So, please revert the change to .eslintrc.yml, then add 'test/integration/**/*.js' to the list of overrides which already use parserOptions: 2018 (L39 or so).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, you're running bin/mocha (which is what invokeMocha does), but using the NODE_OPTIONS environment variable to set --experimental-modules. You don't need to do this; just pass --experimental-modules in your array of args. bin/mocha knows what to do with it.

e.g.

invokeMocha(['path/to/fixture', '--some-other-flag'].concat(+process.versions.node.split('.')[0] >= 13 ? [] : ['--experimental-modules'], function() {})

I realize this means you can just revert the change to .eslintrc.yml entirely, which is maybe what should happen.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted change to eslintrc

Didn't realize I could pass the flag directly though mocha! Since I don't need any lint changes, I don't think this is the place to reorganize what is allowed where.

@@ -92,7 +92,10 @@ exports.handleRequires = async (requires = []) => {
debug('resolved required file %s to %s', mod, modpath);
}
const requiredModule = await requireOrImport(modpath);
if (type(requiredModule) === 'object' && requiredModule.mochaHooks) {
if (
['object', 'module'].includes(type(requiredModule)) &&
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node -e "import('./test/integration/fixtures/options/require/root-hook-defs-esm.fixture.mjs').then(mod => console.log(Object.prototype.toString.call(mod)))"
-> [object Module]

Piped through the type function: https://github.com/mochajs/mocha/blob/master/lib/utils.js#L216
Returns module.

So added handling for that "module" case (which seems reasonable considering that is what we would expect from ESM... but for the sake of discussing alternatives, could add some handling to type() to map module -> object

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, this seems correct. but, we should also add a test in test/node-unit/utils.spec.js to cover the behavior. you don't need to do this unless you want to.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can change it to return object if there are unforeseen consequences. what does typeof requiredModule return?

Copy link
Contributor Author

@JacobLey JacobLey May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node -e "import('./test/integration/fixtures/options/require/root-hook-defs-esm.fixture.mjs').then(mod => console.log(typeof mod))" -> object

I'll let you interpret that... since there must be some reason not to use typeof directly.

add a test to cover the behavior

I can add that as well, so long as we are fine with Modules returning "module"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's no real great reason to use type() here other than trying to maintain some consistency. we should use typeof and not worry about it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ref: #4306

@JacobLey JacobLey requested a review from boneskull May 28, 2020 20:38
Copy link
Member

@boneskull boneskull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls see comments

@@ -0,0 +1,8 @@
export const mochaHooks = () => ({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome

],
{
env: {
...process.env,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, you're running bin/mocha (which is what invokeMocha does), but using the NODE_OPTIONS environment variable to set --experimental-modules. You don't need to do this; just pass --experimental-modules in your array of args. bin/mocha knows what to do with it.

e.g.

invokeMocha(['path/to/fixture', '--some-other-flag'].concat(+process.versions.node.split('.')[0] >= 13 ? [] : ['--experimental-modules'], function() {})

I realize this means you can just revert the change to .eslintrc.yml entirely, which is maybe what should happen.

Add truthy check to handle null edge case

type(ES Module) => "module", but we treat it the same as an object
@boneskull boneskull requested a review from giltayar May 28, 2020 21:59
Copy link
Member

@boneskull boneskull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks. Would like @giltayar to take a gander, if possible

Copy link
Contributor

@giltayar giltayar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@@ -25,7 +30,7 @@ const requireOrImport = async file => {
exports.loadFilesAsync = async (files, preLoadFunc, postLoadFunc) => {
for (const file of files) {
preLoadFunc(file);
const result = await requireOrImport(file);
const result = await exports.requireOrImport(path.resolve(file));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the file in question is a package name, path.resolve will fail (AFAIK with an exception). Why did you add a path.resolve here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see. --require-d files can be module names, and so you didn't want requireOrImport, which is now used also in handleRequire, to resolve them to a path. So you moved it to loadFileAsync, where it is needed (for Windows, it seems...)

Copy link
Member

@juergba juergba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JacobLey please also adapt the esm limitation section in our docs. Thank you.

@JacobLey JacobLey requested a review from juergba May 29, 2020 14:43
@giltayar
Copy link
Contributor

Because "require" is not typically a word used with ESM ("import" is more generally used), I would add specific messaging in the --require documentation that says that it also accepts ES modules.

Copy link
Member

@juergba juergba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just have two questions about resolving the relative path / absolute path / package name (bare specifier). I'm unsure about this.

The docs states:

Bare specifiers, and the bare specifier portion of deep import specifiers, are strings; but everything else in a specifier is a URL.

So the relative path should be an URL as well?

invokeMochaAsync(
[
'--require=' +
require.resolve(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--requireworks with absolute path and module names. Does it work also with relative paths?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, see where you commented on handleRequires, relative paths are transformed into absolute paths.

None of the tests use it though because the actual CWD isn't very clear in-test. But maybe it is possible to calculate a relative path in test and pass that through as well.

Again, that particular functionality is unchanged in this PR

requires.reduce((acc, mod) => {
exports.handleRequires = async (requires = []) => {
const acc = [];
for (const mod of requires) {
let modpath = mod;
// this is relative to cwd
if (fs.existsSync(mod) || fs.existsSync(`${mod}.js`)) {
modpath = path.resolve(mod);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the file in question is a package name, path.resolve will fail [...]

Will this work for packages?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should, fs.existsSync will return false for a package name, so it is not resolved at all (passed raw) to requireOrImport (e.g. requireOrImport('@babel/register')

This behavior is unchanged in this PR

@@ -1007,6 +1007,8 @@ Modules required in this manner are expected to do work synchronously; Mocha won

Note you cannot use `--require` to set a global `beforeEach()` hook, for example — use `--file` instead, which allows you to specify an explicit order in which test files are loaded.

> As of v7.3.0, Mocha supports `--require` for [NodeJS native ESM](#nodejs-native-esm-support). There is no separate `--import` flag.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that the version here may need changing

@boneskull
Copy link
Member

LGTM. Thanks much @JacobLey !

@boneskull boneskull merged commit 273dbbb into mochajs:master Jun 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver-minor implementation requires increase of "minor" version number; "features" type: feature enhancement proposal
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parameter --require cannot load an ES module.
6 participants