Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support WebAssembly (Wasm) imports in ESM modules #13505

Merged
merged 27 commits into from Nov 6, 2022
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
95ef976
Add failing test
kachkaev Oct 24, 2022
645db88
Update `jest-runtime` to fix tests
kachkaev Oct 24, 2022
9985179
Support data:application/wasm imports
kachkaev Oct 27, 2022
5d73f9a
Run tests twice (with and without --experimental-wasm-modules)
kachkaev Oct 27, 2022
bc6d38c
Merge remote-tracking branch 'u/main' into native-esm-wasm
kachkaev Oct 27, 2022
734bbc5
Cover more rows
kachkaev Oct 27, 2022
612d480
Add CHANGELOG entry
kachkaev Oct 27, 2022
dcff11e
Simplify tests
kachkaev Oct 27, 2022
e524e40
Improve `path.endsWith`
kachkaev Oct 27, 2022
08c8906
Delete unused file
kachkaev Oct 27, 2022
66c8312
Make changelog phrase more explicit
kachkaev Oct 27, 2022
6586a24
Fix Wasm casing
kachkaev Oct 27, 2022
01f8d78
Update CHANGELOG.md
kachkaev Oct 30, 2022
27d73b6
Mention Wasm in docs
kachkaev Oct 30, 2022
2aea674
Merge remote-tracking branch 'origin/main' into native-esm-wasm
kachkaev Oct 30, 2022
79e0f40
Use Wasm file from mdn examples
kachkaev Oct 30, 2022
0d4945e
Implement `readFileBuffer`
kachkaev Oct 30, 2022
26db3fd
Call loadEsmModule instead of linkAndEvaluateModule in _importWasmModule
kachkaev Oct 30, 2022
1e6d75b
Update e2e/native-esm/__tests__/native-esm-wasm.test.js
kachkaev Oct 31, 2022
d4e9b73
Merge remote-tracking branch 'origin/main' into native-esm-wasm
kachkaev Nov 1, 2022
ad32edd
Add another test case for dynamic import
kachkaev Nov 1, 2022
397bcd1
Extract `isWasm` function
kachkaev Nov 1, 2022
976ef61
Merge remote-tracking branch 'origin/main' into native-esm-wasm
kachkaev Nov 6, 2022
66a2312
Revert "Implement `readFileBuffer`"
SimenB Nov 6, 2022
7188086
separate buffer cache
SimenB Nov 6, 2022
26e7022
tweak
SimenB Nov 6, 2022
607fc0f
clear new cache as well
SimenB Nov 6, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Expand Up @@ -2,6 +2,8 @@

### Features

- `[jest-runtime]` Support WASM imports ([#13505](https://github.com/facebook/jest/pull/13505))

### Fixes

- `[jest-mock]` Treat cjs modules as objects so they can be mocked ([#13513](https://github.com/facebook/jest/pull/13513))
Expand Down
10 changes: 9 additions & 1 deletion e2e/__tests__/__snapshots__/nativeEsm.test.ts.snap
Expand Up @@ -8,9 +8,17 @@ Time: <<REPLACED>>
Ran all test suites matching /native-esm-deep-cjs-reexport.test.js/i."
`;

exports[`runs WASM test with native ESM 1`] = `
"Test Suites: 1 passed, 1 total
Tests: 4 passed, 4 total
Snapshots: 0 total
Time: <<REPLACED>>
Ran all test suites matching /native-esm-wasm.test.js/i."
`;

exports[`runs test with native ESM 1`] = `
"Test Suites: 1 passed, 1 total
Tests: 34 passed, 34 total
Tests: 33 passed, 33 total
Snapshots: 0 total
Time: <<REPLACED>>
Ran all test suites matching /native-esm.test.js/i."
Expand Down
12 changes: 12 additions & 0 deletions e2e/__tests__/nativeEsm.test.ts
Expand Up @@ -67,3 +67,15 @@ onNodeVersions('>=16.9.0', () => {
expect(exitCode).toBe(0);
});
});

test('runs WASM test with native ESM', () => {
const {exitCode, stderr, stdout} = runJest(DIR, ['native-esm-wasm.test.js'], {
nodeOptions: '--experimental-vm-modules --no-warnings',
});

const {summary} = extractSummary(stderr);

expect(summary).toMatchSnapshot();
expect(stdout).toBe('');
expect(exitCode).toBe(0);
});
8 changes: 8 additions & 0 deletions e2e/native-esm-wasm/package.json
@@ -0,0 +1,8 @@
{
"type": "module",
"name": "native-esm-wasm",
"jest": {
"testEnvironment": "node",
"transform": {}
}
}
Binary file added e2e/native-esm/42.wasm
Binary file not shown.
37 changes: 37 additions & 0 deletions e2e/native-esm/__tests__/native-esm-wasm.test.js
@@ -0,0 +1,37 @@
/**
* Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/

// the point here is that it's the node core module
// eslint-disable-next-line no-restricted-imports
import {readFileSync} from 'fs';
// The file was generated by wasm-pack
import {getAnswer} from '../42.wasm';
Copy link
Contributor Author

@kachkaev kachkaev Oct 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File source: https://github.com/hasharchives/wasm-ts-esm-in-node-jest-and-nextjs

Happy to replace this wasm with some canonical example file, but I don’t know any. Ideally, this file would export a function with arguments (mine is just getAnswer = () => 42).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that seems reasonable 👍

Copy link
Contributor Author

@kachkaev kachkaev Oct 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 79e0f40, using add(i32, i32) from mdn/webassembly-examples.


const wasmFileBuffer = readFileSync('42.wasm');

test('supports native wasm imports', () => {
expect(getAnswer()).toBe(42);
});

test('supports imports from "data:application/wasm" URI with base64 encoding', async () => {
const importedWasmModule = await import(
`data:application/wasm;base64,${wasmFileBuffer.toString('base64')}`
);
expect(importedWasmModule.getAnswer()).toBe(42);
});

test('imports from "data:text/wasm" URI without explicit encoding fail', async () => {
await expect(() =>
import(`data:application/wasm,${wasmFileBuffer.toString('base64')}`),
).rejects.toThrow('Missing data URI encoding');
});

test('imports from "data:text/wasm" URI with invalid encoding fail', async () => {
await expect(() =>
import('data:application/wasm;charset=utf-8,oops'),
).rejects.toThrow('Invalid data URI encoding: charset=utf-8');
});
6 changes: 0 additions & 6 deletions e2e/native-esm/__tests__/native-esm.test.js
Expand Up @@ -255,12 +255,6 @@ test('imports from "data:text/javascript" URI with invalid data fail', async ()
).rejects.toThrow("Unexpected token '.'");
});

test('imports from "data:application/wasm" URI not supported', async () => {
await expect(() =>
import('data:application/wasm,96cafe00babe'),
).rejects.toThrow('WASM is currently not supported');
});

test('supports imports from "data:application/json" URI', async () => {
const data = await import('data:application/json,{"foo": "bar"}');
expect(data.default).toEqual({foo: 'bar'});
Expand Down
167 changes: 119 additions & 48 deletions packages/jest-runtime/src/index.ts
Expand Up @@ -398,9 +398,12 @@ export default class Runtime {

// unstable as it should be replaced by https://github.com/nodejs/modules/issues/393, and we don't want people to use it
unstable_shouldLoadAsEsm(path: string): boolean {
return Resolver.unstable_shouldLoadAsEsm(
path,
this._config.extensionsToTreatAsEsm,
return (
path.endsWith('wasm') ||
Resolver.unstable_shouldLoadAsEsm(
path,
this._config.extensionsToTreatAsEsm,
)
);
}

Expand Down Expand Up @@ -441,6 +444,19 @@ export default class Runtime {
'Promise initialization should be sync - please report this bug to Jest!',
);

if (modulePath.endsWith('wasm')) {
const wasm = this._importWasmModule(
fs.readFileSync(modulePath),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have a readFile helper which caches the result. we should do the same here. readFile currently reads as utf8, so maybe just a separate readFileBuffer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0d4945e. I had to change the type for _cacheFS, which unfortunately spills outside jest-runtme. Happy to improve the implementation if you have any suggestions.

modulePath,
context,
);

this._esmoduleRegistry.set(cacheKey, wasm);

transformResolve();
return wasm;
}

if (this._resolver.isCoreModule(modulePath)) {
const core = this._importCoreModule(modulePath, context);
this._esmoduleRegistry.set(cacheKey, core);
Expand Down Expand Up @@ -567,56 +583,67 @@ export default class Runtime {
}

const mime = match.groups.mime;
if (mime === 'application/wasm') {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite a few lines below are marked as changed because of extra indent. Hiding whitespace in diff viewer makes it a bit easier to see the real difference.

throw new Error('WASM is currently not supported');
}

const encoding = match.groups.encoding;
let code = match.groups.code;
if (!encoding || encoding === 'charset=utf-8') {
code = decodeURIComponent(code);
} else if (encoding === 'base64') {
code = Buffer.from(code, 'base64').toString();
} else {
throw new Error(`Invalid data URI encoding: ${encoding}`);
}

let module;
if (mime === 'application/json') {
module = new SyntheticModule(
['default'],
function () {
const obj = JSON.parse(code);
// @ts-expect-error: TS doesn't know what `this` is
this.setExport('default', obj);
},
{context, identifier: specifier},

if (mime === 'application/wasm') {
if (!encoding) {
throw new Error('Missing data URI encoding');
}
if (encoding !== 'base64') {
throw new Error(`Invalid data URI encoding: ${encoding}`);
}
module = await this._importWasmModule(
Buffer.from(match.groups.code, 'base64'),
specifier,
context,
);
} else {
module = new SourceTextModule(code, {
context,
identifier: specifier,
importModuleDynamically: async (
specifier: string,
referencingModule: VMModule,
) => {
invariant(
runtimeSupportsVmModules,
'You need to run with a version of node that supports ES Modules in the VM API. See https://jestjs.io/docs/ecmascript-modules',
);
const module = await this.resolveModule(
specifier,
referencingModule.identifier,
referencingModule.context,
);
let code = match.groups.code;
if (!encoding || encoding === 'charset=utf-8') {
code = decodeURIComponent(code);
} else if (encoding === 'base64') {
code = Buffer.from(code, 'base64').toString();
} else {
throw new Error(`Invalid data URI encoding: ${encoding}`);
}

return this.linkAndEvaluateModule(module);
},
initializeImportMeta(meta: ImportMeta) {
// no `jest` here as it's not loaded in a file
meta.url = specifier;
},
});
if (mime === 'application/json') {
module = new SyntheticModule(
['default'],
function () {
const obj = JSON.parse(code);
// @ts-expect-error: TS doesn't know what `this` is
this.setExport('default', obj);
},
{context, identifier: specifier},
);
} else {
module = new SourceTextModule(code, {
context,
identifier: specifier,
importModuleDynamically: async (
specifier: string,
referencingModule: VMModule,
) => {
invariant(
runtimeSupportsVmModules,
'You need to run with a version of node that supports ES Modules in the VM API. See https://jestjs.io/docs/ecmascript-modules',
);
const module = await this.resolveModule(
specifier,
referencingModule.identifier,
referencingModule.context,
);

return this.linkAndEvaluateModule(module);
},
initializeImportMeta(meta: ImportMeta) {
// no `jest` here as it's not loaded in a file
meta.url = specifier;
},
});
}
}

this._esmoduleRegistry.set(specifier, module);
Expand Down Expand Up @@ -1640,6 +1667,50 @@ export default class Runtime {
return evaluateSyntheticModule(module);
}

private async _importWasmModule(
source: Buffer,
identifier: string,
context: VMContext,
) {
const wasmModule = await WebAssembly.compile(source);

const exports = WebAssembly.Module.exports(wasmModule);
const imports = WebAssembly.Module.imports(wasmModule);

const moduleLookup: Record<string, VMModule> = {};
for (const {module} of imports) {
if (moduleLookup[module] === undefined) {
moduleLookup[module] = await this.linkAndEvaluateModule(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should these be put into this. _esmoduleRegistry as well?

Copy link
Contributor Author

@kachkaev kachkaev Oct 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve replaced linkAndEvaluateModule with loadEsmModule in 26db3fd. Not 100% sure if this is right, but AFAIU, loadEsmModule is a wrapper function that deals with _esmoduleRegistry. If that’s not the best solution, feel free to push to my branch if the required change is not trivial.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm I should have kept linkAndEvaluateModule. Using loadEsmModule here can result

TypeError: modulePath.endsWith is not a function
  at isWasm (../../../node_modules/jest-runtime/build/index.js:210:41)

This happens when Wasm imports refer back to a javascript file that calls it. E.g.:

imports: [
  {
    module: './index_bg.js',
    name: '__wbindgen_json_parse',
    kind: 'function'
  },
  {
    module: './index_bg.js',
    name: '__wbindgen_json_serialize',
    kind: 'function'
  },
  {
    module: './index_bg.js',
    name: '__wbindgen_throw',
    kind: 'function'
  }
]

loadEsmModule returns

modulePath: SourceTextModule {
  status: 'linking',
  identifier: '/path/to/index_bg.js',
  context: {
    clearInterval: [Function: clearInterval],
    clearTimeout: [Function: clearTimeout],
    setInterval: [Function: setInterval],
    setTimeout: [Function: setTimeout] {
      [Symbol(nodejs.util.promisify.custom)]: [Getter]
    },

which is then mistakenly passed to to isWasm instead of a string.

I can try submiting a follow-up PR with a fix in a few days.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

await this.resolveModule(module, identifier, context),
);
}
}

const syntheticModule = new SyntheticModule(
exports.map(({name}) => name),
function () {
const importsObject: WebAssembly.Imports = {};
for (const {module, name} of imports) {
if (!importsObject[module]) {
importsObject[module] = {};
}
importsObject[module][name] = moduleLookup[module].namespace[name];
}
const wasmInstance = new WebAssembly.Instance(
wasmModule,
importsObject,
);
for (const {name} of exports) {
// @ts-expect-error: TS doesn't know what `this` is
this.setExport(name, wasmInstance.exports[name]);
}
},
{context, identifier},
);

return syntheticModule;
}

private _getMockedNativeModule(): typeof nativeModule.Module {
if (this._moduleImplementation) {
return this._moduleImplementation;
Expand Down