ESM support: soliciting feedback

[cspotcode]

Please use this ticket to provide feedback on our native ESM support. Your involvement is greatly appreciated to ensure the feature works on real-world projects.

Experimental warning

Node's loader hooks are EXPERIMENTAL and subject to change. ts-node's ESM support is as stable as it can be, but it relies on APIs which node can and will break in new versions of node.

When node breaks their APIs, it breaks loaders using their APIs. You have been warned!

Third-party docs: "Guide: ES Modules in NodeJS"

Someone has been maintaining a great reference document explaining how to use ts-node's ESM loader.

Guide: ES Modules in NodeJS

First-party docs

Our website explains the basics:

CommonJS vs native ECMAScript modules
Options: esm

Usage

Requirements

• Set "module": "ESNext" or "ES2015" so that TypeScript emits import/export syntax.
• Set "type": "module" in your package.json, which is required to tell node that .js files are ESM instead of CommonJS. To be compatible with editors, the compiler, and the TypeScript ecosystem, we cannot name our source files .mts nor .mjs.
• Include file extensions in your import statements, or pass --experimental-specifier-resolution=node Idiomatic TypeScript should import foo.ts as import 'foo.js'; TypeScript understands this.
  • The language service accepts configuration to include the file extension in automatically-written imports. In VSCode:
    

Invocation

ts-node-esm ./my-script.ts

ts-node --esm ./my-script.ts

# If you add "esm": true to your tsconfig, you can omit the CLI flag
ts-node ./my-script.ts

# If you must invoke node directly, pass --loader
node --loader ts-node/esm ./my-script.ts

# To force the use of a specific tsconfig.json, use the TS_NODE_PROJECT environment variable
TS_NODE_PROJECT="path/to/tsconfig.json" node --loader ts-node/esm ./my-script.ts

# To install the loader into a node-based CLI tool, use NODE_OPTIONS
NODE_OPTIONS='--loader ts-node/esm' greeter --config ./greeter.config.ts sayhello

ts-node-esm / --esm / "esm": true work by spawning a subprocess and passing it the --loader flag.

Configuration

When running ts-node --esm, ts-node-esm, or ts-node all CLI flags and configuration are parsed as normal. However, when passing --loader ts-node/esm, the following limitations apply:

• tsconfig.json is parsed.
• CLI flags are not parsed.
• Environment variables are parsed.
• ts-node must be installed locally, not globally. npm install ts-node or yarn add ts-node.

tsconfig will be resolved relative to process.cwd() or to TS_NODE_PROJECT. Specify ts-node options in your tsconfig file. For details, see our docs.

Use TS_NODE_PROJECT to tell ts-node to use a specific tsconfig, and put all ts-node options into this config file.

Versioning

As long as node's APIs are experimental, all changes to ESM support in ts-node, including breaking changes, will be released as minor or patch versions, NOT major versions. This conforms to semantic versioning's philosophy for version numbers lower than 1.0. Stable features will continue to be versioned as normal.

node's API change: v16.12.0, v17.0.0

Node made a breaking change in their ESM API in version 17, backported to 16.12.0. It may also be backported to 14 and 12.
This is the change: nodejs/node#37468

ts-node automatically supports both APIs, thanks to #1457. This relies on hard-coded version number checks. If/when this is backported to node 14 and 12, we will publish a new version of ts-node with the appropriate version number checks. Be sure you are always using the latest version of ts-node to avoid problems.

---

---

---

---

Note: things below this line may be out-of-date or inaccurate. These notes were used during initial implementation, but have not been updated since

Pending development work

• Make resolution lookup use our fs caches
• Create esm-script.mjs to do --script-mode?
  • Can read process.argv for config resolution?
• Implement require('ts-node').esmImport(module, 'import-path')
  • use https://npm.im/tsimportlib
• Throw error when CJS attempts to require ESM, matching node's behavior for .js
  • See below: "Changes to existing functionality" > "require() hook"

The proposal

Below is the official proposal, explaining our implementation in detail.

---

I am asking node's modules team questions here: nodejs/modules#351

I was reading the threads about ESM support in ts-node, e.g. #935.

The @K-FOSS/TS-ESNode implementation is unfortunately incomplete; it does not attempt to typecheck. (it uses transpileModule)

So I did some research. Below is a proposal for ESM support in ts-node, describing the required behavior in detail.

This doesn't feel like an urgent feature to me, but I like having an official proposal we can work on.

---

Usage

node --loader ts-node/esm ./entrypoint.ts

Cannot be invoked as ts-node because it requires node flags; hooks cannot be enabled at runtime. This is unavoidable.

For simplicity, --require ts-node/register can be eliminated, because ts-node/esm automatically does that.

Alternatively, we publish an experimental ts-node-esm entry-point which invokes a node subprocess.

---

Don't forget allowJs! Affects the treatment of .js files. (Not .mjs nor .cjs because the TS language service won't look at them)

ESM hooks

Must implement ESM hooks to resolve extensionless imports to .ts files, resolve .js to .ts, classify .ts(x) and .jsx files as CJS or MJS, and compile .ts(x) and .jsx files.

resolve() hook:

Match additional file extensions: .ts, .tsx, .jsx.

Resolve .ts, .tsx, and .jsx if the import specifier says .js. Obey preferTsExts when doing this.

_

[Good idea?] Always ask default resolver first. If it finds something, we should not interfere.

--experimental-specifier-resolution=node does not obey require.extensions, unfortunately, so we can't use that.

getFormat hook:

If the resolved file is .ts, .tsx, or .jsx, behave as if extension was .js: use node's package.json discovery behavior to figure out if ESM or CJS.

This can be accomplished by appending .js to the URL path and delegating to built-in getFormat hook.

transformSource hook:

Same as today's code transformer. Relies on projects to be configured correctly for import/export emit.

Changes to existing functionality

require() hook

• Use same getFormat logic to determine if node will treat file as CJS or ESM.
• NOTE node already detects and throws some errors on its own. But if require.resolve points to a .ts file, we need to make the determination.
• If ESM, throw the same error as NodeJS ("cannot load ESM via require()")
  • copying logic from node's source here

require() code transform

• Must somehow allow import() calls.
• Force consumers to use require('ts-node').esmImport(module, 'import-path')?

ts-node bin entry-point

ts-node CLI does NOT need to support import()ing ESM.

WHY? Because ESM hooks are an experimental feature which must be enabled via node CLI flag.

Thus we will be loaded via --require, and Node is responsible for loading the entry-point, either triggering our hook or our require.extensions.

Allow import() in CJS

If "module": "commonjs", compiler transforms import() into __importStar

No way to change this without a custom transformer, which IMO is too much complexity at this time.

Users should run their code as ESM.

If they can't do that, we can recommend the following workaround:

// This is in a CommonJS file:
const dynamicallyImportedEsmModule = await require('ts-node').importESM('./specifier-of-esm-module', module);

Emit considerations

NOTE we have not implemented the following, although initially I thought we might. Instead, we assume tsconfig is configured for either ESM or CJS as needed

We could intelligently emit both "module": "esnext" and "module": "commonjs" depending on the classification of a file.

In transpile-only mode this is simple. Call transpileModule with different options.

When typechecking, we can pull SourceFile ASTs from the language service / incremental compiler.

We'll need a second compiler, one for each emit format. Or we can hack it by using transpileModule for all ESM output. transpileModule is incompatible with certain kinds of TS code, (can't do const enums) but it might work for a first-pass implementation.

[cspotcode]

TODO: turns out, users can tell the language service to include the .js file extension with automatically-written imports. So we do not need to automatically add them, though we do need to check if a .js import might point to a .ts or .tsx file.

The option is passed to the language service in a ts.UserPreferences object.
https://discordapp.com/channels/508357248330760243/640177429775777792/703301413337432114

[cspotcode]

I was trying to figure out if ts-node needs to automatically switch the "module" option between CommonJS and ESNext depending if we need to emit CommonJS or ESM. I concluded we do not want to do this. Here's an explanation anyway, in case I am proven wrong.

Today, ts-node respects the tsconfig's module option. Users are required to set it appropriately. If the user incorrectly sets module to ESNext and then tries to require() a TS file, they get an error because the emitted code has import statements.

Alternatively, we can automatically override the module option to be CommonJS when emitting for require() and ESNext when emitting for ESM. This allows a single tsconfig to be used for both ESM and CommonJS.

After thinking about this, it doesn't make sense. Users will choose either ESM or CommonJS via their package.json file. They won't do a mix of both. Also, this would get pretty messy since we'd be doing something that doesn't match tsc's output.

Nevertheless, if we wanted to implement this:

If the module option is already correct, we can use the languageService's getEmitOutput() like we do today. If not, we can grab a reference to the SourceFile and transform it using the same technique as transpileModule's implementation. This allow custom emit while avoiding an expensive parse.

TypeScript has an internal sourceFileAffectingCompilerOptions array. If any of those options differ, a SourceFile cannot be reused. However, some are only relevant if you care about diagnostics. For swapping out the module flag, I think SourceFile can always be reused.

[cspotcode]

We have released an experimental implementation of this in v8.10.1. Please test and share your feedback here.

[chpeters]

Thanks @cspotcode for the release! Everything seems be working minus one snafu. Importing named exports don't seem to be working, but this may be a Node module quirk. For example in index.ts:

import { graphql } from 'graphql';

will cause a syntax error of:

SyntaxError: The requested module 'graphql' does not provide an export named 'graphql'

but this can be solved by using destructuring:

import Graphql from 'graphql';
const { graphql } = Graphql;

Any way to support importing named exports in ts files?

[blakeembrey]

@chpeters I'd guess that would be because graphql is actually CommonJS and not an ES module. You can read more about it here: https://nodejs.org/api/esm.html#esm_interoperability_with_commonjs. Unfortunately it'll probably be messy for a while with TypeScript since the imports syntax is overloaded to represent both CommonJS and native ES modules.

[NeilujD]

Using mocha and TypeScript with ES modules I am facing an issue and I don't quite understand it.

Running this cmd as my test cmd :

node --experimental-modules --loader ts-node/esm.mjs ./node_modules/mocha/bin/mocha --extension ts

I get this error :

import './unit/authentication.js';
^^^^^^

SyntaxError: Cannot use import statement outside a module

What did I do wrong ?

PS: I have my tsconfig.json module attribute set to "ES2015", my package.json type attribute to "module", ts-node installed locally

[cspotcode]

Please send me a minimal reproduction and I'll be able to tell you.

…

On Fri, May 8, 2020, 12:01 PM Julien Collard ***@***.***> wrote: Using mocha and TypeScript with ES modules I am facing an issue and I don't quite understand it. Running this cmd as my test cmd : node --experimental-modules --loader ts-node/esm.mjs ./node_modules/mocha/bin/mocha --extension ts I get this error : import './unit/authentication.js';^^^^^^ SyntaxError: Cannot use import statement outside a module What did I do wrong ? PS: I have my tsconfig.json module attribute set to "ES2015", my package.json type attribute to "module", ts-node installed locally — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1007 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OCOQXHWRG4OZI2UC63RQQUFZANCNFSM4MGJCWPA> .

[NeilujD]

This is my project architecture :

src
  |_index.ts
test
  |_tests.ts
  |_unit
      |_authentication.ts
package.json
tsconfig.json

My package.json :

{
  "name": "my-project",
  "version": "1.0.0",
  "description": "My project",
  "main": "lib/index",
  "type": "module",
  "files": [
    "lib/**/*"
  ],
  "directories": {
    "test": "test"
  },
  "scripts": {
    "build": "tsc",
    "test": "node --experimental-modules --loader ts-node/esm.mjs ./node_modules/mocha/bin/mocha --extension ts"
  },
  "devDependencies": {
    "@types/chai": "^4.2.11",
    "@types/mocha": "^7.0.2",
    "@types/node": "^13.13.5",
    "chai": "^4.2.0",
    "mocha": "^7.1.2",
    "ts-node": "^8.10.1",
    "typescript": "^3.8.3"
  }
}

My tsconfig.json :

{
  "compilerOptions": {
    "target": "ES2015", 
    "module": "ES2015", 
    "lib": ["es6"], 
    "declaration": true,
    "outDir": "lib",
    "rootDir": "src",
    "strict": true, 
    "noImplicitAny": true,   
    "moduleResolution": "node",   
    "esModuleInterop": true,  
    "forceConsistentCasingInFileNames": true
  },
  "exclude": [
    "test/"
  ]
}

My test/tests.ts :

import './unit/authentication.js'

Typescript is building my files right.
The npm run test cmd returns throw the error I wrote before.

Do you need more context ?

[cspotcode]

@NeilujD this is perfect, thanks.

It looks like, due to missing features in node's ESM support, mocha is using a hack to figure out whether a file should be loaded as ESM or CJS.
https://github.com/mochajs/mocha/blob/master/lib/esm-utils.js#L4-L23

ts-node's require() hook will need to be updated to match the error behavior of node's .js hook. When you try to require() a TS file that should be treated as ESM, we should throw an error.

At first I thought mocha could simply import() everything, since it automatically switches to CommonJS loading as needed. However, that would require our ESM hook to be installed in order to resolve and classify .ts files. They're forced to use require() to cater to legacy require() hooks.

[cspotcode]

I think I can hack this by delegating to node's built-in .js extension, passing a fake filename.

require.extensions['.js']({_compile(){}}, filename + 'DOESNOTEXIST.js')

At the cost of a failed fs call, this will cause the built-in require hook to do its package.json lookup and see if the file should be treated as CommonJS or ESM.

> require.extensions['.js'].toString()
'function(module, filename) {\n' +
  "  if (filename.endsWith('.js')) {\n" +
  '    const pkg = readPackageScope(filename);\n' +
  "    // Function require shouldn't be used in ES modules.\n" +
  "    if (pkg && pkg.data && pkg.data.type === 'module') {\n" +
  '      const parentPath = module.parent && module.parent.filename;\n' +
  "      const packageJsonPath = path.resolve(pkg.path, 'package.json');\n" +
  '      throw new ERR_REQUIRE_ESM(filename, parentPath, packageJsonPath);\n' +
  '    }\n' +
  '  }\n' +
  "  const content = fs.readFileSync(filename, 'utf8');\n" +
  '  module._compile(content, filename);\n' +
  '}'

EDIT I've shared this hack with the node folks nodejs/modules#351 (comment) to see if they have any interest in exposing this API natively.

[castarco]

Responding to #1007 (comment)

Cannot be invoked as ts-node because it requires node flags; hooks cannot be enabled at runtime. This is unavoidable.

Actually... it should be possible, by providing a very simple posix shell script wrapping the whole thing. That would be ideal when using shebangs, as it's not allowed to pass options to shebangs in Linux (although it's possible in Mac).

[cspotcode]

@castarco the problem is cross-platform support that matches the npm ecosystem without adding complexity to ts-node. Typically this is handled by the package manager: npm, yarn, pnpm, etc.

We set up our package.json "bin" field, pointing to a file with a shebang, and it takes care of creating a symlink, .cmd shim, and .ps1 shim.

[cspotcode]

@NeilujD The mocha issue you were seeing should be fixed by #1031 which has been merged to master.

If you're feeling adventurous, you can install ts-node directly from master.

npm i TypeStrong/ts-node#master --save

Or you can download and install the tarball artifact produced by CI.

npm install ts-node-packed.tgz --save

[fbartho]

What does that syntax mean?

[mogzol]

It's not special syntax, node subpath import patterns just have to start with a #. So #~/* is just a shorter alternative to something like #root/*, it just depends on what you have defined in the "imports" field of package.json.

So in this case, any import starting with #~/ will resolve to an import relative to baseUrl. i.e. something like this:

import { someUtil } from "#~/utils/someUtil.js";

resolves to:

import { someUtil } from "<baseUrl>/utils/someUtil.js";

[berekmerilorand]

Receiving unknown file extension ".ts" error after upgrading wdio to 8 and Appium >16.

Detailed description here.

[Danielku15]

This topic seems to be quite a long runner already (didn't read yet through the whole history) but it seems there is not really an end in sight. What do you think about the following intermediate solution:

• Export dist/esm as public API. I think registerAndCreateEsmHooks might be even enough.
• Add a new option to registerAndCreateEsmHooks where a caller can provide a custom pre-processor for module paths. This pre-processor is then called here.

This way, until we have official support, we can at least create 3rd party modules which will rewrite any paths before further resolved by ts-node. I already successfully created a working solution using tsconfig-paths. My current solution unfortunately needs to rely on rather dirty patching of the resolver to inject this rewrite.

[rtritto]

This change should help/fix (it's released in Node v19.6.0 and there is a backport to v18).

Another related/workaround is ts-import.

[karts-with-freaking-lasers]

Environment:
typescript v4.9.5
ts-node v10.9.1 # This seemed to be a necessary upgrade, but was interspersed amongst many other attempts
node v18.15.0
compiler v4.1.6

Explanation:
Chat GPT helped me ultimately find this page which ultimately led me to find a solution to the following nuances:

1. Required type:module inside package.json
2. Using import: import config from './config.ts'
3. It gets really fuzzy on the relevant requirements from here...

I ultimately need to do 2 things:

1. npm run my-oneoff or npm run dev:backend
2. npm run build:backend
3. node build/backend/index.js # a given after build:backend

I hopefullly added only the revelation-data that allowed me to finally put this dying horse to sleep for the night. My transition to Deno can't come soon enough; then I can shoot this poor horse out of its misery and TypeScript can run as a first-class citizen as God intended.

# package.json
{
  "type": "module",  # Removing led to the revelation as `my-oneoff` executed wholly and completely, but... I NEEDS IT!
  "scripts": {
    "my-oneoff": "TS_NODE_PROJECT=custom.tsconfig.json node --loader ts-node/esm src/backend/my-oneoff.ts",
    "dev:backend": "npx nodemon",
    "build:backend": "tsc --project custom.tsconfig.json",
    "postbuild:backend": "tsc-esm-fix --tsconfig='custom.tsconfig.json' --ext='.js'"
  }
}

# nodemon.json
{
  "watch": [
    "src/shared",
    "src/backend"
  ],
  "ext": "ts",
  "exec": "ts-node --project custom.tsconfig.json src/backend"
}

# custom.tsconfig.json
{
  "ts-node": {
    "esm": true,  # suspiciously unequivalent to `ts-node --esm`, which no longer seems to work anyway. ¯\_(⊙︿⊙)_/¯
    "experimentalSpecifierResolution": "node",  # or `node --loader ts-node/esm --experimental-specifier-resolution=node`
  },
  "compilerOptions": {
    "moduleResolution": "node",
    "module": "esnext",
    "target": "esnext",
    "types": ["node"],
    "esModuleInterop": true,
  }
}

I pray this helps but 1 person save the nightmarish day I've had--hopefully it will never be helpful to me again after I transition to Deno. :*

[FFdhorkin]

This workaround:

Does not work. It emits TypeError: importESM is not a function

I confirmed that importESM doesn't exist anywhere...

[Vanilagy]

@karts-with-freaking-lasers This works! Thanks, you did save from me a nightmarish day :)

[kkolinko]

The --loader option in Node has been renamed to --experimental-loader and deprecated in favor of using --import + a call to module.register().

Rename happened quite some time ago (Node 12.11.1, in October 2019).

Deprecation warning is printed when running with Node 20.

Documentation:
https://nodejs.org/dist/latest-v20.x/docs/api/cli.html#--experimental-loadermodule
(Click on "History" to see the rename.)

It means:

1. A new entry point to be used with --import instead of --loader ts-node/esm.

   E.g. one from Export ts-node/esm-register #2073.

2. An update to documentation is needed: In every place that documents the use of --loader.

[mirismaili]

node --import "data:text/javascript,import {register} from 'node:module'; import {pathToFileURL} from 'node:url'; register('ts-node/esm', pathToFileURL('./'))" my-script.ts

It works on Node.js v20.10.0. 😢

---

But you can create a file named ts-loader.js:

import {register} from 'node:module'
import {pathToFileURL} from 'node:url'

register('ts-node/esm', pathToFileURL('./'))

And then:

node --import ./ts-loader.js my-script.ts

Remember: Don't write the loader path as ts-loader.js (if loader file is located in your source). But make sure write it with relative path: ./ts-loader.js!

[nickmccurdy]

Note that as of Node 20 you'll need to name it ts-loader.js to work around ERR_UNKNOWN_FILE_EXTENSION.

[crfrolik]

node --import "data:text/javascript,import {register} from 'node:module'; import {pathToFileURL} from 'node:url'; register('ts-node/esm', pathToFileURL('./'))" my-script.ts

It works on Node.js v20.10.0. 😢

But you can create a file named ts-loader.js:

import {register} from 'node:module'
import {pathToFileURL} from 'node:url'

register('ts-node/esm', pathToFileURL('./'))

And then:

node --import ./ts-loader.js my-script.ts

Remember: Don't write the loader path as ts-loader.js (if loader file is located in your source). But make sure write it with relative path: ./ts-loader.js!

This solution led to an error for me:

file:///<path-to-my-app/myapp.ts:2
Object.defineProperty(exports, "__esModule", { value: true });
                      ^

ReferenceError: exports is not defined in ES module scope
    at file:///<path-to-my-app/myapp.ts:2:23
    at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
    at async loadESM (node:internal/process/esm_loader:34:7)
    at async handleMainPromise (node:internal/modules/run_main:106:12)

Node.js v18.19.0

[GaoJuqian]

I used the graphql plugin of jetbrains idea editor, and the following error occurred.

Node.js v20.11.0

8097-graphql

Code & tsconfig

Error

java.lang.Throwable: (node:79373) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("ts-node/esm", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
Error: Cannot find module '/Users/gjq/WebstormProjects/MemberManagement/frontend/src/api/supabase' imported from /Users/gjq/WebstormProjects/MemberManagement/frontend/graphql.config.ts
    at finalizeResolution node_modules//ts-node@10.9.2_@types+node@20.11.30_typescript@5.4.3/node_modules//dist-raw/node-internal-modules-esm-resolve.js:366:11
    at moduleResolve node_modules//ts-node@10.9.2_@types+node@20.11.30_typescript@5.4.3/node_modules//dist-raw/node-internal-modules-esm-resolve.js:801:10
    at Object.defaultResolve node_modules//ts-node@10.9.2_@types+node@20.11.30_typescript@5.4.3/node_modules//dist-raw/node-internal-modules-esm-resolve.js:912:11
    at node_modules//ts-node@10.9.2_@types+node@20.11.30_typescript@5.4.3/node_modules//src/esm.ts:218:35
    at entrypointFallback node_modules//ts-node@10.9.2_@types+node@20.11.30_typescript@5.4.3/node_modules//src/esm.ts:168:34
    at node_modules//ts-node@10.9.2_@types+node@20.11.30_typescript@5.4.3/node_modules//src/esm.ts:217:14
    at addShortCircuitFlag node_modules//ts-node@10.9.2_@types+node@20.11.30_typescript@5.4.3/node_modules//src/esm.ts:409:21
    at resolve node_modules//ts-node@10.9.2_@types+node@20.11.30_typescript@5.4.3/node_modules//src/esm.ts:197:12

When I changed it to import . js, this problem was solved.

import {supabaseKey, supabaseUrl} from "./src/api/supabase.js";