Add path mapping support to ESM and CJS loaders #1585

geigerzaehler · 2021-12-29T14:37:10Z

The ESM loader now resolves import paths using TypeScripts path mapping feature.

EDIT:
Closes #1586

The ESM loader now resolves import paths using TypeScripts [path mapping][1] feature. [1]: https://www.typescriptlang.org/docs/handbook/module-resolution.html#path-mapping Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

cspotcode · 2021-12-29T18:16:14Z

Thanks for this. Because of the American holidays, I have somewhat of a backlog of issues and PRs which need attention before I'll be able to properly review this. There may be some delay.

I see some refactoring in node-errors; can you explain the motivation for some of those changes? It's good to have an explanation for such changes when reviewing.

Your description says this enables path mapping for the ESM loader. We'd like to also extend the same functionality to the CommonJS loader. I see that your createPathMapper implementation looks clean and modular; do you anticipate any issues if we attempt to use it for CommonJS as well? We'd hook into Module._resolveFilename for this.

Is this feature-flagged? We should decide if it's enabled or disabled by default, and decide how users can opt-in or opt-out.

geigerzaehler · 2021-12-29T18:48:19Z

There may be some delay.

No problem.

I see some refactoring in node-errors; can you explain the motivation for some of those changes?

The new resolving code needs to detect ERR_MODULE_NOT_FOUND errors so it can try more mapped candidates. The builtin NodeError from Node’s internal error module has a code property that allows you to distinguish errors. The change in node-errors also sets this property on the error instances. The new logic for defining errors is similar to the code in Node’s error module.

do you anticipate any issues if we attempt to use it for CommonJS as well?

I don’t think there will be any issues. The path mapping code is does not include any logic that is specific to ESM resolution. The only inputs to it are the TypeScript compiler options.

Is this feature-flagged? We should decide if it's enabled or disabled by default, and decide how users can opt-in or opt-out.

It’s not feature-flagged at the moment because I couldn’t think of a scenario where you would want the path mapping to be disabled. If you have configured the paths compiler option and you are using it in your code, your code will only run if the path mapping is enabled. If path mapping is disabled the code will always fail at the module resolution stage which makes the option unusable. So I’d lean towards not feature-flagging this for simplicity. But if there’s a good reason to put this behind a feature flag it should be easy to accomplish

cspotcode · 2021-12-29T20:13:06Z

If you have configured the paths compiler option and you are using it in your code, your code will only run if the path mapping is enabled.

Good point. And I think someone could disable it like this:

{
  "compilerOptions": {
    "paths": { ... }
  },
  "ts-node": {
    "compilerOptions": {
      "paths": {} // empty object: disable path mappings in ts-node
    }
  }
}

codecov · 2021-12-29T20:24:43Z

Codecov Report

Merging #1585 (6632481) into main (ff78b64) will increase coverage by 1.24%.
The diff coverage is 93.49%.

Impacted Files	Coverage Δ
src/configuration.ts	`86.06% <ø> (ø)`
src/index.ts	`79.56% <71.42%> (-0.13%)`	⬇️
src/esm.ts	`84.72% <92.30%> (+1.38%)`	⬆️
src/path-mapping.ts	`92.85% <92.85%> (ø)`
src/cjs-resolve-hooks.ts	`87.17% <96.00%> (+13.84%)`	⬆️
dist-raw/node-internal-errors.js	`97.50% <100.00%> (+1.84%)`	⬆️
dist-raw/node-internal-modules-cjs-loader.js	`76.42% <0.00%> (+4.56%)`	⬆️
dist-raw/node-internalBinding-fs.js	`100.00% <0.00%> (+7.69%)`	⬆️

cspotcode

I left comments from an initial code review. I plan to implement the requested changes myself, but if you are feeling enthusiastic and would like to do them before me, I will certainly appreciate the assistance. No pressure though.

Overall this looks great to me. I definitely want to add CommonJS support which shouldn't be too difficult. I'm already familiar with the hooking API.

src/esm.ts

cspotcode · 2022-01-24T05:26:07Z

src/path-mapping.ts

+): PathMapper {
+  if (compilerOptions.paths) {
+    if (!compilerOptions.baseUrl) {
+      throw new Error(`Compiler option 'baseUrl' required when 'paths' is set`);


Is compilerOptions.baseUrl guaranteed to be an absolute path, or do we need to resolve it relative to the tsconfig's location?

Good point. I assumed that we always get the configuration from ts.readConfigFile() in src/configuration.ts and the function resolves the base URL relative to the config file. But there are probably other ways to set the base URL like through the command line, API, or environment.

To address this I think it make sense to resolve any base URL that is not a URL but a relative path to the current directory.

I see, we almost always get the configuration from ts.readConfigFile(), so we should be good. I think the only way to set it otherwise is via the API? Since we don't support all of tsc's command-line flags.

Maybe the correct approach is to resolve baseUrl once within create() so that we have a reliable, absolute baseUrl to be used elsewhere.

Maybe the correct approach is to resolve baseUrl once within create() so that we have a reliable, absolute baseUrl to be used elsewhere.

Alternatively, we could require baseUrl to be absolute and throw an error otherwise. This would allow us to avoid implicit, environment-dependent behavior for users of the API. This might prevent some confusion when path mapping doesn’t work as expected but only fails at the resolution stage instead of the construction stage. (For example a user may think the base URL they set on the API is considered relative to tsconfig.json and their scripts work whenever they are run from the project root. But then they run the script from a subdirectory and it fails.)

Sounds good, let's do that. In create() we throw an error if baseUrl is not absolute.

Actually, scratch that, seems we always pass options through TypeScript's API to be normalized, even when provided via our API:

$ node > require('ts-node').create({compilerOptions: {baseUrl: "foobar"}}).config.options.baseUrl '/home/ubuntu/dev/ts-node/ts-node/path-mapping/foobar'

So I guess we're all good.

tests/esm-path-mapping/index.ts

Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

geigerzaehler · 2022-01-24T11:15:08Z

I definitely want to add CommonJS support which shouldn't be too difficult. I'm already familiar with the hooking API.

I’d prefer to keep scope of this PR small and I’m not familiar enough with the CommonJS side of things so I’ll leave this to you.

One thing to consider is that we’d probably want the users to explicitly opt in to path mapping for CommonJS. They’ll be using tsconfig-paths at the moment and it is unclear if the ts-node implementation can be used side-by-side or compatible.

cspotcode · 2022-01-24T16:34:49Z

Fair point. Would it be a great hardship for those users to disable tsconfig-paths? I hope they would appreciate the simplicity of removing a component. I would like our default behaviour to be the best forward-thinking behaviour, but perhaps we add an optional flag which can be used to selectively disable it for commonjs? Or we somehow detect tsconfig-paths and throw a warning when both mappers are competing?

…

On Mon, Jan 24, 2022 at 6:15 AM Thomas Scholtes ***@***.***> wrote: I definitely want to add CommonJS support which shouldn't be too difficult. I'm already familiar with the hooking API. I’d prefer to keep scope of this PR small and I’m not familiar enough with the CommonJS side of things so I’ll leave this to you. One thing to consider is that we’d probably want the users to explicitly opt in to path mapping for CommonJS. They’ll be using tsconfig-paths at the moment and it is unclear if the ts-node implementation can be used side-by-side or compatible. — Reply to this email directly, view it on GitHub <#1585 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OGJBEFQKHZNAXQ5F33UXUYERANCNFSM5K6CBQ2Q> . You are receiving this because you commented.Message ID: ***@***.***>

src/esm.ts

src/index.ts

cspotcode · 2022-01-25T05:08:39Z

I made an attempt at adding the CommonJS resolver hook. Still some outstanding TODOs that I need to address, but this is looking good.

cspotcode · 2022-01-25T05:32:10Z

Looks like we have test failures on Windows due to module specifiers using forward slashes and windows paths using backslashes. I haven't checked the code yet, but we probably have some logic assuming a module specifier and a path will always look the same. Should be an easy fix.

https://github.com/TypeStrong/ts-node/runs/4932463748?check_suite_focus=true#step:15:167

geigerzaehler · 2022-01-25T11:43:17Z

Would it be a great hardship for those users to disable tsconfig-paths?

I don’t think it would. I just wanted to point out, that this is a breaking change. I do believe most users will appreciate this replacement.

Considering the release strategy it probably makes sense to have the commonjs mapping as opt-in and ask users to enable it to collect feedback. Once the feature is stable it can be enabled by default (or always) in a new major release.

Or we somehow detect tsconfig-paths and throw a warning when both mappers are competing?

This sounds complex and error-prone. I think I’d prefer a solution where it is explicitly configured

cspotcode · 2022-01-25T15:11:26Z

I like that plan: opt-in via a flag today, enable by default in the next major version, get feedback in the meantime.

How about experimentalPathMapping: 'both' | 'cjs' | 'esm' | 'none'? Where 'esm' is the default today, and 'both' is the default in the next major version, with the flag renamed to pathMapping?

rhummelmose · 2022-09-07T09:40:53Z

Really need this, please prioritize if possible :)

menosprezzi · 2022-09-13T23:06:20Z

@cspotcode How can I help here?

hpx7 · 2022-12-04T22:58:23Z

Is this work still planned?

RedStars071 · 2022-12-08T11:31:54Z

some news of this pr?

testgitdl · 2022-12-20T08:59:30Z

Hello! Do you know when this will be available? Thanks!

kuksik · 2023-03-16T21:59:42Z

some news about this PR?

mprinc · 2023-03-16T22:48:32Z

Sad that the ts-node tool is too complex (including its dependencies) that it is not possible to handle it promptly enough but it rather takes year and half ... honestly the whole ESM model with ts-node + ts paths is broken because of that, and a lot of libs are now delivering ONLY ESM packages ... Thanks God, I am using that workaround (--experimental-specifier-resolution=node --loader ) and more less I am ok, but it is insane that it cannot be integrated properly and that I have to have that ugly scripts:

node --experimental-specifier-resolution=node --loader $COLABO/ts-esm-loader-with-tsconfig-paths.js index.ts rather than just ts-node index.ts ...

0x80 · 2023-03-17T23:33:43Z

@mprinc The workaround is not yet clear to me from skimming through this discussion. Could you share the content of your ts-esm-loader-with-tsconfig-paths.js please?

trim21 · 2023-03-17T23:37:17Z

suggesting tsx （and also esm/cjs loader in its description） for this feature

Naddiseo · 2023-03-18T17:53:57Z

@0x80

@mprinc The workaround is not yet clear to me from skimming through this discussion. Could you share the content of your ts-esm-loader-with-tsconfig-paths.js please?

This is what I'm using with node 16

/*
loader.js

Usage (I'm also using dotenv, but you can omit the dotenv parts if needed):
DOTENV_CONFIG_PATH=.env node -r dotenv/config --loader=./loader.js /bin/path/to/cli.ts
*/
import { pathToFileURL } from "node:url";
import { getFormat, load, resolve as resolveTs, transformSource } from "ts-node/esm";
import * as tsConfigPaths from "tsconfig-paths";

export { getFormat, transformSource, load };

const { absoluteBaseUrl, paths } = tsConfigPaths.loadConfig();
const matchPath = tsConfigPaths.createMatchPath(absoluteBaseUrl, paths);

export function resolve(specifier, context, defaultResolver) {
  const mappedSpecifier = matchPath(specifier);
  if (mappedSpecifier) {
    specifier = `${mappedSpecifier}.js`;

    /*
    the resolve functionality can only work with file URLs, so we need to convert, this is especially
    the case on windows, where the path might not be a valid URL.
    */
    const url = specifier.startsWith("file:") ? specifier : pathToFileURL(specifier.toString());

    return resolveTs(url.toString(), context, defaultResolver);
  } else {
    // If we can't find a mapping, just pass it on to the default resolver
    return resolveTs(specifier, context, defaultResolver);
  }
}

gudh · 2023-03-21T06:02:11Z

@trim21 thanks, tsx works like a charm, no configuration needed!

troywweber7 · 2023-03-21T14:19:53Z

@trim21 I might be missing something... Why tsx and not mts (similar to mjs)? Or maybe I'm confusing who this was directed at...

trim21 · 2023-03-21T14:52:07Z

@trim21 I might be missing something... Why tsx and not mts (similar to mjs)? Or maybe I'm confusing who this was directed at...

that's a link, click it

troywweber7 · 2023-03-21T15:18:13Z

@trim21 thanks. My client wasn't rendering the link. Boo. Will look in browser.

mprinc · 2023-03-22T18:37:36Z

Hi Thijs (@0x80 ) here is the code I am using ts-esm-loader-with-tsconfig-paths.js:

// https://github.com/TypeStrong/ts-node/discussions/1450#discussioncomment-1806115

// "serve": "cross-env TS_NODE_PROJECT=\"tsconfig.build.json\" node --experimental-specifier-resolution=node --loader ./loader.js src/index.ts",
// TS_NODE_PROJECT=\"tsconfig.build.json\" node --experimental-specifier-resolution=node --loader ./loader.js src/index.ts"

import { resolve as resolveTs } from "ts-node/esm";
import * as tsConfigPaths from "tsconfig-paths";
import { pathToFileURL } from "url";

const { absoluteBaseUrl, paths } = tsConfigPaths.loadConfig();
const matchPath = tsConfigPaths.createMatchPath(absoluteBaseUrl, paths);

export function resolve(specifier, ctx, defaultResolve) {
	const match = matchPath(specifier);
	return match ? resolveTs(pathToFileURL(`${match}`).href, ctx, defaultResolve) : resolveTs(specifier, ctx, defaultResolve);
}

export { load, getFormat, transformSource } from 'ts-node/esm'

together with the command again:

# $COLABO is a path to where the file is tored
node  --experimental-specifier-resolution=node --loader $COLABO/ts-esm-loader-with-tsconfig-paths.js index.ts

mprinc · 2023-03-22T18:40:28Z

NOTE: I am investigating tsx and considering already using esBuild actively in my other projects like codemod tools, which I have a great experience with both efficiency and community, I might soon switch to it. It seems pretty straightforward to do it.

loynoir · 2023-04-24T03:57:11Z

suggesting tsx （and also esm/cjs loader in its description） for this feature

@trim21 Coming from privatenumber/tsx#113 😞

Any news about this PR?

parzhitsky · 2023-09-11T19:51:23Z

dist-raw/node-internal-errors.js

+exports.codes = {};
+
+function defineError(code, buildMessage) {
+  if (!buildMessage) {
+    buildMessage = (...args) => args.join(' ');
  }
-}

-function createErrorCtor(errorMessageCreator) {
-  return class CustomError extends Error {
+  exports.codes[code] = class CustomError extends Error {
    constructor(...args) {
-      super(errorMessageCreator(...args))
+      super(`${code}: ${buildMessage(...args)}`);
+      this.code = code;
    }
  }
 }
+
+defineError("ERR_INPUT_TYPE_NOT_ALLOWED");
+defineError("ERR_INVALID_ARG_VALUE");
+defineError("ERR_INVALID_MODULE_SPECIFIER");
+defineError("ERR_INVALID_PACKAGE_CONFIG");
+defineError("ERR_INVALID_PACKAGE_TARGET");
+defineError("ERR_MANIFEST_DEPENDENCY_MISSING");
+defineError("ERR_MODULE_NOT_FOUND", (path, base, type = 'package') => {
+  return `Cannot find ${type} '${path}' imported from ${base}`;
+});
+defineError("ERR_PACKAGE_IMPORT_NOT_DEFINED");
+defineError("ERR_PACKAGE_PATH_NOT_EXPORTED");
+defineError("ERR_UNSUPPORTED_DIR_IMPORT");
+defineError("ERR_UNSUPPORTED_ESM_URL_SCHEME");
+defineError("ERR_UNKNOWN_FILE_EXTENSION");


Why is this portion of the file exactly identical to dist-raw/node-errors.js?

Baterka · 2024-01-04T15:56:54Z

Any movement on this?

sthasam2 · 2024-01-05T06:24:23Z

Is this still in the works?

Dudeprogram · 2024-01-05T11:04:07Z

no dear

…

On Thu, Jan 4, 2024 at 8:57 PM Baterka ***@***.***> wrote: Any movement on this? — Reply to this email directly, view it on GitHub <#1585 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWO46HSOCMJUSOCBYVE7RETYM3GOVAVCNFSM5K6CBQ22U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBXG4ZTGNZSG44Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

billomore · 2024-02-11T21:19:54Z

tsx doesn't play well with monorepos, namely with decorators in monorepos. swc doesn't play nice with ESM either, not sure if planned to fix. ts-node was the last citadel that worked out of the box until recently, but without any change or update it just stopped... sad :(

genesiscz · 2024-04-24T22:38:56Z

we really need this :(

Add path mapping support to ESM loader

3be1fde

The ESM loader now resolves import paths using TypeScripts [path mapping][1] feature. [1]: https://www.typescriptlang.org/docs/handbook/module-resolution.html#path-mapping Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

cspotcode mentioned this pull request Dec 29, 2021

[EPIC] Full TS-style resolution hooks #1514

Open

1 task

cspotcode mentioned this pull request Dec 29, 2021

Set correct code on errors returned by ESM resolver #1586

Open

geigerzaehler added 3 commits December 29, 2021 22:43

fixup! Add path mapping support to ESM loader

c542d98

fixup! fixup! Add path mapping support to ESM loader

0cbfa6c

fixup! fixup! fixup! Add path mapping support to ESM loader

e7082b1

cspotcode mentioned this pull request Jan 15, 2022

tsconfig.paths not working with ES modules #1596

Closed

cspotcode added this to the next milestone Jan 21, 2022

cspotcode requested changes Jan 24, 2022

View reviewed changes

cspotcode and others added 4 commits January 24, 2022 00:31

Merge remote-tracking branch 'origin/main' into path-mapping

69a397b

address code review comments

2fcb982

Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

map esm paths in all included files

48f5262

Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

improve error message when mapped module is not found

a19454b

Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

cspotcode mentioned this pull request Jan 24, 2022

ESM support: soliciting feedback #1007

Open

4 tasks

cspotcode reviewed Jan 24, 2022

View reviewed changes

src/esm.ts Outdated Show resolved Hide resolved

src/esm.ts Show resolved Hide resolved

src/esm.ts Outdated Show resolved Hide resolved

src/esm.ts Show resolved Hide resolved

src/index.ts Show resolved Hide resolved

Review changes; add CommonJS path mapping

32e26e8

fix failing tests

362935b

add path mapping to docs

e573fd7

cspotcode mentioned this pull request Jan 25, 2022

[META] v11 breaking changes #1505

Open

24 tasks

add flag to enable/disable path mapping in the two loaders

0012b22

kolarski mentioned this pull request Nov 6, 2022

Error [ERR_REQUIRE_ESM] SBoudrias/Inquirer.js#1129

Closed

deodad mentioned this pull request Dec 26, 2022

perf: use swc or esbuild toolchain to transpile code locally farcasterxyz/hub-monorepo#314

Closed

loynoir mentioned this pull request Apr 24, 2023

swc as tsx compile backend alternative privatenumber/tsx#226

Open

nextlevelbeard mentioned this pull request May 10, 2023

@wdio/cucumber-framework: Filter specs /w Cucumber Tag Expression before spawning workers webdriverio/webdriverio#10330

Merged

8 tasks

erwinheitzman mentioned this pull request Aug 24, 2023

[🐛 Bug]: Import giving error ERROR @wdio/runner: Error: Cannot find package with paths defined in tsconfig.json webdriverio/webdriverio#10857

Open

3 tasks

parzhitsky reviewed Sep 11, 2023

View reviewed changes

Naddiseo mentioned this pull request Oct 3, 2023

ESM + path mapping #2068

Open

jaredperreault-okta mentioned this pull request Nov 14, 2023

fix: select authenticator auto selects when methodType differs okta/okta-auth-js#1470

Draft

Add path mapping support to ESM and CJS loaders #1585

Are you sure you want to change the base?

Add path mapping support to ESM and CJS loaders #1585

Conversation

geigerzaehler commented Dec 29, 2021 • edited by cspotcode

cspotcode commented Dec 29, 2021

geigerzaehler commented Dec 29, 2021

cspotcode commented Dec 29, 2021

codecov bot commented Dec 29, 2021 • edited

Codecov Report

cspotcode left a comment

Choose a reason for hiding this comment

cspotcode Jan 24, 2022

Choose a reason for hiding this comment

geigerzaehler Jan 24, 2022

Choose a reason for hiding this comment

cspotcode Jan 25, 2022

Choose a reason for hiding this comment

geigerzaehler Jan 25, 2022

Choose a reason for hiding this comment

cspotcode Jan 25, 2022

Choose a reason for hiding this comment

cspotcode Jan 25, 2022 • edited

Choose a reason for hiding this comment

geigerzaehler commented Jan 24, 2022

cspotcode commented Jan 24, 2022 via email

cspotcode commented Jan 25, 2022

cspotcode commented Jan 25, 2022 • edited

geigerzaehler commented Jan 25, 2022

cspotcode commented Jan 25, 2022 • edited

rhummelmose commented Sep 7, 2022 • edited

menosprezzi commented Sep 13, 2022

hpx7 commented Dec 4, 2022

RedStars071 commented Dec 8, 2022

testgitdl commented Dec 20, 2022

kuksik commented Mar 16, 2023

mprinc commented Mar 16, 2023

0x80 commented Mar 17, 2023

trim21 commented Mar 17, 2023 • edited

Naddiseo commented Mar 18, 2023

gudh commented Mar 21, 2023

troywweber7 commented Mar 21, 2023

trim21 commented Mar 21, 2023

troywweber7 commented Mar 21, 2023

mprinc commented Mar 22, 2023 • edited

mprinc commented Mar 22, 2023

loynoir commented Apr 24, 2023 • edited

parzhitsky Sep 11, 2023

Choose a reason for hiding this comment

Baterka commented Jan 4, 2024

sthasam2 commented Jan 5, 2024

Dudeprogram commented Jan 5, 2024 via email

billomore commented Feb 11, 2024

genesiscz commented Apr 24, 2024

geigerzaehler commented Dec 29, 2021 •

edited by cspotcode

codecov bot commented Dec 29, 2021 •

edited

cspotcode Jan 25, 2022 •

edited

cspotcode commented Jan 25, 2022 •

edited

cspotcode commented Jan 25, 2022 •

edited

rhummelmose commented Sep 7, 2022 •

edited

trim21 commented Mar 17, 2023 •

edited

mprinc commented Mar 22, 2023 •

edited

loynoir commented Apr 24, 2023 •

edited