Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

npx invocation yields ERR_INVALID_MODULE_SPECIFIER #1663

Closed
brianjenkins94 opened this issue Feb 27, 2022 · 20 comments
Closed

npx invocation yields ERR_INVALID_MODULE_SPECIFIER #1663

brianjenkins94 opened this issue Feb 27, 2022 · 20 comments

Comments

@brianjenkins94
Copy link

brianjenkins94 commented Feb 27, 2022

(@cspotcode This is from a while back on Discord)

Search Terms

  • npx

Context

I run my typescript (served via npx) through ts-node so I don't have to maintain a build.

For an example of how I had this working pre-ESM, see this gist.

Expected Behavior

The program runs without error. In the case of the linked gist, it runs hello.ts, printing "Hello, world!".

Actual Behavior

$ npx https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d
Need to install the following packages:
  gist:d73578d7439427a59df28c1bc751e02d
Ok to proceed? (y) y

> node --experimental-specifier-resolution=node --loader=/Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/ts-node/esm /Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/hello.ts

(node:35543) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
node:internal/errors:464
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/ts-node/esm' imported from /Users/bjenkins/Desktop/foo/
    at new NodeError (node:internal/errors:371:5)
    at finalizeResolution (node:internal/modules/esm/resolve:405:11)
    at moduleResolve (node:internal/modules/esm/resolve:932:10)
    at defaultResolve (node:internal/modules/esm/resolve:1044:11)
    at ESMLoader.resolve (node:internal/modules/esm/loader:422:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:222:40)
    at ESMLoader.import (node:internal/modules/esm/loader:276:22)
    at initializeLoader (node:internal/process/esm_loader:74:49)
    at loadESM (node:internal/process/esm_loader:87:11)
    at runMainESM (node:internal/modules/run_main:47:21) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Steps to reproduce the problem

npx https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d

Minimal reproduction

https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d

Specifications

  • ts-node version: v10.4.0
  • node version: v16.13.2
  • TypeScript version: v4.5.4
tsconfig.json
{
	"compilerOptions": {
		"allowJs": true,
		"esModuleInterop": true,
		"forceConsistentCasingInFileNames": false,
		"incremental": true,
		"isolatedModules": true,
		"jsx": "preserve",
		"lib": [
			"dom",
			"dom.iterable",
			"esnext"
		],
		"module": "ESNext",
		"moduleResolution": "Node",
		"noEmit": true,
		"resolveJsonModule": true,
		"skipLibCheck": true,
		"strict": false,
		"alwaysStrict": true,
		//"exactOptionalPropertyTypes": true,
		//"strictNullChecks": true,
		"strictBindCallApply": true,
		"strictFunctionTypes": true,
		//"strictPropertyInitialization": true,
		//"noImplicitAny": true,
		//"noImplicitThis": true,
		//"useUnknownInCatchVariables": true,
		"noImplicitOverride": true,
		"noImplicitReturns": true,
		"noPropertyAccessFromIndexSignature": true,
		"target": "ES2020"
	},
	"include": [
		"next-env.d.ts",
		"**/*.ts",
		"**/*.tsx"
	],
	"exclude": [
		"node_modules",
		"scripts/prev.ts"
	]
}
  • Operating system and version: macOS Monterey 12.1 (21C52)
@cspotcode
Copy link
Collaborator

I think this path might be wrong:

https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d#file-npx-js-L49

When I look at the log output you shared, these are some of the paths I see:

/Users/bjenkins/Desktop/foo/
/Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/hello.ts
/Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/ts-node/esm

The first, Desktop/foo, appears to be your working directory.
The second shows that npx is installing modules into ./caa40e90fbe6bfad/node_modules
So I assume that ts-node is also being installed into ./caa40e90fbe6bfad/node_modules.

This would explain why it cannot find caa40e90fbe6bfad/ts-node/esm, because ts-node is not installed there.

@brianjenkins94
Copy link
Author

brianjenkins94 commented Feb 28, 2022

Whoops, fixed the path and now I get: ERR_INVALID_MODULE_SPECIFIER.

Note: You'll need to clear your npx cache: rm -rf ~/.npm/_npx/*

@brianjenkins94 brianjenkins94 changed the title npx invocation yields ERR_MODULE_NOT_FOUND npx invocation yields ERR_INVALID_MODULE_SPECIFIER Feb 28, 2022
@cspotcode
Copy link
Collaborator

Whoops, fixed the path and now I get: ERR_INVALID_MODULE_SPECIFIER.

Are you able to share the full error, including npx invocation and stack trace?

@brianjenkins94
Copy link
Author

brianjenkins94 commented Feb 28, 2022

$ npx https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d

> node --experimental-specifier-resolution=node --loader=/Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/ts-node/esm /Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/hello.ts

(node:38413) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:38413) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
(node:38413) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
TypeError [ERR_INVALID_MODULE_SPECIFIER]: Invalid module "file:///Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/hello.ts" 
    at new NodeError (node:internal/errors:371:5)
    at ESMLoader.load (node:internal/modules/esm/loader:324:13)
    at async ESMLoader.moduleProvider (node:internal/modules/esm/loader:230:47)
    at async link (node:internal/modules/esm/module_job:67:21) {
  code: 'ERR_INVALID_MODULE_SPECIFIER'
}

@cspotcode
Copy link
Collaborator

Thanks, I can take a closer look tonight. In the interim, I see that you're using ts-node 10.4.0. It may be worth confirming that 10.5.0 exhibits the same behavior.

@brianjenkins94
Copy link
Author

brianjenkins94 commented Feb 28, 2022

Same result by the looks of it:

$ npx https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d

> node --experimental-specifier-resolution=node --loader=/Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/ts-node/esm /Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/hello.ts

(node:38772) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:38772) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
(node:38772) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
TypeError [ERR_INVALID_MODULE_SPECIFIER]: Invalid module "file:///Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/hello.ts" 
    at new NodeError (node:internal/errors:371:5)
    at ESMLoader.load (node:internal/modules/esm/loader:324:13)
    at async ESMLoader.moduleProvider (node:internal/modules/esm/loader:230:47)
    at async link (node:internal/modules/esm/module_job:67:21) {
  code: 'ERR_INVALID_MODULE_SPECIFIER'
}

@cspotcode
Copy link
Collaborator

cspotcode commented Mar 1, 2022

I am seeing that the path may still be wrong.

Instead of

/Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/ts-node/esm

it should be

/Users/bjenkins/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/node_modules/ts-node/esm

@cspotcode
Copy link
Collaborator

Also, you might need to include the .mjs extension in ts-node/esm.mjs since you're specifying an absolute path. I'm not sure

@cspotcode cspotcode reopened this Mar 1, 2022
@cspotcode
Copy link
Collaborator

@brianjenkins94

I figured it out. The problem is primarily:

a) hello.ts file is ignored

When you run stuff via npx, it gets installed into a node_modules directory. This is ignored by ts-node by default. specifying skipIgnore fixes this.

Since the file is ignored, it does not get special treatment by our ESM loader. Node's default behaviour is used, and node does not understand the .ts file extension.

Explained here:
https://typestrong.org/ts-node/docs/how-it-works/#skipping-node_modules


If you configure skipIgnore via your tsconfig.json, you may also hit a second problem:

b) we can't automatically pick up your tsconfig.json

This is a limitation of invoking node --loader that will be avoided with #1655

When you run ts-node hello.ts, you're invoking our bin entrypoint, so we are able to locate hello.ts and find your tsconfig.json relative to it, before we execute the entrypoint. However, when you do node --loader ts-node/esm.mjs we have to load tsconfig based on your cwd before we receive the path to hello.ts. When you're using npx your cwd is very different from where npx is installing stuff. We may find a different tsconfig in your cwd, or we may not find any tsconfig and fallback to default settings.

Your npx.js bootstrapper can avoid this by setting the TS_NODE_PROJECT environment variable.

@brianjenkins94
Copy link
Author

Is it possible to pass options to the loader in the command? Like so?:

node --experimental-specifier-resolution=node --loader=ts-node/esm --skipIgnore hello.ts
                                                                   ^^^^^^^^^^^^

I couldn't really tell from the docs. Ideally I could set the --project flag the same way.

@cspotcode
Copy link
Collaborator

cspotcode commented Mar 1, 2022 via email

@brianjenkins94
Copy link
Author

brianjenkins94 commented Mar 1, 2022

Huzzah, it works!

$ npx https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d 

> node --experimental-specifier-resolution=node --loader=/Users/bjenks/.npm/_npx/caa40e90fbe6bfad/node_modules/ts-node/esm /Users/bjenks/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/hello.ts

Success!

It doesn't print out "Hello, world!" for some reason but I can dig into that on my own.

Thanks so much for your help, @cspotcode !

For the future reader, here's what worked:

https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d#file-npx-js-L10-L53

@brianjenkins94
Copy link
Author

Hmm, apparently it was just failing silently with an ENOENT, so I'm still seeing the issue and all the paths appear to be correct:

  • /Users/bjenks/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/tsconfig.json
  • /Users/bjenks/.npm/_npx/caa40e90fbe6bfad/node_modules/ts-node/esm.mjs
  • /Users/bjenks/.npm/_npx/caa40e90fbe6bfad/node_modules/sampleThatReproducesTheIssue/hello.ts

All resolve to the correct files.

I set the env property in the options for the workarounds needed for ts-node:

const options = {
	"cwd": process.cwd(),
	"encoding": "utf8",
	"env": {
+		"TS_NODE_PROJECT": path.join(directory, "tsconfig.json"), // https://github.com/TypeStrong/ts-node/pull/1655
+		"TS_NODE_SKIP_IGNORE": true
	},
	"stdio": "inherit"
};

So something still seems afoot. 🤔

Stacktrace (only visible from the debugger for some reason: npx --node-options=--inspect-brk https://gist.github.com/brianjenkins94/d73578d7439427a59df28c1bc751e02d):

Screen Shot 2022-03-02 at 9 04 03 AM

@brianjenkins94 brianjenkins94 reopened this Mar 2, 2022
@cspotcode
Copy link
Collaborator

cspotcode commented Mar 2, 2022 via email

@brianjenkins94
Copy link
Author

Ah, cool, I would not have come up with that.

It works. Thanks again!

@brianjenkins94
Copy link
Author

brianjenkins94 commented Mar 9, 2022

This isn't conclusively anything yet, but there seems to be some issue with path concatenation on Windows:

$ npx brianjenkins94/prev

> node --experimental-specifier-resolution=node --loader=file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/ts-node/esm.mjs file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/prev/scripts/prev.ts


$ node --experimental-specifier-resolution=node --loader=file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/ts-node/esm.mjs file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/prev/scripts/prev.ts
(node:25328) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\dist-raw\node-esm-resolve-implementation.js:369
    throw new ERR_MODULE_NOT_FOUND(
          ^
CustomError: Cannot find module '/C:/Users/User/Desktop/foo/file:/C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/pr
ev/scripts/prev.ts' imported from C:\Users\User\Desktop\foo\
    at finalizeResolution (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\dist-raw\node-esm-resolve-impleme
ntation.js:369:11)
    at moduleResolve (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\dist-raw\node-esm-resolve-implementati
on.js:818:10)
    at Object.defaultResolve (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\dist-raw\node-esm-resolve-impl
ementation.js:929:11)
    at C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\src\esm.ts:228:33
    at entrypointFallback (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\src\esm.ts:179:34)
    at resolve (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\src\esm.ts:227:12)
    at ESMLoader.resolve (node:internal/modules/esm/loader:530:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:251:18)
    at ESMLoader.import (node:internal/modules/esm/loader:332:22)
    at node:internal/modules/run_main:54:28

It builds the (clearly invalid) path:

'/C:/Users/User/Desktop/foo/file:/C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/pr
ev/scripts/prev.ts'

for some reason.

Can't tell if it's something I'm doing or something ts-node is doing just yet.


And depending on whether or not I preface the file argument with file:///, I am revisited by my old friend:

$ node --experimental-specifier-resolution=node --loader=file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/ts-node/esm.mjs C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/prev/scripts/prev.ts
(node:15816) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:15816) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
(node:15816) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
TypeError [ERR_INVALID_MODULE_SPECIFIER]: Invalid module "file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/pr
ev/scripts/prev.ts"
    at new NodeError (node:internal/errors:371:5)
    at ESMLoader.load (node:internal/modules/esm/loader:380:13)
    at async ESMLoader.moduleProvider (node:internal/modules/esm/loader:280:47)
    at async link (node:internal/modules/esm/module_job:70:21) {
  code: 'ERR_INVALID_MODULE_SPECIFIER'
}

@cspotcode
Copy link
Collaborator

To double-check, are you using the absolute latest versions of node and ts-node? This stuff is unstable in node and we've recently published new versions of ts-node, so I want to be sure we don't accidentally waste effort debugging against old versions.

@brianjenkins94
Copy link
Author

brianjenkins94 commented Mar 9, 2022

My Windows box:

$ npx ts-node -vv
ts-node v10.7.0
node v17.5.0
compiler v4.6.2

I'll upgrade node to v17.6.0.


Different error on Windows (Node.js v17.6.0):

npx swallowing the error:

$ npx brianjenkins94/prev
Need to install the following packages:
  github:brianjenkins94/prev
Ok to proceed? (y) y

> node --experimental-specifier-resolution=node --loader=file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/ts-node/esm.mjs file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/prev/scripts/prev.ts

Copying and pasting what it was supposed to spawn:

$ node --experimental-specifier-resolution=node --loader=file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/ts-node/esm.mjs file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/prev/scripts/prev.ts
(node:25528) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\dist-raw\node-esm-resolve-implementation.js:369
    throw new ERR_MODULE_NOT_FOUND(
          ^
CustomError: Cannot find module '/C:/Users/User/Desktop/foo/file:/C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/prev/scripts/prev.ts' imported from C:\Users\User\Desktop\foo\
    at finalizeResolution (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\dist-raw\node-esm-resolve-implementation.js:369:11)
    at moduleResolve (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\dist-raw\node-esm-resolve-implementation.js:818:10)
    at Object.defaultResolve (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\dist-raw\node-esm-resolve-implementation.js:929:11)
    at C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\src\esm.ts:228:33
    at entrypointFallback (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\src\esm.ts:179:34)
    at resolve (C:\Users\User\AppData\Local\npm-cache\_npx\59469f58d9fd3e53\node_modules\ts-node\src\esm.ts:227:12)
    at ESMLoader.resolve (node:internal/modules/esm/loader:536:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:250:18)
    at ESMLoader.import (node:internal/modules/esm/loader:336:22)
    at node:internal/modules/run_main:54:28

Removing the leading file:/// from the file argument (which previously yielded ERR_INVALID_MODULE_SPECIFIER):

$ node --experimental-specifier-resolution=node --loader=file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/ts-node/esm.mjs C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/prev/scripts/prev.ts
(node:23812) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:23812) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
(node:23812) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
RangeError [ERR_UNKNOWN_MODULE_FORMAT]: Unknown module format: null for URL file:///C:/Users/User/AppData/Local/npm-cache/_npx/59469f58d9fd3e53/node_modules/prev/scripts/prev.ts
    at new NodeError (node:internal/errors:372:5)
    at ESMLoader.load (node:internal/modules/esm/loader:384:13)
    at async ESMLoader.moduleProvider (node:internal/modules/esm/loader:282:11)
    at async link (node:internal/modules/esm/module_job:70:21) {
  code: 'ERR_UNKNOWN_MODULE_FORMAT'
}

@cspotcode
Copy link
Collaborator

Thanks, would you mind copy-pasting this into a new issue? Seems like it's a new error, and this issue is closed, so would be easier to track it fresh and go from there.

@brianjenkins94
Copy link
Author

Yes, will do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants