Can't import a hybrid ESM + CommonJS package from CommonJS using Node16 resolution

[ajwootto]

Bug Report

🔎 Search Terms

esm commonjs node16 hybrid resolution

🕗 Version & Regression Information

• I was unable to test this on prior versions because it concerns an option added in Typescript 5

⏯ Playground Link

CodeSandbox

I'm seeing this behaviour locally with Typescript 5.1, but Codesandbox doesn't seem to have that version as an option. Might need to download locally to see the problem.

💻 Code

See the Codesandbox

🙁 Actual behavior

The line in test.js that imports from the hybrid module gives this compiler error:

The current file is a CommonJS module whose imports will produce 'require' calls; however, the referenced file is an ECMAScript module and cannot be imported with 'require'. Consider writing a dynamic 'import("./hybrid")' call instead.

(behaviour on the Codesandbox is different, I can't set the version to Typescript 5.1, might need to download locally to see the problem)

🙂 Expected behavior

In this setup I have a module hybrid which declares itself as an ESModule (type: module) in its package.json. However, it also exports commonjs javascript and types, using the approach described in this documention:

https://www.typescriptlang.org/docs/handbook/esm-node.html#commonjs-interop

The module and moduleResolution settings of the outer .ts file which imports this package is node16. I expected based on that documentation that Typescript would be able to resolve hybrid as CommonJS because of the package.json setup, but it seems to insist that it's an ESModule. If I change the "types" field of the package.json to the .d.cts file, then resolution works, but I imagine it now wouldn't work when importing from an ESModule. The documentation also specifies that it should be .d.ts, and also that it's only there for the benefit of older versions of Typescript. With that in mind, I also tried removing the line entirely and that broke resolution again.

Not sure if this is a bug, a documentation issue, or something I'm doing wrong with the setup.

[fatcerberus]

This is almost always a misconfiguration on the part of the package you're importing (usually using the same .d.ts file for both ESM and CommonJS exports). See for example #50466.

Also see https://github.com/arethetypeswrong/arethetypeswrong.github.io

[ajwootto]

In this case the package that I'm importing is defined in the same sandbox, and definitely has a separate .d.cts file

[RyanCavanaugh]

For putative module resolution bugs, we'll need an exact breakdown of why you think something is provably wrong. We've ironed out basically everything here and the configuration space is extremely large (thus the odds of a misconfiguration are very high).

See also (WIP) module docs https://gist.github.com/andrewbranch/79f872a8b9f0507c9c5f2641cfb3efa6

[ajwootto]

@RyanCavanaugh happy to provide any more info that's required.

I actually just read those docs as well.

Basically what I'm seeing is an inability to import a hybrid module with a package.json specified (as far as I can tell) identically to what's specified in the current docs:

// package.json
{
    "name": "my-package",
    "type": "module",
    "exports": {
        ".": {
            // Entry-point for `import "my-package"` in ESM
            "import": {
                // Where TypeScript will look.
                "types": "./types/esm/index.d.ts",
                // Where Node.js will look.
                "default": "./esm/index.js"
            },
            // Entry-point for `require("my-package")` in CJS
            "require": {
                // Where TypeScript will look.
                "types": "./types/commonjs/index.d.cts",
                // Where Node.js will look.
                "default": "./commonjs/index.cjs"
            },
        }
    },
    // Fall-back for older versions of TypeScript
    "types": "./types/index.d.ts",
    // CJS fall-back for older versions of Node.js
    "main": "./commonjs/index.cjs"
}

https://www.typescriptlang.org/docs/handbook/esm-node.html#packagejson-exports-imports-and-self-referencing

When consuming this package from CommonJS code using module and moduleResolution of node16, I receive an error that the module being imported is an ESModule. However, my understanding is that in Node, this would be interpreted as CommonJS when being imported using a require from CommonJS, since it defines both CommonJS and ESM exports. Thus I would not expect to receive an error in this case.

[fatcerberus]

When consuming this package

🐽

[ajwootto]

Update: I think I've figured out some more details of what's going on here.

In my original example, the package hybrid that is being required is actually "nested" inside the folder, and is being required with a relative import like ./hybrid. That was an attempt by me to create a simplified version of the issue I'm seeing, which is actually taking place inside a monorepo with packages depending on each other.

Locally if I take my example and move the hybrid package inside of a node_modules/ folder while changing the import to hybrid, then the types resolve correctly. I'm not sure if this is intended behaviour or not, but it seems like Node will accept that the files in the original structure are CommonJS when required and execute correctly.

Edit: just to clarify the above a bit more, this works:

node_modules/
  hybrid/
    index.js
    index.cjs
    index.d.ts
    index.d.cts
    package.json
app.ts
package.json
tsconfig.json

with app.ts importing hybrid like so:

// app.ts
import { test } from 'hybrid'

but this doesn't:

hybrid/
  index.js
  index.cjs
  index.d.ts
  index.d.cts
  package.json
app.ts
package.json
tsconfig.json

// app.ts
import { test } from './hybrid'

However, a transpiled commonjs version of app.ts works in both cases when running in Node

[RyanCavanaugh]

If you have

  "main": "./index.cjs",

then the correct corresponding entry in package.json is

  "types": "./index.d.cts",

which works as expected.

[ajwootto]

@RyanCavanaugh gotcha, in that case does this example in the docs need to be updated?

[RyanCavanaugh]

Indeed all of the docs need updating 😅

[ajwootto]

@RyanCavanaugh I opened a PR here to make that change:
microsoft/TypeScript-Website#2865

however I realized in doing so, that if the purpose of this field is for "old" versions of Typescript, wouldn't it fail if that version of Typescript does not recognize .cts files?

[bradennapier]

I am now having issues with many many packages that seem to have this configured wrong. When using Node18 tsconfig bsae with 5.2.0 beta which requires you change moduleResolution from node to node16 it creates hundreds of errors across packages and codebases such as

src/packages/server/plugins/registerGlobalPlugins.ts:16:32 - error TS2834: Relative import paths need explicit file extensions in EcmaScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Consider adding an extension to the import path.

and

error TS7016: Could not find a declaration file for module 'ccxt'. '/Users/Shared/Development/projects/@auroradao/project9/node_modules/ccxt/dist/ccxt.cjs' implicitly has an 'any' type.
  Try `npm i --save-dev @types/ccxt` if it exists or add a new declaration (.d.ts) file containing `declare module 'ccxt';`

It seems like the impact could be significantly breaking many (probably misconfigured) npm packages.

So far I have run into this with

• nanoid
• ccxt
• all local scripts using import('./blah') instead of import('./blah.js') (which is annoyance but clearly fixable in code at least without requiring many libraries to fix their code -- which they are fighting tsconfig-bases, nanoid

This is more of a gotcha and probably should have notes in the release notes since obviously we can just not use the tsconfig-base or change the module/moduleResolution out of Node16... although it feels sad to not use the tsconfig-base for Node18 ;-)

[andrewbranch]

however I realized in doing so, that if the purpose of this field is for "old" versions of Typescript, wouldn't it fail if that version of Typescript does not recognize .cts files?

Indeed, the example on the website is correct/intentional for this reason.

The website example assumes that the package described by the package.json will be loaded from inside somebody’s node_modules by a non-relative import, e.g. import "hybrid". Your CodeSandbox, by contrast, loads your example by relative import, i.e. import { test } from "./hybrid". Node’s resolution algorithm treats these two cases super differently! Specifically, "exports" are never looked up for relative imports, so in your test, the main and types fields are getting used by Node and TypeScript, respectively, rather than exports.

The point of the example on the website is that main and types will never be used by modern versions of Node and modern versions of TypeScript. Instead, the assumption is that exports will always be used. If an ancient version of Node is used, main should direct it to a .cjs file since ESM won’t be supported in that version, but Node doesn‘t care what file extension a CommonJS script has. If an ancient version of TypeScript is used, types needs to point to a .d.ts file since a .d.cts file won’t be readable. Technically, this mismatch of extensions would be incorrect if a modern version of TypeScript were to perform this resolution, but we’re relying on the fact that modern versions of TypeScript should never see that resolution since exports “shadows” it. The relative import breaks that assumption, but that’s an edge case that I think library authors can usually ignore.

[ajwootto]

@andrewbranch that all makes sense, though I find it unfortunate in Node's case that these two cases should be different.

Thanks for writing those new docs by the way, it's the clearest explanation I've found about this whole situation. If it isn't already called out in there, I think this difference in how package.json files are interpreted between relative and node_modules resolutions would be good to add.

It may also be helpful to describe that this is the case being hit in the error message.

The real issue I ran into here is when attempting to use newer versions of Typescript in an Nx monorepo.
They use an approach that relies on a list of tsconfig path entries to "link" different packages together in the repo during development. It then allows for an "incremental" build system where building a package first builds all its dependencies and outputs them to a dist/ directory. Once a dependency is built, the package that needs it gets built and the tsconfig aliases are "swapped" to point to the output of the dependency in the dist/ directory. At that time, the build throws type errors because the output of the dependent package in dist/ is a hybrid module, but as you've said it's not being interpreted as such because it's a relative import.

However, each package is meant to be published to NPM, so when compiled and published that path which was previously an aliased "path" now becomes a literal import for the same-named package. Eg. there's a path alias for @my-org/package-1 that points to its source files, but those files end up being published in a package named the same, so that imports between those dependent packages work when installed.

Even as I'm writing this I realize it's a gigantic edge case, so I'll probably need to go take it up with Nx.

[andrewbranch]

Yeah, paths really confuses the situation in cases like that. It’s one reason why I highly recommend workspace-style node_modules symlinks over paths when simulating installed packages in monorepo local development.