V3 breaks webpack and jest

[heath-freenome]

Please document how to get things working with jest and webpack like formatJS does for react-intl: https://formatjs.io/docs/react-intl#jest

Right now my webpack blows up with:

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: <path>/node_modules/node-fetch/src/index.js
require() of ES modules is not supported.
require() of <path>/node_modules/node-fetch/src/index.js from <path>/server.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from <path>/node_modules/node-fetch/package.json.

    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1080:13)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.node-fetch (<path>/external "node-fetch":1:1)
    at __webpack_require__ (<path>/build/webpack:/app/webpack/bootstrap:24:1)
    at fn (<path>/build/webpack:/app/webpack/runtime/hot module replacement:61:1)
    at Module../src/server/createNodeWretch.js (<path>/build/server.js:7943:68)
    at __webpack_require__ (<path>/build/webpack:/app/webpack/bootstrap:24:1)

and Jest with:

  ● Test suite failed to run

    Jest encountered an unexpected token

    Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.

    Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.

    By default "node_modules" folder is ignored by transformers.

    Here's what you can do:
     • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/configuration
    For information about custom transformations, see:
    https://jestjs.io/docs/code-transformation

    Details:

    <path>/node_modules/node-fetch/src/index.js:9
    import http from 'http';
    ^^^^^^

    SyntaxError: Cannot use import statement outside a module

      2 | import url from 'url';
      3 | import FormData from 'form-data';
    > 4 | import fetch from 'node-fetch';
        | ^
      5 |

Expected behavior
I've have always used the import syntax as described in the migration guide... but the guide is incomplete since webpack/jest now blow up when I try upgrading

Your Environment

software	version
node-fetch	3.0.0
node	14.16.0
npm	6.14.11
Operating System	macOS big sur

[heath-freenome]

Hmmm, just for the heck of it I tried something similar to what react-intl suggests and it doesn't work... Maybe something is missing in how node-fetch is defined in the package.json? or perhaps it is missing the index.js file in the main directory like react-intl has?

[loynoir]

Update: A minimal jest.config.ts example tested worked with node-fetch.

• Can choose experimental, full ESM support
• Or choose stable, partial but enough ESM support
• Tested with pnpm, yarn, npm, yarn@berry

https://github.com/loynoir/example-jest-config-esm/blob/master/jest.config.ts

Good luck, bro! 🎉

[loynoir]

@RedGuy12 If you check the source code of jest, you will find out the core problem here is not require/import.
Jest is running under vm module, and Node.js vm module support ESM is experimental

--experimental-vm-modules

[LinusU]

Documentation for getting this to work with Jest is here:

https://jestjs.io/docs/ecmascript-modules

[loynoir]

@LinusU

Jest

Note that due to its experimental nature there are many bugs and missing features in Jest's implementation, both known and unknown. You should check out the tracking issue and the label on the issue tracker for the latest status.

Node.js

--experimental-vm-modules

We both want ESM world, but reality is ecosystem is not ready yet.

Instead of "just use all the experimental features", which seems like an uncontrolable black box,

Can't we just dual-modulize node-fetch?

[jimmywarting]

Can't we just dual-modulize node-fetch?

this are being discussed here: #1227 (read everything before jumping in to discussions)

[LinusU]

We both want ESM world, but reality is ecosystem is not ready yet.

I'm not sure about that, Jest is just one of many test runners. I don't think that just because its support is experimental, the whole ecosystem isn't ready.

Can't we just dual-modulize node-fetch?

I think a better approach is to stick with node-fetch 2.x if you are having troubles

[loynoir]

@jimmywarting

this are being discussed here

I know. But here is a specific real-world scenario, just jest, not some abstraction.

@LinusU

I'm not sure about that, Jest is just one of many test runners. I don't think that just because its support is experimental, the whole ecosystem isn't ready.

Node.js mark as --experimental-vm-modules, jest official document states many known and unknown bugs.
I don't think is not very convincing both Node.js and jest official state them as experimental.

---

• Just use all the experimental features if you want node-fetch@3
• If you don't, just don't update, and stay on node-fetch@2

Emm, seems very frustrating.

[LinusU]

Node.js mark as --experimental-vm-modules, jest official document states many known and unknown bugs.
I don't think is not very convincing both Node.js and jest official state them as experimental.

Yes, but I don't think that vm-modules isn't often used that much at all? At least I've personally never seen it used outside of Jest...

• Just use all the experimental features if you want node-fetch@3
• If you don't, just don't update, and stay on node-fetch@2

Emm, seems very frustrating.

Why does this seem frustrating?

[loynoir]

Yes, but I don't think that vm-modules isn't often used that much at all?

Here is the problem. Please check jest source code.
All jest running under vm.

[loynoir]

• Just use all the experimental features if you want node-fetch@3
• If you don't, just don't update, and stay on node-fetch@2

Why does this seem frustrating?

Because there is a proper solution for current time point, dual-module.
But I know you against dual-module...

[LinusU]

Yes, but I don't think that vm-modules isn't often used that much at all?

Here is the problem. Please check jest source code.
All jest running under vm.

Yes I know that Jest is using vm-modules. What I meant was that the majority of the ecosystem is not using vm-modules, and thus holding back the entire ecosystem just because Jest does it feels a bit strange to me 🤔

• Just use all the experimental features if you want node-fetch@3
• If you don't, just don't update, and stay on node-fetch@2

Why does this seem frustrating?

Because there is a proper solution for current time point, dual-module.
But I know you against dual-module...

Well, I am personally against dual-module, but that is because there are clear drawbacks with it which is discussed in the larger thread.

I think it's better if we concentrate specifically on Jest and WebPack compatibility (without going away from ESM-only) in this issue. In fact, it might be better to split it up into two issues: one for Jest, and one for WebPack.

I've used ESM-only modules in WebPack before so I'm not entirely sure what the problems are there, but I believe that there are workarounds/fixes available...

[Thomasan1999]

I'm using ESM and Rollup in my project but Jest still had the same problem with loading the package. Here's what worked for me:

jest.config.ts:

export default {
  preset: 'ts-jest/presets/js-with-ts-esm',
  transformIgnorePatterns: [
      'node_modules/(?!(fetch-blob|node-fetch)/)'
  ]
}

tsconfig.json:

{
  "compilerOptions": {             
    "allowJs": true,                       
    "esModuleInterop": true
  }
}

Credit goes to: https://stackoverflow.com/a/66480267/8746088

[heath-freenome]

@Thomasan1999... That did it for me!!! it was fetch-blob that was the savior...

[heath-freenome]

@Thomasan1999... That did it for me!!! it was fetch-blob that was the savior...

False alarm... I forgot to npm install with node-fetch v3 :(

[loynoir]

@heath-freenome Not false alarm. I saw and tested it! 😄
@Thomasan1999 Awesome! As long as node-fetch not using hard core ESM, like top level await, now I can get rid of experimental flag. 😘

PS: I use pnpm, adjust to 'node_modules/(?!node-fetch|fetch-blob)' if you are not.

[tempDir] $ pnpm i -D jest ts-jest ts-node typescript node-fetch && jest && tail -n +1 jest.config.ts tsconfig.json esmDeps.spec.ts

==> jest.config.ts <==
export default {
  preset: 'ts-jest/presets/js-with-ts-esm',
  transformIgnorePatterns: [
      'node_modules/\\.pnpm/(?!node-fetch|fetch-blob)',
  ]
}

==> tsconfig.json <==
{
  "compilerOptions": {             
    "allowJs": true,                       
    "esModuleInterop": true
  }
}

==> esmDeps.spec.ts <==
import nf from 'node-fetch'

describe("esmDeps", () => {
  it("esmDeps", () => {
    console.log(nf)
  });
});

[heath-freenome]

@heath-freenome Not false alarm. I saw and tested it! 😄
@Thomasan1999 Awesome! As long as node-fetch not using hard core ESM, like top level await, now I can get rid of experimental flag. 😘

I'm not using typescript in my project (yet)... And there still isn't a workaround for my webpack issues

[loynoir]

Since @Thomasan1999 got a solution without any experimental features!

I currently have no real-world problem with node-fetch. 😄

FYI @jimmywarting @LinusU, not hard core ESM yet, like top-level await will break this solution. (I also tested it)

[loynoir]

@heath-freenome
The core tech here is using transformer, exclude all esm-only package from transformIgnorePatterns, which means

• let transformer convert ALL ESM-only package to CJS on-the-fly of jest

Maybe you need babel-jest or jest-babel something, let babel convert those edge packages.

FYI, ESM-only package node-fetch depends on ESM-only fetch-blob.
So, you need to at least node-fetch and fetch-blob, add adjust according to error msg.

Good luck!

[xbreaker]

@heath-freenome

For Node JS project that's worked for me, may be you can do the same for webpack:

// jest.config.js
export default {
  transform: {
    '^.+\\.js$': 'babel-jest',
  },
  transformIgnorePatterns: [
      'node_modules/(?!(fetch-blob|node-fetch)/)'
  ]
}

// babel.config.cjs
module.exports = {
  presets: [
    [
      '@babel/preset-env',
      {
        targets: {
          node: 'current'
        }
      }
    ]
  ],
  plugins: ['babel-plugin-transform-es2015-modules-commonjs', 'babel-plugin-transform-import-meta']
}

[ChromeQ]

Thanks @xbreaker that worked great until node-fetch updated to v3.1.0 now.
The dependencies changed (https://github.com/node-fetch/node-fetch/blob/v3.1.0/package.json#L65) so the transformIgnorePatterns needs updating to include them

transformIgnorePatterns: [
    'node_modules/(?!(data-uri-to-buffer|formdata-polyfill|fetch-blob|node-fetch)/)',
]

There are other issues with node-fetch@3.1 which causes problems with jest, specifically the usage of node:prefixes. I have added another workaround details here: #1367 (comment)

Hope this helps somebody