[Bug] Yarn plug and play does not work with ESM

[dmail]

Describe the bug

I have tried yarn plug and play with node 13+.
When running node --require=./.pnp.cjs index.js there is an error saying Error [ERR_REQUIRE_ESM]: Must use import to load ES Module.
Running npm install && node index.js works as expected.

To Reproduce

1. git clone https://github.com/dmail/yarn-pnp.git
2. cd yarn-pnp
3. node --require=./.pnp.cjs index.js

Screenshots

Environment if relevant (please complete the following information):

• OS: MacOS
• Node version: 13.12
• Yarn version: 2.0.0-rc.31

Additional context

Please note how I had to put an empty package.json inside .yarn/releases/ so that .yarn/releases/yarn-berry.js works.

Also your documentation asks to run yarn set version berry but it fails saying Invalid version descriptor "berry".

.

Am I doing something wrong ? Is it expected that plug and play is not yet compatible with node 13+ ?

[dmail]

According to #1150 (comment) it seems expected. PnP is not yet compatible with es modules.

[DaneTheory]

Could you explain the reason for requiring .pnp.js/.pnp.cjs be added to node process at runtime? .pnp.js isn't meant to act as a module even though it's now written as a module.exports. It provides an optimized way to map deps across projects that v1 did not in any way come close to accomplishing. In fact, if you log out the node process object during the pnp lifecycle you can already see .pnp.js is in fact already included as a runtime requirement.

[dmail]

I don't understand what you want to know.

To my understanding plug and play exists to avoid creating the whole node_modules immediately. Instead they are dynamically unzipped from an archive. To achieve this yarn override require to unzip a file when it's required.

Knowing this I think it's mandatory to execute pnp.js that will override node require mecanism before actually running any other code. Right?

[DaneTheory]

I think I understand what you’re asking, but please let me know if not. You’d like to enable specific, native features within the context of the active node process at runtime that are not enabled already by default (i.e, passing flags as arguments to node like ‘—experimental-modules’, etc.). To further clarify, you are not asking how to require/run external modules at runtime (i.e. passing ‘—require esm —require dotenv/config’ to ‘node’ so as to not have to manually enable these features by including them first before anything else in your app runs). Is that correct? If it is, fortunately the solution is made very simple particularly with ‘node v13.x’. Provide the runtime features you’d like enabled as a string to the environment variable ‘NODE_OPTIONS’. Run your code using ‘yarn node /path/to/script.js’ as recommended in the docs. When run, you will then see the features enabled. Yes, using Plug’N’Play does result in the contents of ‘node_modules’ from being included within a project in the traditional way they used to; where each module and any dependencies they have are directly downloaded and included as directories within ‘node_modules’ as paths that ‘node’ then is able to resolve natively. But that’s the bi-product of Plug’N’Play, not why it exists. It exists because the traditional way ‘node’ module resolves are handled is not super performant, results in heavily bloated project size, and is limited in any approach by which a developer could share compartmentalized code throughout the scope of a project (i.e. what workspaces are all about). Plug’N’Play abstracts module resolution away from ‘node’ completely, providing as an alternative instead a subset of extendable features via custom fetchers, resolvers, etc. all available with the context provided via ‘Yarn v2’. All that said, the ‘.pnp.js’ file is generated automatically during yarn’s execution lifecycle. It isn’t an actual module, it’s a map which describes how/when/what/where/why all the packages within your project, their dependencies, and their dependencies, all the way down the line need to look like for ‘node’ to then properly resolve and thus making them “requirable”. If you want to include external modules directly to ‘node’ at runtime as project dependencies, not just devDependencies, and have them configured and wired up ready to run at the start of your project initialization, there’s a way to go about doing that as well. Rather than explain it all out here, check out the @yarnpkg repo itself to see how to do this. In short, it’s doable by changing the ‘.yarnrc.yml’ variable ‘yarnPath’ value to a custom path which offers a script file that essentially acts as a go between for yarn to be run from. That’s one way. Another would be to follow the instructions provided in the ‘pnpapi’ docs. This approach ieffects how the final ‘.pnp.js’ file is ultimately generated in a way that seems to be what would be considered “best practice”. Hope this helps. Cheers!

[dmail]

I want to run a JavaScript file having a dependency where both are written import/export syntax.
And I want to use node 13+ because it support package written using import/export.

index.js

import { value } from "whatever";

console.log(value);

deps/whatever/index.js

export const value = "hello world";

Check https://github.com/dmail/yarn-pnp to see the whole file structure.

After running yarn install in this context whatever package is inside a zip.

• node index.js would throw module not found.
• node --require=./.pnp.cjs index.js throw must use import to load ES Module
• yarn node index.js throw must use import to load ES Module too

[DaneTheory]

@dmail and @larixer
I've created a repo based on the demo of the issue you mentioned. The README.md is purposefully over-verbose for anyone else who might run into this in the future. The actual solution is really very simple. Check out the repo here:

https://github.com/DaneTheory/yarn-pnp-with-esm

Let me know if you've got any further questions. I'd be happy to create a plugin that handles automating everything if that's something the community is doing here. The solution doesn't exactly fit into what the yarn-compat plugin is doing under the hood. Cheers!

[dmail]

I like having that sort of demo repository it helps a lot, thank you very much.

If I understand correctly the final command looks like this

yarn node --require esm ./file.js

And does the following:

• yarn node: Injects yarn plug and play module resolution into node (using pnp.js).
• node --require esm: Injects esm conversion into node.
  Node CommonJS module resolution was already modified by yarn node but esm seems aware of yarn pnp as shown by hook/pnp.js.
• ./file.js: Execute file.js. Before actually executing file.js, it is first converted to CommonJS format by esm thanks to node --require esm. Also require will be handled by yarn thanks to yarn node

I am attracted by yarn plug and play especially zero install philosophy. I am ready to use yarn node path/to/file.js to run code and benefit from it.
However relying on esm conversion disturbs me a bit. Also having to use CommonJS format. I have to test it on actual projects and see how it goes.

That would be great to see pnp.js hook into node experimental loader. I can understand yarn wants to wait for loader to stabilize first.

[yarnbot]

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolution faster). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label.

[paul-soporan]

Tracked in #638.

[r1y4h]

@dmail and @larixer
I've created a repo based on the demo of the issue you mentioned. The README.md is purposefully over-verbose for anyone else who might run into this in the future. The actual solution is really very simple. Check out the repo here:

https://github.com/DaneTheory/yarn-pnp-with-esm

Let me know if you've got any further questions. I'd be happy to create a plugin that handles automating everything if that's something the community is doing here. The solution doesn't exactly fit into what the yarn-compat plugin is doing under the hood. Cheers!

@DaneTheory
Will this also work for dotenv? I am currently using yarn 2 with pnp enabled. Ofcourse I can no longer use this:

node -r dotenv/config your_script.js dotenv_config_path=/custom/path/to/.env

I already spent hours trying to figure out. I haven't tried your suggestion yet. We use dotenv in our build scripts to switch between env files.

[hegelstad]

@r1y4h hey, did you figure out how to use an expression like that with Yarn 2?

[r1y4h]

Use env-cmd instead

https://create-react-app.dev/docs/deployment/#customizing-environment-variables-for-arbitrary-build-environments

[merceyz]

Doesn't either of these work for you?

yarn node -r dotenv/config your_script.js dotenv_config_path=/custom/path/to/.env
node -r ./.pnp.cjs -r dotenv/config your_script.js dotenv_config_path=/custom/path/to/.env

[r1y4h]

@merceyz

the first script using "yarn node" did not work for me:

yarn node -r dotenv/config react-scripts build dotenv_config_path=/custom/path/to/.env

node -r ./.pnp.cjs -r is the same as yarn node right?

I used env-cmd instead

env-cmd -f .env.something yarn run build

[merceyz]

did not work for me:

That tells me nothing, what isn't working? Does it throw?

node -r ./.pnp.cjs -r is the same as yarn node right?

They aim to solve the same thing but yarn node uses the NODE_OPTIONS env variable instead of a CLI flag

[rofrol]

This works:

% mkdir node-yarnpnp-esm && cd $_
% yarn set version latest
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.2.0/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.2.0.cjs
➤ YN0000: Done in 1s 467ms
% yarn init
{
  name: 'node-yarnpnp-esm',
  packageManager: 'yarn@3.2.0'
}
% yarn config set pnpEnableEsmLoader true
➤ YN0000: Successfully set pnpEnableEsmLoader to true
% cat .yarnrc.yml
pnpEnableEsmLoader: true
% yarn add rxjs
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 1s 30ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ rxjs@npm:7.5.5 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ tslib@npm:2.3.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0000: └ Completed
➤ YN0000: Done with warnings in 1s 110ms
% ls -1a
.
..
.editorconfig
.git
.gitignore
.pnp.cjs
.pnp.loader.mjs
.yarn
.yarnrc.yml
README.md
index.mjs
package.json
yarn.lock
% cat <<EOT > index.mjs
import { of } from "rxjs"
import { map } from "rxjs/operators"

let stream$ = of(1,2,3).pipe(
  map(x => x + "!!!")
)

stream$.subscribe((val) => {
  console.log(val) // 1!!! 2!!! 3!!!
})
EOT
% yarn node index.mjs
1!!!
2!!!
3!!!