Pnp workspaces aren't transpiled (yarn berry) #75

goszczynskip · 2020-03-20T16:26:31Z

I HAVE READ THE FAQ AND MY PROBLEM WAS NOT DESCRIBED THERE

Are you trying to transpile a local package or an npm package?
Local package

Describe the bug
I'm trying to make transpilation work with yarn berry. I have multiple workspaces in my repo and listing them as an configuration parameter doesn't trigger transpilation.

My config:

const withTM = require("next-transpile-modules")(["@myapp/feature-x", "@myapp/foo"]);

module.exports = withTM();

Import in one of pages:

import { FeatureX } from "@myapp/feature-x"

Error in console:

...
[ error ] ../features/x/index.tsx 4:26
Module parse failed: Unexpected token (4:26)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| import { useFoo } from "@myapp/foo";
|
> export const FeatureX: FC = () => {
|   const foo = useFoo();
|   return (
[ event ] build page: /next/dist/pages/_error
...

To Reproduce

Create simple nextjs mono repo with yarn berry pnp workspaces.
Require something from other workspace

Expected behavior
It should be transpiled

Setup

Next.js version: 9.3.1
next-transpile-modules version: 3.1.0
Node.js version: 13.2.0
npm/yarn version: 2.0.0-rc.30
Operating System: Windows 10

The text was updated successfully, but these errors were encountered:

martpie · 2020-03-20T16:31:38Z

Hello 👋

Create simple nextjs mono repo with yarn berry pnp workspaces.

Can you provide one? I don't have experience with pnp.

goszczynskip · 2020-03-20T18:13:49Z

Sure. I've created one. Follow readme to get the error. https://github.com/goszczynskip/next-transpile-modules-bug-75

It looks that yarn changes package paths underneath to local ones.

I've tried to add all workspaces as aliases in webpack config but this forces me to include them in package.json of application. I'd rather avoid it because it pollutes package.json's dependencies.

nkalinov · 2020-04-27T08:29:08Z

Any updates here? Known workaround?

martpie · 2020-04-27T10:23:57Z

There is a PR opened, but its creator needs to update it in order for the tests to be ran against it. Also, integrations tests would be welcome.

martpie · 2020-04-27T10:25:02Z

It's not a lot of work left, but it needs to be done and I do not have time to invest on a setup I don't know too well sorry. So a branch fork + new PR is more than welcome!

goszczynskip · 2020-04-27T12:50:58Z

I'll try to update it today.

EDIT:
PR updated

martpie · 2020-04-28T08:50:38Z

Should be fixed in 3.3.0 🚀

th0th · 2020-07-19T17:29:45Z

I am MacOS and still having this issue. I cloned @goszczynskip 's repo and updated next-transpile-modules to 3.3.0 but no luck. Would it help if I created an example repository?

andrewmclagan · 2020-08-31T10:13:30Z

@martpie Also attempted to use this in our monorepo with Yarn 2 pnp. Upgraded to version ^3.3.0 to no avail. Then attempted to run within the demo repo above with the same version and failed to transpile.

andrewmclagan · 2020-09-01T00:51:16Z

Also attempted to transpile a simple repo on the 4.x.x branch to no avail.

I have little idea of the inner workings of WebPack, so please ignore this comment if it is ignorant..!

It seems that this package hijacks the WebPack node modules resolution process by using a custom regex pattern based upon the modules a developer has passed into the package config? From my understanding webpack module resolution is a superset of the node modules resolution api called "enhanced-resolve".

The reason this package does not support yarn pnp is that it's looking for a node_modules dir that does not exist. Using the custom regex. Essentially performing its own form of module resolution.

Here comes my ignorance...

Would it not be better to utilise WebPack's built in "enhanced-resolve" algorithm that includes PnP support to resolve the modules? It can be imported as a package: https://github.com/webpack/enhanced-resolve.

const fs = require("fs");
const { CachedInputFileSystem, ResolverFactory } = require("enhanced-resolve");

// create a resolver
const nextModulesResolver = ResolverFactory.createResolver({ ...options });

nextModulesResolver.resolve(...);

PnP compatibility table: https://yarnpkg.com/features/pnp#compatibility-table

martpie · 2020-09-01T08:11:01Z

To anyone facing this issue, I currently unfortunately have no time to work on this before one or two weeks (I also never used yarn 2), but I'd be more than willing to accept PRs 👍

The first step to take is to create a failing test, so we can check easily if it's fixed or not when working on it.

next-transpile-modules/src/__tests__/integrations.test.js

Lines 2 to 4 in e824eba

    
           ['npm', 3500], // name of the test, port 
        
           ['yarn-worspaces', 3501], 
        
           ['webpack-5', 3502]

https://github.com/martpie/next-transpile-modules/blob/master/jest-puppeteer.config.js
https://github.com/martpie/next-transpile-modules/tree/master/src/__tests__/__apps__

I also plan to refactor the whole library at some point in the next months as it grew a bit out of hand, it's hard to test, and complex regexes are more confusing than helping.

Hope it helps 👍

andrewmclagan · 2020-09-01T09:28:19Z

Amazing! I also dont have time in the next couple of sprints. Although after that can contribute for sure.

andrewmclagan · 2020-09-01T23:12:07Z

@martpie Here is some context regarding "enhance-resolve" and PnP. This is a conversation between Yarn guys and Webpack guys: webpack/enhanced-resolve#162

If I could be so bold as to make some suggestions... I believe the best approach would be to create a canary branch for a rewrite of this package. The rewrite would be focused on dropping regex for integrating native webpack module resolution api. The version would be only supporting webpack 5 moving forward as the PnP resolution is only supported in Webpack 5. Legacy users could simply use previous versions of this package. That way development can remain focused.

Key take aways would be:

Support webpack 5 only
Migrate to native webpack module resolution
Get Yarn PnP for free 🎉

andrewmclagan · 2020-09-03T06:24:01Z

For those wanting a solution asap I created a custom next webpack plugin that does the following and it works

All our workspace pakages start with @pkg for example @pkg/graphql or @pkg/forms

      config.module.rules.push({
        test: /\.(tsx|ts|js|mjs|jsx)$/,
        include: (input) => {
          if (input.includes('@pkg')) {
            return true;
          }
          return false;
        },
        use: options.defaultLoaders.babel
      });

martpie · 2020-09-03T08:05:05Z

Yes, this should do the trick indeed (hmr won't work unfortunately), be careful that with this solution, if you use input.includes('abdc'), it will also matches package with names like abcdef etc.

andrewmclagan · 2020-09-04T00:33:19Z

yep certainly a hack, not a long term solution.

goszczynskip · 2020-09-15T06:54:28Z

@martpie I've tried to create failing tests for yarn2 in current files layout. Unfortunately I haven't found easy way to make it work (without rewriting tests bootstraping code). Yarn 2 doesn't allow nested packages that aren't part of parent workspaces. We can overcome this limitation by running tests inside of Docker image.

borekb · 2020-09-15T08:26:25Z

Using a specific path seems to work for us:

- withTM(['@scope/lib']
+ withTM(['../lib']

It's uglier but works.

ukarlsson · 2020-10-27T07:09:24Z

I was able to get it working using the following hack that seems more general than the have proposed above by @andrewmclagan

  webpack(config, options) {
    config.module.rules.push({
      test: /\.(tsx|ts|js|mjs|jsx)$/,
      include: input =>
        !input.includes('/.yarn/cache/') &&
        !input.includes('/.yarn/$$virtual/') &&
        !input.includes('/.yarn/unplugged/'),
      use: options.defaultLoaders.babel,
    })
   return config
}

Is this something we could include in next-transpile-modules?

borekb · 2020-10-27T09:56:18Z

@ukarlsson @andrewmclagan is this something you use instead of next-transpile-modules, or together with it?

martpie · 2020-10-27T10:01:20Z

@ukarlsson thank you for diving into it and providing a solution! There is #115 where I try to keep track of all the things I would like to do for next-transpile-modules :)

ukarlsson · 2020-10-27T15:45:47Z

@borekb I use it instead of next-transpile-modules, just putting that directly into next.config.js.

martpie · 2020-10-27T16:01:13Z

Note that hot-reloading will probably not work ;) (that's also one the things next-transpile-modules is trying to solve)

martpie · 2020-10-27T16:02:26Z

Sidenote: It looks like the solution provided by @ukarlsson transpiles ALL the depedencies, which must be quite bad for build performances, so I'd recommend to restrict it to your packages only ;)

ukarlsson · 2020-10-31T14:56:35Z

The idea with the solution I provided above was to exclude the dependencies handled by Yarn, therefore it includes aforementioned .yarn/cache etc :)

borekb · 2020-11-04T08:15:56Z

@martpie do you think that supporting PnP is a relatively large overall change in the source code, or do you think it's a small fix somewhere? I'd be willing to look at it closer but would like to know what I'm subscribing for 😄.

martpie · 2020-11-04T10:10:19Z

Reference for myself:

Basically, node.js resolution is short circuited by compilers resolution systems. In theory, switching to resolvers instead of path matchers should solve the issue.

cf. #132 #115

borekb · 2020-11-04T10:16:57Z

Yep, as you already described above.

In our app, when we use PnP and scoped packages, the error during next build looks like this:

../.yarn/$$virtual/@company-lib-virtual-b5aec39a1d/1/lib/components/Example.tsx 5:0
Module parse failed: The keyword 'interface' is reserved (5:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| import { UrlObject } from 'url';
| 
> interface Props extends LinkProps {
|   prefix?: string;
| }


> Build error occurred
Error: > Build failed because of webpack errors
    at build (/Users/borekb/dev/company/project/packages/.yarn/$$virtual/next-virtual-e93fc6ab86/0/cache/next-npm-9.5.2-504d9b0249-bcd1379ae6.zip/node_modules/next/dist/build/index.js:15:918)

I.e., there are various .yarn/$$virtual paths.

borekb · 2020-11-04T10:17:33Z

36% of Next.js users use it

Wow 😲 that a lot. Congrats I guess?! 😄

martpie · 2020-11-10T13:42:42Z

Hello there! 👋

Could someone test the branch nextjs-10 and tell me if it works fine with Yarn 2?

yarn add git+https://github.com/martpie/next-transpile-modules.git#nextjs-10 should do the trick.

borekb · 2020-11-11T16:21:29Z

@martpie I tried the branch. BTW, this seems to be the correct command, at least for Yarn 2:

yarn add next-transpile-modules@martpie/next-transpile-modules#nextjs-10

# Effectively the same as:
yarn add next-transpile-modules@https://github.com/martpie/next-transpile-modules.git#commit=79375a68b0a21c7eb234cf81048144d8f22491df

It produces this error:

$ next build

> Build error occurred
Error: next-transpile-modules tried to access resolve, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

Required package: resolve (via "resolve")
Required by: next-transpile-modules@https://github.com/martpie/next-transpile-modules.git#commit=79375a68b0a21c7eb234cf81048144d8f22491df (via /Users/borekb/dev/company/project/packages/.yarn/cache/next-transpile-modules-https-998e3c593d-2cef455db2.zip/node_modules/next-transpile-modules/src/)

Require stack:
- /Users/borekb/dev/company/project/packages/.yarn/cache/next-transpile-modules-https-998e3c593d-2cef455db2.zip/node_modules/next-transpile-modules/src/next-transpile-modules.js
- /Users/borekb/dev/company/project/packages/citygolf/next.config.js
- /Users/borekb/dev/company/project/packages/.yarn/$$virtual/next-virtual-e93fc6ab86/0/cache/next-npm-9.5.2-504d9b0249-bcd1379ae6.zip/node_modules/next/dist/next-server/server/config.js
- /Users/borekb/dev/company/project/packages/.yarn/$$virtual/next-virtual-e93fc6ab86/0/cache/next-npm-9.5.2-504d9b0249-bcd1379ae6.zip/node_modules/next/dist/build/index.js
- /Users/borekb/dev/company/project/packages/.yarn/$$virtual/next-virtual-e93fc6ab86/0/cache/next-npm-9.5.2-504d9b0249-bcd1379ae6.zip/node_modules/next/dist/cli/next-build.js
- /Users/borekb/dev/company/project/packages/.yarn/$$virtual/next-virtual-e93fc6ab86/0/cache/next-npm-9.5.2-504d9b0249-bcd1379ae6.zip/node_modules/next/dist/bin/next
    at internalTools_makeError (/Users/borekb/dev/company/project/packages/.pnp.js:29690:34)
    at resolveToUnqualified (/Users/borekb/dev/company/project/packages/.pnp.js:30648:23)
    at resolveRequest (/Users/borekb/dev/company/project/packages/.pnp.js:30740:29)
    at Object.resolveRequest (/Users/borekb/dev/company/project/packages/.pnp.js:30806:26)
    at Function.external_module_.Module._resolveFilename (/Users/borekb/dev/company/project/packages/.pnp.js:29923:34)
    at Function.external_module_.Module._load (/Users/borekb/dev/company/project/packages/.pnp.js:29788:48)
    at Module.require (internal/modules/cjs/loader.js:1025:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/Users/borekb/dev/company/project/packages/.yarn/cache/next-transpile-modules-https-998e3c593d-2cef455db2.zip/node_modules/next-transpile-modules/src/next-transpile-modules.js:2:17)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)

martpie · 2020-11-11T16:55:41Z

Thank you very much 🙌 I've fixed it, could you try again? 😄 (8c9e47d)

borekb · 2020-11-12T08:18:02Z

This time, I get this:

$ yarn build

> Build error occurred
Error: Cannot find module '@company/lib' from '/Users/borekb/dev/company/project/packages/.yarn/cache/next-transpile-modules-https-31f28f89ec-8d20c5cde4.zip/node_modules/next-transpile-modules/src'
    at Function.resolveSync [as sync] (/Users/borekb/dev/company/project/packages/.yarn/cache/resolve-patch-46f4fba2f6-8ad4c7742f.zip/node_modules/resolve/lib/sync.js:90:15)
    at /Users/borekb/dev/company/project/packages/.yarn/cache/next-transpile-modules-https-31f28f89ec-8d20c5cde4.zip/node_modules/next-transpile-modules/src/next-transpile-modules.js:35:32
    at Array.map (<anonymous>)
    at generateResolvedModules (/Users/borekb/dev/company/project/packages/.yarn/cache/next-transpile-modules-https-31f28f89ec-8d20c5cde4.zip/node_modules/next-transpile-modules/src/next-transpile-modules.js:34:6)
    at withTM (/Users/borekb/dev/company/project/packages/.yarn/cache/next-transpile-modules-https-31f28f89ec-8d20c5cde4.zip/node_modules/next-transpile-modules/src/next-transpile-modules.js:61:29)
    at /Users/borekb/dev/company/project/packages/.yarn/cache/next-compose-plugins-npm-2.2.0-daa8cb9862-f27d897f26.zip/node_modules/next-compose-plugins/lib/compose.js:100:23
    at Array.forEach (<anonymous>)
    at composePlugins (/Users/borekb/dev/company/project/packages/.yarn/cache/next-compose-plugins-npm-2.2.0-daa8cb9862-f27d897f26.zip/node_modules/next-compose-plugins/lib/compose.js:77:11)
    at /Users/borekb/dev/company/project/packages/.yarn/cache/next-compose-plugins-npm-2.2.0-daa8cb9862-f27d897f26.zip/node_modules/next-compose-plugins/lib/index.js:22:38
    at normalizeConfig (/Users/borekb/dev/company/project/packages/.yarn/$$virtual/next-virtual-17b8ed4149/0/cache/next-npm-9.5.2-504d9b0249-bcd1379ae6.zip/node_modules/next/dist/next-server/server/config.js:5:1990) {
  code: 'MODULE_NOT_FOUND'
}

Do you have a PnP project you're testing it on? Maybe it's somehow different but I think our Yarn workspaces setup is pretty standard.

martpie · 2020-11-12T09:30:58Z

Thank you very much for helping me testing this.

I will setup a local project in order to debug that. I tried to do test automation, but berry does not like to find package.json up in the tree :(

I'll keep you in touch

borekb · 2020-11-12T09:37:02Z

I'd be happy to prepare a public repo for you but if you're already on it, maybe no need?! Let me know if I can help.

martpie · 2020-11-12T10:50:27Z

Can you confirm that using ntm directly in your project (without using it from yarn add next-transpile-modules) will make everything work fine?

(if you copy paste the following in your project: https://raw.githubusercontent.com/martpie/next-transpile-modules/nextjs-10/src/next-transpile-modules.js)

const withTM = require('./ntm')([/* whatever */]);

goszczynskip · 2020-11-12T12:49:45Z

@martpie As I said earlier creating automatic tests with current architecture isn't possible. I can prepare yarn2 testing but that will require from us to change approach to use docker images for testing. Inside each image I can prepare target env (npm, yarn1, yarn2) and we'll run tests inside containers.

martpie · 2020-11-12T12:52:24Z

@goszczynskip Thank you very much, I just created a small reproduction setup on my laptop so I can quickly prototype a solution.

It seems so far that v5 will support berry fine, but there is just a module resolution issue left, and I am trying to pinpoint the root cause: apparently, when you copy-paste the updated plugin to your codebase, things work fine now, but yarn add ... does not work ;)

so we're almost there!

borekb · 2020-11-12T14:19:17Z

@martpie I did the above but then I get this error again:

Error: Your application tried to access resolve, but it isn't declared in your dependencies; this makes the require call ambiguous and unsound.

Required package: resolve (via "resolve")

(Same as above.)

martpie · 2020-11-12T14:20:56Z

I'm so sorry, I forgot to mention you need to install resolve manually as you are not installing the package from npm 🙈

yarn add resolve

so sorry for all the back and forths

borekb · 2020-11-12T14:28:07Z

No problem 😄. With that, I'm getting this error.

In other words, I don't see any difference between installing n-t-m via yarn add or using it manually.

This is roughly my next.config.js, in case I'm doing some stupid mistake:

const withPlugins = require('next-compose-plugins');
const withTM = require('./ntm');

module.exports = withPlugins(
  [
    withTM(['@company/lib']),
  ],
  {
    env: {
    },
    typescript: {
      ignoreDevErrors: true,
    },
    webpack: (config) => {
      // etc.
      return config;
    },
  }
);

martpie · 2020-11-12T14:29:50Z

Ok, then your problem is probably in your main field in your @company/lib package.json, make sure that your main field resolve to an actual file 👍

borekb · 2020-11-12T14:58:39Z

Oh interesting, we don't have any main file (i.e., our lib doesn't have index.js), the imports in our apps look like this:

import { Xyz } from '@company/lib/components/Xyz';

This works fine for us without PnP, and even with PnP if I make this change to the next.config.js:

const withPlugins = require('next-compose-plugins');
const withTM = require('./ntm');

module.exports = withPlugins(
  [
-    withTM(['@company/lib']),
+    withTM(['../lib']),
  ],
  {
    webpack: (config) => {
      // etc.
    },
  }
);

I'd like to avoid the main field for now, for various smaller reasons, do you think our current setup can be supported by n-t-m?

martpie · 2020-11-12T22:43:24Z

Would be interesting if you can try to add a main field and just point it to an empty index.js just for testing. I had a similar situation and it made it work.

borekb · 2020-11-13T13:14:36Z

Oh yeah that works! 🎉 That's a good-enough workaround for us, I guess n-t-m should work even without it but it's a great tip, thank you!

martpie · 2020-11-13T13:38:33Z

Great, at least that's one point solved :) I will investigate the two things, and when we're ready, I can probably release the v5:

Make sure that require the module after an installation from npm works fine
See if there is a way to not rely on main (I doubt it)

thank you very much for your time!

ScriptedAlchemy · 2020-11-13T22:16:04Z

You can also transpile both paths so see what main and module go to. Right now i point to empty files when it doesn't matter (just to make compile pass and resolve)

this release seems to work for me and I've PRd it back to this project, but if anyone it stuck - this might work in the interm

@module-federation/next-transpile-modules@5.0.0-beta.6

borekb · 2020-11-20T17:07:33Z

@martpie Can I use just the company scope without a specific package name?

For example, this works: withTM(['@company/lib']). Is this also supposed to work? withTM(['@company'])

martpie · 2020-11-20T17:08:33Z

With next-transpile-modules 5 yes, not before :(

borekb · 2020-11-20T17:10:29Z

Exciting! 🎉 Any of the current branches implement that so that I could try?

martpie · 2020-11-20T17:16:34Z

Yep, you can try the nextjs-10 branch (yes, the name is misleading).

I did not specifically test this case, but from the changes I've made, it should solve this issue. cf #90

The rest of the work in progress is there -> #132

martpie · 2020-11-25T14:45:34Z

Is anyone willing to try the branch again? I tested a yarn berry and managed to fix the resolution issue (without having to copy paste the library in your project).

You can also try @5.0.0-beta.15, it should be easier to install than the branch.

~~ok, the resolution step works, but the compilation takes forever. One issue at a time 😞~~ actually nevermind. The 404 takes forever to compile.

goszczynskip mentioned this issue Mar 24, 2020

fix: absolute paths on windows #76

Merged

martpie closed this as completed Apr 28, 2020

martpie reopened this Aug 31, 2020

borekb mentioned this issue Nov 2, 2020

What's the workflow for published (compiled / built) modules? #133

Closed

martpie mentioned this issue Nov 10, 2020

v5 #115

Closed

15 tasks

martpie mentioned this issue Nov 20, 2020

feat: Improved HMR and Cache management on Webpack 5 #136

Closed

martpie closed this as completed Nov 27, 2020

mkotsollaris mentioned this issue Dec 22, 2021

Yarn PnP Virtual Workspaces Filepaths Not Properly Transpiled #199

Closed

2 tasks

Pnp workspaces aren't transpiled (yarn berry) #75

Pnp workspaces aren't transpiled (yarn berry) #75

Comments

goszczynskip commented Mar 20, 2020

martpie commented Mar 20, 2020

goszczynskip commented Mar 20, 2020

nkalinov commented Apr 27, 2020

martpie commented Apr 27, 2020

martpie commented Apr 27, 2020

goszczynskip commented Apr 27, 2020 • edited

martpie commented Apr 28, 2020

th0th commented Jul 19, 2020

andrewmclagan commented Aug 31, 2020

andrewmclagan commented Sep 1, 2020 • edited

martpie commented Sep 1, 2020 • edited

andrewmclagan commented Sep 1, 2020

andrewmclagan commented Sep 1, 2020

andrewmclagan commented Sep 3, 2020

martpie commented Sep 3, 2020

andrewmclagan commented Sep 4, 2020

goszczynskip commented Sep 15, 2020

borekb commented Sep 15, 2020

ukarlsson commented Oct 27, 2020

borekb commented Oct 27, 2020

martpie commented Oct 27, 2020

ukarlsson commented Oct 27, 2020

martpie commented Oct 27, 2020

martpie commented Oct 27, 2020

ukarlsson commented Oct 31, 2020

borekb commented Nov 4, 2020

martpie commented Nov 4, 2020 • edited

borekb commented Nov 4, 2020

borekb commented Nov 4, 2020

martpie commented Nov 10, 2020 • edited

borekb commented Nov 11, 2020

martpie commented Nov 11, 2020 • edited

borekb commented Nov 12, 2020

martpie commented Nov 12, 2020

borekb commented Nov 12, 2020

martpie commented Nov 12, 2020 • edited

goszczynskip commented Nov 12, 2020

martpie commented Nov 12, 2020

borekb commented Nov 12, 2020

martpie commented Nov 12, 2020

borekb commented Nov 12, 2020

martpie commented Nov 12, 2020

borekb commented Nov 12, 2020

martpie commented Nov 12, 2020

borekb commented Nov 13, 2020

martpie commented Nov 13, 2020

ScriptedAlchemy commented Nov 13, 2020

borekb commented Nov 20, 2020

martpie commented Nov 20, 2020

borekb commented Nov 20, 2020

martpie commented Nov 20, 2020 • edited

martpie commented Nov 25, 2020 • edited

goszczynskip commented Apr 27, 2020 •

edited

andrewmclagan commented Sep 1, 2020 •

edited

martpie commented Sep 1, 2020 •

edited

martpie commented Nov 4, 2020 •

edited

martpie commented Nov 10, 2020 •

edited

martpie commented Nov 11, 2020 •

edited

martpie commented Nov 12, 2020 •

edited

martpie commented Nov 20, 2020 •

edited

martpie commented Nov 25, 2020 •

edited