jest-worker breaks building the project with "write EPIPE" error on macOS

[TeosVerdi]

What version of Next.js are you using?

11.0.1

What version of Node.js are you using?

16.4.2

What browser are you using?

Firefox

What operating system are you using?

macOs Big Sur 11.4 (MacBook Pro 15" 2017)

How are you deploying your application?

next build

Describe the Bug

I've decided to move my project on to next@11.0.1, using macOS for development.
When i try building it, i get the following lines:

info  - Loaded env from /Users/brrrrrr/.env
(node:99496) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)
info  - Checking validity of types
info  - Creating an optimized production build .node:events:371
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at ChildProcess.target._send (node:internal/child_process:849:20)
    at ChildProcess.target.send (node:internal/child_process:722:19)
    at ChildProcessWorker.initialize (/Users/brrrrrr/.yarn/cache/jest-worker-npm-27.0.6-83200713fc-8d7ab8cdf6.zip/node_modules/jest-worker/build/workers/ChildProcessWorker.js:181:11)
    at ChildProcessWorker._onExit (/Users/brrrrrr/.yarn/cache/jest-worker-npm-27.0.6-83200713fc-8d7ab8cdf6.zip/node_modules/jest-worker/build/workers/ChildProcessWorker.js:277:12)
    at ChildProcess.emit (node:events:394:28)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
Emitted 'error' event on ChildProcess instance at:
    at node:internal/child_process:853:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

At some point, the error was gone, but i cannot reproduce the steps necessary to completely fix the issue.
This issue is related to macOS and nextjs >11 versions only as this error doesn't pop up on other machines/versions.
Related: This project uses the pnp packaging system from yarn version berry

Expected Behavior

The project should build.

To Reproduce

Calling next build on macOS with next@11

[bhoodream]

The same problem! My build is running in docker and periodically crashes with this error!

Next.js: 11.0.1
Node.js: 14.17.3
Yarn: 2.4.2 (PnP)

events.js:352
#9 98.36       throw er; // Unhandled 'error' event
#9 98.36       ^
#9 98.36 
#9 98.36 Error: write EPIPE
#9 98.36     at ChildProcess.target._send (internal/child_process.js:839:20)
#9 98.36     at ChildProcess.target.send (internal/child_process.js:710:19)
#9 98.36     at ChildProcessWorker.initialize (/app/.yarn/cache/jest-worker-npm-27.0.0-next.5-6f53fab806-7564f4c78c.zip/node_modules/jest-worker/build/workers/ChildProcessWorker.js:181:11)
#9 98.36     at ChildProcessWorker._onExit (/app/.yarn/cache/jest-worker-npm-27.0.0-next.5-6f53fab806-7564f4c78c.zip/node_modules/jest-worker/build/workers/ChildProcessWorker.js:275:12)
#9 98.36     at ChildProcess.emit (events.js:375:28)
#9 98.36     at ChildProcess.emit (domain.js:470:12)
#9 98.36     at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
#9 98.36 Emitted 'error' event on ChildProcess instance at:
#9 98.36     at internal/child_process.js:843:39
#9 98.36     at processTicksAndRejections (internal/process/task_queues.js:77:11) {
#9 98.36   errno: -32,
#9 98.36   code: 'EPIPE',
#9 98.36   syscall: 'write'
#9 98.36 }

Downgrading to version 10.2.3 solves the problem.

And yes, we're on yarn 2 and PnP.

[borekb]

We've been fighting this as well as part of our attempt to upgrade to Next.js 11. We're not entirely sure what's causing it but some observations are:

• It only happens with Yarn's PnP mode, node-modules linker is fine.
• We tried Node.js 16 (normally, we're on v14) – no difference.
• We've tried updating all dependencies in our packages to "latest" and to use next@v11.0.2-canary.7, this seems to almost solve the issue – we see the EPIPE error much less after that but it still happens (I'd say about 10% chance; before, it was like 95%).
• If I have .next folder from a previous successful run, yarn build will typically succeed. When I delete it (rm -rf .next), I get EPIPE error again.
• We've tried pinning jest-worker to a newer version – "jest-worker": "27.0.6" in the resolutions field in package.json – didn't help.
• EPIPE with errno: -32 might be a clue but I don't know what that is – I've seen EPIPE connected to memory leaks, ungracefully closed SSL connections and many other things...

[borekb]

These two issues might be related:

• nx build produces unhandled error "Error: write EPIPE" from worker sporadically next build produces unhandled error "Error: write EPIPE" from worker sporadically #25309
• Next 10.1 respawns infinite jest-worker child processes Next 10.1 respawns infinite jest-worker child processes #23519

[borekb]

This option in next.config.js seems to resolve the issue for us:

module.exports = {
  experimental: {
    cpus: 1,
  },
};

It affects how new Worker from jest-worker is created:

next.js/packages/next/build/index.ts

Line 675 in d60c95c

numWorkers: config.experimental.cpus,

experimental.cpus is also used for other things like setting the parallelism for webpack build, see e.g. this full-text search:

Specifically for jest-worker, it sets numWorkers in jest-worker here:

const workerPoolOptions: WorkerPoolOptions = {
  numWorkers: this._options.numWorkers ?? Math.max(cpus().length - 1, 1),
  //...
};

On my machine which is MBP 16'' 2019 with a 6-core Intel i7, cpus().length is 12 so jest-worker would then use 11 as numWorkers. For experimental.cpus, I can use the values 1, 2 and 3 for the build to succeed ✅. 4 and above lead to that EPIPE error 🚫.

[borekb]

Just want to report that we had to give up on PnP, unfortunately 😕. There is something quite strange going on between Next.js, webpack, Yarn PnP and who-knows-what. This is a cross-post from our internal issue:

---

After avoiding the EPIPE error, builds started showing PnP errors like this:

$ YARN_NODE_LINKER=pnp YARN_ENABLE_GLOBAL_CACHE=false yarn build
info  - Loaded env from /Users/borekb/dev/shoptet/sofa/packages/citygolf/.env
warn  - You have enabled experimental feature(s).
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

warn  - Minimum recommended TypeScript version is v4.3.2, older versions can potentially be incompatible with Next.js. Detected: 4.1.5
info  - Checking validity of types
info  - Creating an optimized production build
Failed to compile.

ModuleNotFoundError: Module not found: Error: Can't resolve 'encoding' in '/Users/borekb/dev/shoptet/sofa/packages/.yarn/cache/node-fetch-npm-2.6.0-29c7a53447-394197397d.zip/node_modules/node-fetch/lib'


> Build error occurred
Error: > Build failed because of webpack errors
    at /Users/borekb/dev/shoptet/sofa/packages/.yarn/$$virtual/next-virtual-6178b5f3d3/0/cache/next-npm-11.0.1-c40bc7f380-552185654b.zip/node_modules/next/dist/build/index.js:15:924
    at async Span.traceAsyncFn (/Users/borekb/dev/shoptet/sofa/packages/.yarn/$$virtual/next-virtual-6178b5f3d3/0/cache/next-npm-11.0.1-c40bc7f380-552185654b.zip/node_modules/next/dist/telemetry/trace/trace.js:6:584)

I originally started dealing with those one by one – either installing it to our package.json files or adding them to .yarnrc.yml but then I stopped and explored this specific "error" coming from node-fetch.

Optional peer dependencies, PnP and webpack

node-fetch has this in their compiled output:

let convert;
try {
  convert = require('encoding').convert;
} catch (e) {}

That's a correct way to try to load an optional peerDependency, and plain Node.js + Yarn PnP don't have any problem with that: try creating a small project with index.js and a single dependency on node-fetch, this source code:

const fetch = require('node-fetch');
console.log('done');

and run it with yarn node index.js. It will work fine because PnP will throw an error inside a try/catch block, which will be swallowed by that try/catch and everything will work fine:

However, when run via webpack, this fails, so it's for further exploration in [internal issue].

BTW, by default, next build in both apps will fail with the error above. I also tried adding this to .yarnrc.yml:

packageExtensions:
  'node-fetch@2.6.0':
    peerDependencies:
      encoding: '*'
    peerDependenciesMeta:
      encoding:
        optional: true

Then it fails with a different error:

I still think this isn't right – it's an optional peer dependency so it shouldn't be required to be provided by one of its ancestors.

Loose mode as a workaround

Added in [commit hash], packages now use loose mode.

Build now passes but generates a lot of warnings:

These warnings can be silenced with PNP_DEBUG_LEVEL=0 in Yarn 3 (we're currently on 2.4.1) but better would be to find out why we're seeing failed webpack builds on a perfectly valid code.

---

Another strange thing is that trying to use webpack 4 produces very different output from webpack 5, about which I've posted #27045.

All in all, Next.js 11 + Yarn PnP seems quite broken at this point.

[jakekrog]

I've tried everything suggested here with no luck, and had to switch back to NPM for now 😞

[timmywil]

I don't think this is limited to mac btw. I'm seeing this on Windows as well.

[c10ckw0rk]

We've started getting it in our pipeline builds which run on alpine

[tadjohnston]

This doesn't seem to work on the following either:

System: MacOs Big Sur 11.4
Processor Name:	6-Core Intel Core i7
Processor Speed:	2.6 GHz
Number of Processors:	1
Total Number of Cores:	6
RAM: 16gb
Next.js: 11.1.0
Node.js: 14.17.4
Yarn: 3.0.1 (PnP)

We get the crashes on yarn@2.4.2 and next@11.0.0 and node@16.x as well.

      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at ChildProcess.target._send (internal/child_process.js:839:20)
    at ChildProcess.target.send (internal/child_process.js:710:19)
    at ChildProcessWorker.initialize (/Users/tjohnston/source/rent/.yarn/cache/jest-worker-npm-27.0.0-next.5-6f53fab806-d56bfe67cd.zip/node_modules/jest-worker/build/workers/ChildProcessWorker.js:181:11)
    at ChildProcessWorker._onExit (/Users/tjohnston/source/rent/.yarn/cache/jest-worker-npm-27.0.0-next.5-6f53fab806-d56bfe67cd.zip/node_modules/jest-worker/build/workers/ChildProcessWorker.js:275:12)
    at ChildProcess.emit (events.js:400:28)
    at ChildProcess.emit (domain.js:470:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
Emitted 'error' event on ChildProcess instance at:
    at internal/child_process.js:843:39
    at processTicksAndRejections (internal/process/task_queues.js:77:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

It will consistently crash running yarn build unless we set cpus to either 1, 2 or 3 as suggested.

experimental: {
    cpus: 1,
}

For some reason, next@10.2.3 seems to be stable for us as well.

[vcnc-hex]

Is there anyone who can test my modified PR(#28420) ?
I checked it in my local project, but I want to see if it works in other projects as well.

[mattoni]

This is still happening to me on next@11.1.2 node v16.2.0 and yarn 3.0.0.

[c10ckw0rk]

Yeah, me too, just less often

[ijjk]

Hi, please create a new issue with a repo with a minimal reproduction to allow further investigation!

[mattoni]

Much like the problem listed in this issue, it's intermittent and difficult to isolate, and it mostly happens randomly during high CPU load when doing a build. I can try to reproduce in a reliable way later but it seems to be pretty sporadic.

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.