Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"fatal error: all goroutines are asleep - deadlock!" #3287

Closed
kentcdodds opened this issue Aug 5, 2023 · 7 comments
Closed

"fatal error: all goroutines are asleep - deadlock!" #3287

kentcdodds opened this issue Aug 5, 2023 · 7 comments

Comments

@kentcdodds
Copy link

Hello! I know you've received a few reports like in the past and they weren't actionable because they were not reproducible. I think I can reproduce this though.

So my case is kinda unique. I'm using mdx-bundler which uses esbuild under the hood to compile over a hundred mdx files when I startup my app (and then I call it during runtime as well, but my issues happen during startup). I have only successfully reproduced this issue when trying to deploy to Fly. Here are the logs:

2023-08-05T02:28:01.231 app[9080554b003287] sjc [info] 2023/08/05 02:28:01 listening on [fdaa:0:23df:a7b:b2e2:1f93:e7ac:2]:22 (DNS: [fdaa::3]:53)
2023-08-05T02:28:01.519 app[9080554b003287] sjc [info] setting up swapfile...
2023-08-05T02:28:01.571 app[9080554b003287] sjc [info] swapfile setup complete
2023-08-05T02:28:01.596 app[9080554b003287] sjc [info] Cloning into '/myapp/workshop-content'...
2023-08-05T02:28:04.912 health[9080554b003287] sjc [error] Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.
2023-08-05T02:28:15.328 app[9080554b003287] sjc [info] Updating files: 10% (1839/17917) Updating files: 11% (1971/17917) Updating files: 12% (2151/17917) Updating files: 13% (2330/17917) Updating files: 14% (2509/17917) Updating files: 15% (2688/17917) Updating files: 16% (2867/17917) Updating files: 17% (3046/17917) Updating files: 18% (3226/17917) Updating files: 19% (3405/17917) Updating files: 20% (3584/17917) Updating files: 21% (3763/17917) Updating files: 22% (3942/17917) Updating files: 22% (4001/17917) Updating files: 23% (4121/17917) Updating files: 24% (4301/17917) Updating files: 25% (4480/17917) Updating files: 26% (4659/17917) Updating files: 27% (4838/17917) Updating files: 28% (5017/17917) Updating files: 29% (5196/17917) Updating files: 30% (5376/17917) Updating files: 31% (5555/17917) Updating files: 32% (5734/17917) Updating files: 33% (5913/17917) Updating files: 34% (6092/17917) Updating files: 35% (6271/17917) Updating files: 36% (6451/17917) Updating files: 37% (6630/17917) Updating files: 37% (6793/17917) Updating files: 38% (6809/17917) Updating files: 39% (6988/17917) Updating files: 40% (7167/17917) Updating files: 41% (7346/17917) Updating files: 42% (7526/17917) Updating files: 43% (7705/17917) Updating files: 44% (7884/17917) Updating files: 45% (8063/17917) Updating files: 46% (8242/17917) Updating files: 47% (8421/17917) Updating files: 48% (8601/17917) Updating files: 49% (8780/17917) Updating files: 50% (8959/17917) Updating files: 51% (9138/17917) Updating files: 52% (9317/17917) Updating files: 53% (9497/17917) Updating files: 54% (9676/17917) Updating files: 54% (9825/17917) Updating files: 55% (9855/17917) Updating files: 56% (10034/17917) Updating files: 57% (10213/17917) Updating files: 58% (10392/17917) Updating files: 59% (10572/17917) Updating files: 60% (10751/17917) Updating files: 61% (10930/17917) Updating files: 62% (11109/17917) Updating files: 63% (11288/17917) Updating files: 64% (11467/17917) Updating files: 65% (11647/17917) Updating files: 66% (11826/17917) Updating files: 67% (12005/17917) Updating files: 68% (12184/17917) Updating files: 69% (12363/17917) Updating files: 70% (12542/17917) Updating files: 70% (12624/17917) Updating files: 71% (12722/17917) Updating files: 72% (12901/17917) Updating files: 73% (13080/17917) Updating files: 74% (13259/17917) Updating files: 75% (13438/17917) Updating files: 76% (13617/17917) Updating files: 77% (13797/17917) Updating files: 78% (13976/17917) Updating files: 79% (14155/17917) Updating files: 80% (14334/17917) Updating files: 81% (14513/17917) Updating files: 82% (14692/17917) Updating files: 83% (14872/17917) Updating files: 84% (15051/17917) Updating files: 85% (15230/17917) Updating files: 86% (15409/17917) Updating files: 86% (15483/17917) Updating files: 87% (15588/17917) Updating files: 88% (15767/17917) Updating files: 89% (15947/17917) Updating files: 90% (16126/17917) Updating files: 91% (16305/17917) Updating files: 92% (16484/17917) Updating files: 93% (16663/17917) Updating files: 94% (16842/17917) Updating files: 95% (17022/17917) Updating files: 96% (17201/17917) Updating files: 97% (17380/17917) Updating files: 98% (17559/17917) Updating files: 99% (17738/17917) Updating files: 100% (17917/17917) Updating files: 100% (17917/17917), done.
2023-08-05T02:28:26.045 app[9080554b003287] sjc [info] 🐨 Let's get learning!
2023-08-05T02:28:26.046 app[9080554b003287] sjc [info] Local: http://localhost:8080
2023-08-05T02:28:26.046 app[9080554b003287] sjc [info] On Your Network: http://172.19.133.114:8080
2023-08-05T02:28:26.046 app[9080554b003287] sjc [info] Press Ctrl+C to stop
2023-08-05T02:28:30.350 app[9080554b003287] sjc [info] fatal error: all goroutines are asleep - deadlock!
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] goroutine 1 [chan receive]:
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/helpers.(*ThreadSafeWaitGroup).Wait(...)
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/helpers/waitgroup.go:36
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] main.runService.func2()
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:114 +0x59
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] main.runService(0x1)
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:160 +0x4ed
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] main.main()
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/main.go:240 +0xa29
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] goroutine 6 [chan receive]:
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] main.runService.func1()
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:98 +0x4a
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] created by main.runService
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:97 +0x1e5
2023-08-05T02:28:30.354 app[9080554b003287] sjc [info] goroutine 7 [chan receive]:
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] main.(*serviceType).sendRequest(0xc000023ef0, {0x90a9e0, 0xc0001e6900})
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:192 +0xfa
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] main.runService.func3()
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:125 +0x39
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] created by main.runService
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:122 +0x31c
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] goroutine 8 [chan receive]:
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler.(*scanner).scanAllDependencies(0xc00008f680)
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:1854 +0x232
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler.ScanBundle(_, {_, _, _, _, _, _}, {_, _}, 0xc0000643c0, ...)
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:1259 +0xb36
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.rebuildImpl({0xc0000643c0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x1, 0x0, ...}, ...}, ...)
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api_impl.go:1495 +0x305
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.(*internalContext).rebuild(_)
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api_impl.go:1024 +0x28c
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.(*internalContext).Rebuild(...)
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api_impl.go:1085
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.Build({0x0, 0x4, 0x0, 0xc0000f8300, 0x0, {0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api.go:383 +0xb6
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] main.(*serviceType).handleBuildRequest(0xc000023ef0, 0x0?, 0xc000023f80?)
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:763 +0xe58
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] main.(*serviceType).handleIncomingPacket.func2()
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:235 +0x68
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] created by main.(*serviceType).handleIncomingPacket
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:233 +0x1c5
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] goroutine 12 [chan receive]:
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] main.(*serviceType).sendRequest(0xc000023ef0, {0x90a9e0, 0xc0001e6990})
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:192 +0xfa
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] main.(*serviceType).convertPlugins.func2.4({{0xc000028370, 0x49}, {0x97998b, 0x4}, {0x0, 0x0}, {0x0, 0x0}})
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:1054 +0x626
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.(*pluginImpl).onLoad.func1({{0x0, 0x0}, {{0xc000028370, 0x49}, {0x97998b, 0x4}, {0x0, 0x0}, 0x0}})
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api_impl.go:1969 +0x19d
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler.runOnLoadPlugins({0xc000064420?, 0x1, 0x40d20a?}, 0xc00004cc00?, {0xa62038, 0xc000062400?}, 0x9770a0?, {0xc0000fc620, 0xc0000104e0, 0xc0000104f8, ...}, ...)
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:945 +0xe33
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler.parseFile({{0xa62038, 0xc000062400}, {0xc0000fc620, 0xc0000104e0, 0xc0000104f8, 0xc0000623e0, 0x4, 0xc0000f8780}, 0xc00008f200, 0xc0000643c0, ...})
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:135 +0x227
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] created by github.com/evanw/esbuild/internal/bundler.(*scanner).maybeParseFile
2023-08-05T02:28:30.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:1382 +0xab6
2023-08-05T02:28:31.265 app[9080554b003287] sjc [info] INFO Main child exited normally with code: 1
2023-08-05T02:28:31.265 app[9080554b003287] sjc [info] INFO Starting clean up.
2023-08-05T02:28:31.265 app[9080554b003287] sjc [info] WARN hallpass exited, pid: 257, status: signal: 15 (SIGTERM)
2023-08-05T02:28:31.270 app[9080554b003287] sjc [info] 2023/08/05 02:28:31 listening on [fdaa:0:23df:a7b:b2e2:1f93:e7ac:2]:22 (DNS: [fdaa::3]:53)
2023-08-05T02:28:32.264 app[9080554b003287] sjc [info] [ 31.342665] reboot: Restarting system
2023-08-05T02:28:33.306 runner[9080554b003287] sjc [info] machine did not have a restart policy, defaulting to restart
2023-08-05T02:28:34.096 app[9080554b003287] sjc [info] [ 0.043705] PCI: Fatal: No config space access function found
2023-08-05T02:28:34.310 app[9080554b003287] sjc [info] INFO Starting init (commit: b572577)...
2023-08-05T02:28:34.331 app[9080554b003287] sjc [info] INFO Preparing to run: `docker-entrypoint.sh /bin/sh -c node ./setup-swap.js && rm -rf ${KCDSHOP_CONTEXT_CWD} && git clone https://github.com/epicweb-dev/web-auth ${KCDSHOP_CONTEXT_CWD} && cd ${KCDSHOP_CONTEXT_CWD} && npx kcdshop start` as root
2023-08-05T02:28:34.341 app[9080554b003287] sjc [info] INFO [fly api proxy] listening at /.fly/api

You might be able to reproduce it locally with the same Dockerfile. You'll find the Dockerfile and relevant files here: https://github.com/epicweb-dev/web-auth/tree/ebe638c4966e0d44da0ebc88094af55dd0defcc5/scripts/deployed

It's not much, so I'll just stick it here

Dockerfile

FROM node:18-bookworm-slim as base

RUN apt-get update && apt-get install -y git

ENV KCDSHOP_CONTEXT_CWD="/myapp/workshop-content"
ENV KCDSHOP_DEPLOYED="true"
ENV FLY="true"
ENV PORT="8080"
ENV NODE_ENV="production"

WORKDIR /myapp

ADD . .

RUN npm install --omit=dev

CMD node ./setup-swap.js && \
    rm -rf ${KCDSHOP_CONTEXT_CWD} && \
    git clone https://github.com/epicweb-dev/web-auth ${KCDSHOP_CONTEXT_CWD} && \
    cd ${KCDSHOP_CONTEXT_CWD} && \
    npx kcdshop start

setup-swap.js

#!/usr/bin/env node

import { $ } from 'execa'
import { writeFile, stat } from 'node:fs/promises'

const swapExists = await stat('/swapfile').catch(() => false)

console.log('setting up swapfile...')

if (swapExists) {
	console.log('swapfile already exists')
} else {
	await $`fallocate -l 512M /swapfile`
	await $`chmod 0600 /swapfile`
	await $`mkswap /swapfile`
	await writeFile('/proc/sys/vm/swappiness', '10')
	await $`swapon /swapfile`
	await writeFile('/proc/sys/vm/overcommit_memory', '1')
	console.log('swapfile setup complete')
}

package.json

{
  "name": "deployed-web-auth",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "kcdshop start"
  },
  "author": "Kent C. Dodds <me@kentcdodds.com> (https://kentcdodds.com/)",
  "license": "GPL-3.0-only",
  "dependencies": {
    "@kentcdodds/workshop-app": "^2.12.3",
    "execa": "^7.2.0"
  },
  "engines": {
    "node": ">=18",
    "npm": ">=8.16.0"
  }
}

fly.toml

app = "epicweb-dev-web-auth"
primary_region = "sjc"
kill_signal = "SIGINT"
kill_timeout = 5
processes = [ ]

[experimental]
allowed_public_ports = [ ]
auto_rollback = true

[[services]]
internal_port = 8080
processes = [ "app" ]
protocol = "tcp"
script_checks = [ ]

  [services.concurrency]
  hard_limit = 100
  soft_limit = 80
  type = "connections"

  [[services.ports]]
  handlers = [ "http" ]
  port = 80
  force_https = true

  [[services.ports]]
  handlers = [ "tls", "http" ]
  port = 443

  [[services.tcp_checks]]
  grace_period = "1s"
  interval = "15s"
  restart_limit = 0
  timeout = "2s"

  [[services.http_checks]]
  interval = "10s"
  grace_period = "5s"
  method = "get"
  path = "/"
  protocol = "http"
  timeout = "2s"
  tls_skip_verify = false
  headers = { }

When I last reported this issue (#1608), it was solved by adding queuing around where I call bundleMDX (which is what calls esbuild) to ensure it's only called one at a time. I've added that same queuing to this, but that has not affected the outcome at all. The error I pasted above has been consistent through all the things I've tried.

I hope this report helps to identify what's causing this issue (#1608, #1842, #2485, #2727) for once and for all!

@evanw
Copy link
Owner

evanw commented Aug 5, 2023

I don't know anything about docker. But I tried anyway and couldn't get this to work. Here's what I tried:

$ git clone https://github.com/epicweb-dev/web-auth
$ cd web-auth
$ git checkout ebe638c4966e0d44da0ebc88094af55dd0defcc5
$ cd scripts/deployed
$ docker build -t "test:Dockerfile" .
$ docker run -t "test:Dockerfile"
setting up swapfile...
node:internal/process/esm_loader:97
    internalBinding('errors').triggerUncaughtException(
                              ^

[Error: EROFS: read-only file system, open '/proc/sys/vm/swappiness'] {
  errno: -30,
  code: 'EROFS',
  syscall: 'open',
  path: '/proc/sys/vm/swappiness'
}

Node.js v18.17.0

What commands are necessary to reproduce this with the code that you linked to?

@kentcdodds
Copy link
Author

Yeah, I worried about that. I think it may only be reproducible on a lower end machine. If you want to try it locally, remove the bit about the swap file. Then the Dockerfile should work locally. But like I said it may only reproduce on a low-end machine (maybe you can simulate that in docker? I don't know). You could probably reproduce this on fly (within the free tier) if you want to try that.

Create an account and install fly. You'll need to change the app name in the fly.toml to something unique, then do:

fly apps create [your app name]

Then you can run

fly deploy --remote-only

And it should deploy and reproduce. You can see logs in the fly dashboard.

@evanw
Copy link
Owner

evanw commented Aug 6, 2023

I tried making an account (with a disposable email) but then I get this:

Error: We need your payment information to continue! Add a credit card or buy credit

I'm not doing this, sorry.

I also tried removing the swapfile stuff and then running things with docker run -t "test:Dockerfile" but no deadlock occurs. I'm marking this issue as unactionable because I'm not going to proceed here.

@kentcdodds
Copy link
Author

Bummer. Sorry for another dead end on this. Do you have any ideas what could possibly cause this?

@kentcdodds
Copy link
Author

And a follow-up question. Do you know why this would cause my server process to exit even if I add a catch?

@kentcdodds
Copy link
Author

Ok, here's an update: I realized the reason my server was crashing was due to accidentally watching a lot of files with chokidar. After adding more logging and finding the issue, I fixed it and now I'm not getting the deadlock issues. I'm not sure whether that's related or not, but the fact is I'm not getting this issue anymore so I think we can safely close it.

@csbenjamin
Copy link

I am also encountering the fatal error: all goroutines are asleep - deadlock! issue while using esbuild in my development environment. I am working on a Mac M1 Pro Max and facing this deadlock issue specifically when I have an Angular server running alongside multiple Node.js projects using webpack --watch. The deadlock occurs during the build process of an Angular application.

Interestingly, when I stop the Angular server and the webpack --watch processes, the build completes successfully without any errors. This seems to suggest some sort of resource contention or deadlock situation arising due to the concurrent execution of these processes.

Additionally, it's worth noting that my setup involves building the Angular application (version 17.1.3) inside a Docker container based on the node:18.17.1 image. This setup has worked well for me until I encountered this deadlock issue with esbuild.

I am looking for any guidance or suggestions on how to resolve or work around this issue. Has anyone else experienced something similar in a Dockerized environment, particularly with the mentioned versions of Node.js and Angular, on a Mac M1 Pro Max? Any insights or recommendations would be greatly appreciated.

Thank you in advance for your assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants