Nodemon is not able to gracefully restart applications in Windows environment

[chriswoodle]

Possibly related to #1567

• nodemon -v: 6.13.4

• node -v: v12.16.1

• Operating system/terminal environment:

  • Windows Command Prompt, Windows Git Bash, Windows Terminal WSL

       Edition: Windows 10 Pro 
       Version: 1909
  Installed on: 5/22/2019
      OS Build: 18363.836 
  

• Command ran: nodemon --signal SIGHUP index.js

Expected behaviour

On Windows, Nodemon should wait for the process to complete if it handles a shut down signal.

Notice the express server closed message

WSL:

Actual behaviour

The application is killed and restarted right away. This behavior does not exist in Linux, Mac and WSL.

CMD:

Steps to reproduce

I have created a sample Express app here: https://github.com/chriswoodle/nodemon-1567 that shuts down the express server once the application is signaled to stop ['SIGTERM', 'SIGINT', 'SIGHUP']

git clone https://github.com/chriswoodle/nodemon-1567.git
cd nodemon-1567
npm install
npm start
touch index.js

Let me know if I can provide any more information or test changes, thanks.

---

[remy]

If I get time, I'll look into this, but I wanted to stop and say:

Thank you for filing an issue with a good level of detail and a good example of pared down code to replicate the issue. I'll be pointing people to this issue as an example of what makes the world of difference in debugging. 🙏

[nicholas-ochoa]

I came here with the same exact problem and I'm glad I found this issue. On *nix this works perfectly, on Windows it's almost like the signals are ignored and a SIGKILL is used instead? I'm seeing the same behavior regardless during restarts for file changes (or typing 'rs') - the application is IMMEDIATELY restarted and my shutdown handlers are never called.

[stale]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[nicholas-ochoa]

Still happening..

[JulianNicholls]

Hi Remy.

Just a suggestion

I hadn't seen this problem on Windows via WSL until I enabled a subscription in my GraphQL server.
GraphQL Subscriptions use websockets, which may be relevant.

This seems to be consistent, activating a subscription will cause nodemon to not terminate the server before attempting to restart. It usually leaves more than one instance running.

I can't offer any solutions, but I'd be happy to help you debug it.

[countzero]

Hi there,

I dug a little bit deeper into this rabbit hole and like to share my findings:

Problem: taskkill does not send signals

If nodemon is running in a Windows context it is using the taskkill command to terminate the node process:

nodemon/lib/monitor/run.js

Line 347 in a74f5dc

exec('taskkill /pid ' + child.pid + ' /T /F');

Sadly taskkill does not send any signals prior to reaping the process. Therefore no process event listeners within the Node.js application will trigger.

Solution: use windows-kill

There is actually a way to programmatically send the following signals to a process in Windows:

• SIGINT ( CTRL + C )
• SIGBREAK ( CTRL + Break )

There is a project called windows-kill which implements this:

.\windows-kill.exe -SIGINT 16312 

The node process (16312) received a SIGINT POSIX signal, starting graceful shutdown...

I used the windows-kill_x64_1.1.4_lib_release.zip binary.

Next steps

@remy I think the fix would be a drop in replacement of taskkill with windows-kill. But I have also seen the big comments above the taskkill invocation and do not know all edge-cases you have already tested for. And I am not sure, if it is possible to simply add the windows-kill.exe binary to this project (yet another dependency, license issues etc.). What do you think?

[koustavmandal95]

Facing issue nodemon stuck at restarting the server on windows 10 platform . Any trick around ?

[usm076]

Still isn't resolved

[sridharjatla]

i have some issue with nodemon when i use command nodemon app.js -e js,hbs after making some changes in hbs files nodemon is not restarting only for the first time its working later i have to shutdown the app.js and rerun the command

[remy]

Finally getting around to this - given @chriswoodle's code example, my primary question is: how does this work without nodemon?

@countzero I'm not sure this is the right solution - my reading is that this problem is a failing of the code running in a system that doesn't have POSSIX signals, and therefore the source code shouldn't be listening for these. Now, I could be entirely wrong, but that fact it works in WSL suggests that the windows-kill project is actually monkey patching in signals. Nodemon definitely does not want to do this because it'll change the running environment without nodemon (and one of the core principles that I first built nodemon with is: not to change the behaviour of runtime - i.e. no code changes required, etc).

I don't often have a windows machine to hand test this, but there's a good number of you on this thread that could try @chriswoodle's example git repo that replicates the problem.

The tasks is: run without nodemon, does the application behave as expected?

[chriswoodle]

Finally getting around to this - given @chriswoodle's code example, my primary question is: how does this work without nodemon?

@countzero I'm not sure this is the right solution - my reading is that this problem is a failing of the code running in a system that doesn't have POSSIX signals, and therefore the source code shouldn't be listening for these. Now, I could be entirely wrong, but that fact it works in WSL suggests that the windows-kill project is actually monkey patching in signals. Nodemon definitely does not want to do this because it'll change the running environment without nodemon (and one of the core principles that I first built nodemon with is: not to change the behaviour of runtime - i.e. no code changes required, etc).

I don't often have a windows machine to hand test this, but there's a good number of you on this thread that could try @chriswoodle's example git repo that replicates the problem.

The tasks is: run without nodemon, does the application behave as expected?

Using windows cmd, running node index and pressing CTRL + C sends SIGINT and waits for the process to exit.

Even when adding a timeout period.

[remy]

@chriswoodle I was pretty sure that the command prompt checked with confirmation whether you wanted to exit (the "are you sure y/n") on ctrl-c.

Why wouldn't that happen in this case? Or is this a particular flavour of Windows/the prompt/something else?

[remy]

Hmm... unless it's node intercepting the interrupt and doing something bespoke...

[chriswoodle]

@chriswoodle I was pretty sure that the command prompt checked with confirmation whether you wanted to exit (the "are you sure y/n") on ctrl-c.

Why wouldn't that happen in this case? Or is this a particular flavour of Windows/the prompt/something else?

I believe this was the case in the past, "Terminate batch job (Y/N)?" is promoted when exiting batch/cmd scripts. I think node/npm switched over to powershell ps1 scripts, which do not have this prompt. Not %100 certain though, but I think it's something like this.

[remy]

Then I suspect this is what nodemon will need. I really don't want to add a bespoke binary 3rd party to do the windows-kill thing - it doesn't feel right. But I really know nothing about powershell so am looking to Windows users to help advise/drive.

[chriswoodle]

Just to note,
Looking at my windows node install:

Runnng node from cmd

Directly executes node.exe, so no powershell involved.

[remy]

I wonder what the nodemon executable looks like on Windows? Isn't it one of those .cmd files?

[GIDustin]

nodemon is a cmd file, which calls node.exe with arguments:

so you have a node.exe for nodemon, monitoring a node.exe for the app:

From tasklist:

[oviecodes]

@remy how can I help with this. I've got a Windows machine.

[stale]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[caesay]

Just bumping this because of the message from the bot and this does not appear to be resolved as of yet. Ideally nodemon should send a signal and wait for the process to exit before restarting and it does not currently do this.

To clarify on the "Terminate batch job (Y/N)?" question, it's true that this only happens with a batch/cmd file which is executed directly from terminal. For example, upon Ctrl-C:

• node index.js no prompt
• nodemon index.js (installed with -g) prompt present
• npm run nodemon-pkg-script (installed locally as dev-dep) prompt present
• node "node_modules/nodemon/bin/nodemon.js" no prompt

Additionally, I do not believe you need a third party binary to do this if you are happy to use powershell, the following line of powershell will give you a list of all of the processes running, their pid, and their parent pid - so from here it would be possible to terminate a process tree recursively with no additional binaries:

gwmi win32_process | select -property Name,ProcessId,ParentProcessId

[remy]

Happy to take a PR that detects powershell and uses that method. I'm not in Windows at all, so I'm no help here (and unless someone actually contributes a PR to address this, nothing is really going to change - though I'm sure I've said this before )

[countzero]

Progress: I found a solution to hide the command prompt window (see above commit for details). Now there is only one bug left.

The "restart" event is triggered twice on the bus:

[nodemon] restarting due to changes...
The node process (13076) received a SIGINT POSIX signal, starting graceful shutdown...
Closing V8 inspector at: ws://0.0.0.0:9229/4b694f27-e0b6-4adf-ae55-f88d422552d3...
Terminating process with exit code 0...
[nodemon] restarting due to changes...
[nodemon] starting `node --expose-gc --inspect=0.0.0.0:9229 ./application/server/index.js`

@remy: I already detected the debug feature of nodemon and will look into the process order. My first hunch is, that nodemon always expects a kill signal on its child ("cmd.exe") and not on the childs child ("node"). Which perhaps causes two exit events somehow? Have to read some code to understand the context...

[countzero]

Short status report:

I made a minimal proof of concept implementation with all the nodemon modifications I made so far. It is public and should be as developer friendly as possible to

1. help narrowing down the bug of duplicate restart events and
2. rapidly test the implementation in a Windows environment.

https://github.com/countzero/nodemon_windows_kill

@caesay & @chriswoodle: Do you have some time to test this?

[countzero]

Hi,

I optimized the https://github.com/countzero/nodemon_windows_kill proof of concept code further: Now it is cross-platform compatible, so that it can run side by side in a PowerShell context and a WSL Debian context:

My findings from this are, that nodemon sometimes triggers POSIX signals multiple times in BOTH the Windows AND the UNIX context! I am not sure, if that is intended behaviour. @remy Is this actually intended behaviour, or is this a bug?

 npm run watch

> nodemon_windows_kill@0.0.1 watch D:\Privat\Projekte\nodemon_windows_kill
> npx nodemon --signal SIGINT ./index.js

[nodemon] main: 626a168ccaf6806b12a392660180508f21d0a7ee
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node ./index.js`
The server started listening on http://localhost:5938
[nodemon] restarting due to changes...
The node process (33676) received a SIGINT POSIX signal - initiating shutdown...
Terminating process with exit code 0...
[nodemon] starting `node ./index.js`
The server started listening on http://localhost:5941
[nodemon] restarting due to changes...
The node process (27500) received a SIGINT POSIX signal - initiating shutdown...
[nodemon] restarting due to changes...
The node process (27500) received a SIGINT POSIX signal - already shutting down, ignoring...
Terminating process with exit code 0...
[nodemon] starting `node ./index.js`
The server started listening on http://localhost:5943

[cowboyd]

fyi, we wrote a very small package to do nothing except send a CTRL_C event to a windows process to handle situations exactly like this. https://github.com/thefrontside/ctrlc-windows/

Don't know if it would be a handy thing to use here, but it's worked well for us, and is extremely lightweight.

[wrexbe]

Here is a way to send ctrl-c to the node process.

Command

powershell.exe -nologo -noprofile -executionpolicy bypass .\ctrlc.ps1 ProcessIdHere

File: ctrlc.ps1

Add-Type -Names 'w' -Name 'k' -M @(
    '[DllImport("kernel32.dll")] public static extern bool FreeConsole();',
    '[DllImport("kernel32.dll")] public static extern bool AttachConsole(uint p);',
    '[DllImport("kernel32.dll")] public static extern bool GenerateConsoleCtrlEvent(uint e, uint p);'
)
If ([w.k]::FreeConsole()) {
    If([w.k]::AttachConsole($args[0])) {
        If([w.k]::GenerateConsoleCtrlEvent(0, 0)){
            exit 0
        }
    }
}
throw;

Edit:

Made it so I didn't need a separate file. This approach doesn't work though. It sends the Ctrl-C to too many things.

const pid = 2500;
const ctrlcScript = Buffer.from(\`Add-Type -Names 'w' -Name 'k' -M @(
    '[DllImport("kernel32.dll")] public static extern bool FreeConsole();',
    '[DllImport("kernel32.dll")] public static extern bool AttachConsole(uint p);',
    '[DllImport("kernel32.dll")] public static extern bool GenerateConsoleCtrlEvent(uint e, uint p);'
);
If ([w.k]::FreeConsole()) {
    If([w.k]::AttachConsole(${pid})) {
        If([w.k]::GenerateConsoleCtrlEvent(0, 0)){
            exit 0;
        }
    }
}
exit 1;`.replace(/(\r\n|\n|\r)/gm, " "), 'utf16le').toString('base64');
var cp = require('child_process');
cp.exec(`powershell.exe -nologo -noprofile -executionpolicy bypass -EncodedCommand ${ctrlcScript}\`, function(e, stdout, stderr) {
  console.log(stdout);
  console.log(stderr);
});

[wrexbe]

@countzero I tried your solution. The link for windows-kill was dead, but it worked for me after I changed it. Are you going to bundle the exe, and make a pull request?

[countzero]

@remy I removed the provisioning script and added the windows-kill.exe binary instead.

You were right there to be sceptic. This way nodemon is way more resilient to external changes: Actually the maintainer of windows-kill changed his GitHub username recently from "alirdn" to "ElyDotDev" which broke the provisioning!

I updated https://github.com/countzero/nodemon_windows_kill to match my latest commit and added a PR for you: #1853

[countzero]

@cowboyd & @wrexbe: Thank you for your suggestions. I am very interested in the native PowerShell only solution... What are your thoughts on why your suggestions are better than the windows-kill binary?

[wrexbe]

@countzero I think the windows-kill binary is better. I couldn't get it to work correctly with just powershell, it is probably possible, but it might be a lot of work.

[remy]

Folks, there's a debug build available on npm from @countzero's great work.

Those with Windows, could you test the install using npm i -g nodemon@debug which should give you 2.0.8-alpha.a - and then check that it shuts the windows sub-process down correctly.

If you can comment here or on PR #1853 that would be perfect to help sign these changes off.

[wrexbe]

2.0.8-alpha.a is correctly shutting down, and restarting for me.

It might help if 'SIGKILL' mapped to the previous method, to give people another option if the program they are working on ignores SIGINT, or just takes a long time, and they don't care about it safely exiting.

[countzero]

@wrexbe: I like that idea!

So the use case is: As a developer on Windows I like nodemon to terminate the node process by force, to quickly restart my node process that does not require a graceful shutdown.

Under Windows nodemon would then have two behaviours:

1. Graceful Shutdown: If the --signal option is not SIGKILL nodemon will gracefully terminate the node process group with a SIGINT. That means SIGINT will be send to the process group and nodemon waits for node to gracefully shutdown itself.

2. Hard Shutdown: If the --signal option is SIGKILL nodemon will terminate the node process group by force with taskkill /t /f. In this case we are not able to send any signal under windows because sadly there is no such thing as SIGKILL.

@remy: If you are OK with this behaviour, I can extend the PR. Just need some time...

[countzero]

@wrexbe Done. Please take a look at countzero@f7c6e0d and @remy this commit is also included in the PR #1853!

[countzero]

@wrexbe @chriswoodle @cowboyd @nicholas-ochoa @JulianNicholls @usm076 @caesay @oviecodes @GIDustin @sridharjatla Do you have some time to test this solution to our shared problem?

You can simply check out https://github.com/countzero/nodemon_windows_kill to rapidly test this fix.

The SIGINT behaviour can be tested with:

npm run watch-graceful-shutdown

And the "old" SIGKILL behaviour with:

npm run watch-hard-shutdown

See https://github.com/countzero/nodemon_windows_kill/blob/main/package.json for technical details.

@remy I am not quite sure what else can be done here. What are the next steps?

[rjamesnw]

@remy That alpha works for me also, thanks! :) Just ran into this, but glad to find this thread.

[GIDustin]

Your test project worked fine for me. I even tested without a SIGINT handler to ensure that apps without one would still function as they did in the past and that worked as well.

[chriswoodle]

@countzero Seems to work for me in CMD, PowerShell, Git Bash:

However, I'm not sure if this is related, but your example repo doesen't detect file changes when running in WSL (I tested in WSL 2 Debian). Saving index.js does not trigger a restart

I tested your repo on the example repo I linked (https://github.com/chriswoodle/nodemon-1567) and I get the same behavior.

[countzero]

@chriswoodle I am currently using WSL 1 because I am working frequently with VirtualBox. This works with Debian:

Please note, that I did NOT modify the process restart behaviour of nodemon in a UNIX context. If you can reproduce this behaviour in a Debian under Microsofts WSL 2 you found a new bug which is unrelated to the changes discussed in this issue!

I am curious if you can reproduce this with the latest nodemon version...

[chriswoodle]

@chriswoodle I am currently using WSL 1 because I am working frequently with VirtualBox. This works with Debian:

Please note, that I did NOT modify the process restart behaviour of nodemon in a UNIX context. If you can reproduce this behaviour in a Debian under Microsofts WSL 2 you found a new bug which is unrelated to the changes discussed in this issue!

I am curious if you can reproduce this with the latest nodemon version...

@countzero Yea, looks like this is unrelated bug, that issue happens on latest nodemon 2.0.7

[chriswoodle]

Yep, I've opened a new issue: #1866

@countzero 's repo seems to work just fine.

[remy]

Merged and live.