Without --wait, shell subprocesses can deadlock / are not reaped

[bitglue]

For example:

watchmedo shell-command -c 'cat' /some/file

The relevant code:

        self.process = subprocess.Popen(command, shell=True)                                    
        if self.wait_for_process:                                                               
            self.process.wait()    

One thing to note, the subprocess will inherit watchmedo's file descriptors, which maybe isn't desirable. (Or maybe it is?) In this case it means cat will never exit since stdin is inherited from watchmedo, which if watchmedo was run from an interactive shell (for example) is a TTY that may never be closed. So an endless stream of cats which never exit will be spawned.

But also, wait(2) is never called, so even if the subprocess does exit, it's never reaped. This accumulates zombie processes until watchmedo exits, at which point init will inherit those subprocesses and reap them. Of course that may never happen. You can see this with ps -f for example:

29547 ?        Sl     3:00  \_ /usr/bin/python /usr/local/bin/watchmedo shell-command -R -c [...]
19616 ?        Z      0:00      \_ [sh] <defunct>
19680 ?        Z      0:00      \_ [sh] <defunct>
19682 ?        Z      0:00      \_ [sh] <defunct>
19724 ?        Z      0:00      \_ [sh] <defunct>
19735 ?        Z      0:00      \_ [sh] <defunct>
19896 ?        Z      0:00      \_ [sh] <defunct>
20063 ?        Z      0:00      \_ [sh] <defunct>
20530 ?        Z      0:00      \_ [sh] <defunct>
23257 ?        Z      0:00      \_ [sh] <defunct>

One solution would be making --wait the default -- it's what I'd want personally for the use cases I have in mind.

Alternately, something needs to wait around for the child processes to exit. That could be a thread which calls wait which will block, or something like epoll_wait can do it in a non-blocking way.

[tgpfeiffer]

But also, wait(2) is never called, so even if the subprocess does exit, it's never reaped.

Same issue exists in latest watchmedo auto-restart (2.1.8), if the subprocess exits it is not reaped and continues as a zombie process. Just run watchmedo auto-restart sleep infinity and kill the sleep process by PID, then the process tree looks like

 7471 pts/1    Sl+    0:00  |   |   |   \_ .../bin/python .../bin/watchmedo auto-restart sleep infinity
 7472 ?        Zs     0:00  |   |   |       \_ [sleep] <defunct>

[taleinat]

I have the same issue with watchmedo auto-restart: When the sub-process exits, watchmedo neither restarts the sub-process nor exits.

[taleinat]

@BoboTiG, I don't think this should be closed following PR #896 being merged. That PR addresses only the auto-restart trick, while this issue is foremost about the shell command trick.