Get feedback from task-runners and build tools

[VanCoding]

task runners

• wireit Proposal: A protocol watch processes can implement google/wireit#608
• rush [rush] Avoid cold start overhead by allowing a toolchain process to continue running (as with "webpack --watch") microsoft/rushstack#1151
• turborepo Support executing multiple dependent long-running tasks in parallel  vercel/turbo#1497
• nx Support for dependency-aware "watch" tasks on nx run-many nrwl/nx#5570

build tools

• TypeScript Would you implement a simple protocol for the watch command? microsoft/TypeScript#52297

[eboody]

could you provide an example of how I would get started using your protocol for managing tsup --watch or tsc -w to restart some file-watching server when some other package has a file change?

[VanCoding]

@eboody Well... the point of this issue is to get task runners and build tools to implement the protocol. If tools don't support it, you can't use it.

What I currently have is an example task-runner and an example typescript build tool, but you shouldn't use it for anything yet.

[DibyodyutiMondal]

Off the top of my head, while it does make sense to have a protocol per se, won't sharing stdout between the processes be a bit problematic?

Say for example A depends on B depends on C.

If C outputs 'DETECTED_CHANGES', then C's task runner rebuilds C, and then outputs 'START'.
This will be seen by both A and B task runners and both will rebuild simultaneously, but I don't want A to start until B also starts.

Would it not be better, then, to use process messages? The message content will contain information whether it is a watch-protocol message. Then, we can look at other data in the message to determine which kind of protocol signal it is as well as which task sent it, and react/ignore accordingly.

EDIT:
Also, instead of waiting for build-tools to implement this, it'll probably be better to offer them a utility that will take a configuration like the following:

{
  'task-a': {
    command: 'npm run a',
    dependsOn: ['task-b']
  },
  'task-b': {
    command: 'npm run b',
    dependsOn: ['task-c']
  },
  'task-c': {
    command: 'npm run c',
    dependsOn: []
  }
}

A main process the spawns child process for these in the correct order, but they're all direct child processes. The main process listens for messages from the child processes notifies only the correct dependents. That way, each task has only one instance, yet there is a clean cascading effect of triggers.

Additionally, providing a wrapper utility that can run a command, look at the STDOUT and accordingly post watch protocol messages to the process, will enable normal users to write scripts that wrap the build tools as they currently are. As more and and more people do this, the build tool maintainers will feel more motivated to actively incorporate the protocol into their code.

[VanCoding]

Thanks for taking part in this discussion! :)

I think there's a misunderstanding what the task-runner is in the context of the watch-task-protocol. There's only one task-runner and it's the process the starts the task-processes. Each task is then its own process. The protocol is between the task-runner and these processes. So stdout and stdin is not "shared" between these task processes. They don't know anything about each other. It's the task-runners job to monitor a task and trigger a rebuild on another task.

Here's an illustration:

First run:
[task-runner] ----starts---> [task A]
[task-runner] <--finished--- [task A] (process keeps running)
[task-runner] ---starts----> [task B]
[task-runner] <--finished--- [task B] (process keeps running)

A detects changes:
[task-runner] <--changes--- [task A]
[task-runner] ---rebuild--> [task A]
[task-runner] <--finished-- [task A]
[task-runner] ---rebuild--> [task B]
[task-runner] <--finished-- [task B]

About process messages:
As far as I know, process messages are a Node.js only feature. So it would not be possible to communicate to a build-tool not running atop Node.js.

A main process the spawns child process for these in the correct order, but they're all direct child processes. The main process listens for messages from the child processes notifies only the correct dependents. That way, each task has only one instance, yet there is a clean cascading effect of triggers.

That's basically what task-graph-processor is doing right now.

Additionally, providing a wrapper utility that can run a command, look at the STDOUT and accordingly post watch protocol messages to the process, will enable normal users to write scripts that wrap the build tools as they currently are. As more and and more people do this, the build tool maintainers will feel more motivated to actively incorporate the protocol into their code.

Yes detecting if a task is finished via the STDOUT is no problem. The hard part is telling the task-process not to rerun until we tell it to do so. That's something we cannot do from the outside, sadly. And it's basically the only feature we really need the build tools to implement.