[Feature Request]: Support management of daemons and other long-lived binaries using the same lifetime as MSBuild worker nodes

[baronfel]

Summary

Daemons/build servers for tools like Roslyn and Razor have a life cycle that is currently managed outside of the MSBuild experience, even though those tools are used primarily from the Razor compilation and Csc tasks. This leads to complicated lifecycle management code, orphaned processes, etc.

Providing a way for such tools to signal that they are long-lived processes that should live as long as the MSBuild build nodes allows for unified cleanup of these resources and enables simpler management for end users - now commands like dotnet build-server shutdown msbuild and /nodeReuse:false would seamlessly apply to all tools that participate in the build.

Background and Motivation

As a performance enhancement, some areas of the .NET Toolchain have implemented persistent, reusable daemons to consolidate and share data that is expensive to compute across separate compilations. These daemons all tend to have their own customizations around their lifetimes, expected lifecycles, and shutdown mechanisms, and so a unified control plane was created in the form of the dotnet build-server shutdown command. However, users still have problems with daemons from other SDK versions, or daemons that were not shut down correctly, and so on, and so as a result have to kill orphaned processes, clear out tracking files that the daemons create, and other 'cleanup' actions.

In one use case that users often report, they would like to use the daemons in a kind of 'one-shot' mode - compiler servers would be started at the beginning of a build/at the first use of them during a build, kept available over the course of a build, and then shut down at the end of a build. This kind of matching-lifecycle behavior is currently very hard to do by using the dotnet CLI as it is today - the build-server shutdown command is very broad and will shut down all servers (of a certain kind), not just the ones that participated in a given build.

It makes natural sense for MSBuild as a task executor to also be able to function as a process-manager of sorts. Persistent daemon authors could ideally author tasks in such a way as to signal to the engine that they should live for the lifetime of the build (or beyond?) and the engine would handle calling certain lifecycle methods on the daemons based on what the Task authors opted in to.

Some examples of useful methods might be (this is not exhaustive):

• Build Started
• Build Finished
• Build Cancelled

Proposed Feature

We should provide a pattern, interface or base class for Task authors to use that would opt them into daemon tracking and lifecycle management during a build. Tasks authored in this way would be registered by the engine in some way, and a build node would be chosen to become responsible for communicating build and daemon lifecycle events to the Tasks (and other nodes). At minimum this management node would be able to spawn and terminate the daemons in a structured manner, but many other control/communication mechanisms might be defined and acted on.

Alternative Designs

This is the sketch of an idea that I wanted to write down from a meeting with @jaredpar, @rainersigwald, and @marcpopMSFT this week before I lost the key points - more detail/fleshing out is necessary.