Introduce cross-process resource management for tasks

documentation/specs/resource-management.md

@@ -0,0 +1,41 @@
+# Managing tools with their own parallelism in MSBuild
+
+MSBuild supports building projects in parallel using multiple processes. Most users opt into `NUM_PROCS` parallelism at the MSBuild layer.
+
+In addition, tools sometimes support parallel execution. The Visual C++ compiler `cl.exe` supports `/MP[n]`, which parallelizes compilation at the translation-unit (file) level. If a number isn't specified, it defaults to `NUM_PROCS`.
+
+When used in combination, `NUM_PROCS * NUM_PROCS` compiler processes can be launched, all of which would like to do file I/O and intense computation. This generally overwhelms the operating system's scheduler and causes thrashing and terrible build times.
+
+As a result, the standard guidance is to use only one multiproc option: MSBuild's _or_ `cl.exe`'s. But that leaves the machine underloaded when things could be happening in parallel.
+
+## Design
+
+`IBuildEngine` will be extended to allow a task to indicate to MSBuild that it would like to consume more than one CPU core (`BlockingWaitForCore`). These will be advisory only—a task can still do as much work as it desires with as many threads and processes as it desires.
+
+A cooperating task would limit its own parallelism to the number of CPU cores MSBuild can reserve for the requesting task.
+
+All resources acquired by a task will be automatically returned when the task's `Execute()` method returns, and a task can optionally return a subset by calling `ReleaseCores`.
+
+MSBuild will respect core reservations given to tasks for tasks that opt into resource management only. If a project/task is eligible for execution, MSBuild will not wait until a resource is freed before starting execution of the new task. As a result, the machine can be oversubscribed, but only by a finite amount: the resource pool's core count.
+
+Task `Yield()`ing has no effect on the resources held by a task.
+
+## Example
+
+In a 16-process build of a solution with 30 projects, 16 worker nodes are launched and begin executing work. Most block on dependencies to projects `A`, `B`, `C`, `D`, and `E`, so they don't have tasks running holding resources.
+
+Task `Work` is called in project `A` with 25 inputs. It would like to run as many as possible in parallel. It calls
+
+TODO: what's the best calling pattern here? the thread thing I hacked up in the sample task seems bad.
+
+```C#
+int allowedParallelism = BuildEngine7.RequestCores(Inputs.Count); // Inputs.Count == 25
+```
+
+When `Work` returns, MSBuild automatically returns all resources reserved by the task to the pool. Before moving on with its processing, `Work2` calls `RequestCores` again, and this time receives a larger reservation.
+
+## Implementation
+
+The initial implementation of the system will use a Win32 [named semaphore](https://docs.microsoft.com/windows/win32/sync/semaphore-objects) to track resource use. This was the implementation of `MultiToolTask` in the VC++ toolchain and is a performant implementation of a counter that can be shared across processes.
+
+On platforms where named semaphores are not supported (.NET Core MSBuild running on macOS, Linux, or other UNIXes), `RequestCores` will always return `0`. We will consider implementing full support using a cross-process semaphore (or an addition to the existing MSBuild communication protocol, if it isn't prohibitively costly to do the packet exchange and processing) on these platforms in the future.

ref/Microsoft.Build.Framework/net/Microsoft.Build.Framework.cs

@@ -211,6 +211,11 @@ public partial interface IBuildEngine7 : Microsoft.Build.Framework.IBuildEngine,
     {
         bool AllowFailureWithoutError { get; set; }
     }
+    public partial interface IBuildEngine8 : Microsoft.Build.Framework.IBuildEngine, Microsoft.Build.Framework.IBuildEngine2, Microsoft.Build.Framework.IBuildEngine3, Microsoft.Build.Framework.IBuildEngine4, Microsoft.Build.Framework.IBuildEngine5, Microsoft.Build.Framework.IBuildEngine6, Microsoft.Build.Framework.IBuildEngine7
+    {
+        void ReleaseCores(int coresToRelease);
+        int RequestCores(int requestedCores);
+    }
     public partial interface ICancelableTask : Microsoft.Build.Framework.ITask
     {
         void Cancel();

ref/Microsoft.Build.Framework/netstandard/Microsoft.Build.Framework.cs

@@ -211,6 +211,11 @@ public partial interface IBuildEngine7 : Microsoft.Build.Framework.IBuildEngine,
     {
         bool AllowFailureWithoutError { get; set; }
     }
+    public partial interface IBuildEngine8 : Microsoft.Build.Framework.IBuildEngine, Microsoft.Build.Framework.IBuildEngine2, Microsoft.Build.Framework.IBuildEngine3, Microsoft.Build.Framework.IBuildEngine4, Microsoft.Build.Framework.IBuildEngine5, Microsoft.Build.Framework.IBuildEngine6, Microsoft.Build.Framework.IBuildEngine7
+    {
+        void ReleaseCores(int coresToRelease);
+        int RequestCores(int requestedCores);
+    }
     public partial interface ICancelableTask : Microsoft.Build.Framework.ITask
     {
         void Cancel();

ref/Microsoft.Build.Utilities.Core/net/Microsoft.Build.Utilities.Core.cs

@@ -353,6 +353,7 @@ public abstract partial class Task : Microsoft.Build.Framework.ITask
         public Microsoft.Build.Framework.IBuildEngine5 BuildEngine5 { get { throw null; } }
         public Microsoft.Build.Framework.IBuildEngine6 BuildEngine6 { get { throw null; } }
         public Microsoft.Build.Framework.IBuildEngine7 BuildEngine7 { get { throw null; } }
+        public Microsoft.Build.Framework.IBuildEngine8 BuildEngine8 { get { throw null; } }
         protected string HelpKeywordPrefix { get { throw null; } set { } }
         public Microsoft.Build.Framework.ITaskHost HostObject { get { throw null; } set { } }
         public Microsoft.Build.Utilities.TaskLoggingHelper Log { get { throw null; } }

ref/Microsoft.Build.Utilities.Core/netstandard/Microsoft.Build.Utilities.Core.cs

@@ -198,6 +198,7 @@ public abstract partial class Task : Microsoft.Build.Framework.ITask
         public Microsoft.Build.Framework.IBuildEngine5 BuildEngine5 { get { throw null; } }
         public Microsoft.Build.Framework.IBuildEngine6 BuildEngine6 { get { throw null; } }
         public Microsoft.Build.Framework.IBuildEngine7 BuildEngine7 { get { throw null; } }
+        public Microsoft.Build.Framework.IBuildEngine8 BuildEngine8 { get { throw null; } }
         protected string HelpKeywordPrefix { get { throw null; } set { } }
         public Microsoft.Build.Framework.ITaskHost HostObject { get { throw null; } set { } }
         public Microsoft.Build.Utilities.TaskLoggingHelper Log { get { throw null; } }

ref/Microsoft.Build/net/Microsoft.Build.cs

@@ -996,6 +996,7 @@ public partial class BuildParameters
         public string OutputResultsCacheFile { get { throw null; } set { } }
         public Microsoft.Build.Evaluation.ProjectLoadSettings ProjectLoadSettings { get { throw null; } set { } }
         public bool ResetCaches { get { throw null; } set { } }
+        public string ResourceManagerSemaphoreName { get { throw null; } set { } }
         public bool SaveOperatingEnvironment { get { throw null; } set { } }
         public bool ShutdownInProcNodeOnBuildFinish { get { throw null; } set { } }
         public Microsoft.Build.Evaluation.ToolsetDefinitionLocations ToolsetDefinitionLocations { get { throw null; } set { } }

ref/Microsoft.Build/netstandard/Microsoft.Build.cs

@@ -991,6 +991,7 @@ public partial class BuildParameters
         public string OutputResultsCacheFile { get { throw null; } set { } }
         public Microsoft.Build.Evaluation.ProjectLoadSettings ProjectLoadSettings { get { throw null; } set { } }
         public bool ResetCaches { get { throw null; } set { } }
+        public string ResourceManagerSemaphoreName { get { throw null; } set { } }
         public bool SaveOperatingEnvironment { get { throw null; } set { } }
         public bool ShutdownInProcNodeOnBuildFinish { get { throw null; } set { } }
         public Microsoft.Build.Evaluation.ToolsetDefinitionLocations ToolsetDefinitionLocations { get { throw null; } set { } }

src/Build.UnitTests/BackEnd/MockHost.cs

@@ -2,6 +2,7 @@
 // Licensed under the MIT license. See LICENSE file in the project root for full license information.
 
 using Microsoft.Build.BackEnd;
+using Microsoft.Build.BackEnd.Components.ResourceManager;
 using Microsoft.Build.BackEnd.Logging;
 using System;
 using Microsoft.Build.BackEnd.SdkResolution;
@@ -59,6 +60,8 @@ internal class MockHost : MockLoggingService, IBuildComponentHost, IBuildCompone
 
         private ISdkResolverService _sdkResolverService;
 
+        private readonly ResourceManagerService _taskResourceManager;
+
         #region SystemParameterFields
 
         #endregion;
@@ -102,6 +105,9 @@ public MockHost(BuildParameters buildParameters)
 
             _sdkResolverService = new MockSdkResolverService();
             ((IBuildComponent)_sdkResolverService).InitializeComponent(this);
+
+            _taskResourceManager = new ResourceManagerService();
+            ((IBuildComponent)_taskResourceManager).InitializeComponent(this);
         }
 
         /// <summary>
@@ -170,6 +176,7 @@ public IBuildComponent GetComponent(BuildComponentType type)
                 BuildComponentType.ResultsCache => (IBuildComponent)_resultsCache,
                 BuildComponentType.RequestBuilder => (IBuildComponent)_requestBuilder,
                 BuildComponentType.SdkResolverService => (IBuildComponent)_sdkResolverService,
+                BuildComponentType.TaskResourceManager => (IBuildComponent)_taskResourceManager,
                 _ => throw new ArgumentException("Unexpected type " + type),
             };
         }

src/Build/BackEnd/BuildManager/BuildParameters.cs

@@ -213,6 +213,7 @@ public class BuildParameters : ITranslatable
         private string[] _inputResultsCacheFiles;
 
         private string _outputResultsCacheFile;
+        private string _resourceManagerSemaphoreName = $"MSBuild.{Guid.NewGuid().ToString()}";
 
         /// <summary>
         /// Constructor for those who intend to set all properties themselves.
@@ -770,6 +771,12 @@ public string OutputResultsCacheFile
             set => _outputResultsCacheFile = value;
         }
 
+        public string ResourceManagerSemaphoreName
+        {
+            get => _resourceManagerSemaphoreName;
+            set => _resourceManagerSemaphoreName = value;
+        }
+
         /// <summary>
         /// Determines whether MSBuild will save the results of builds after EndBuild to speed up future builds.
         /// </summary>
@@ -838,6 +845,7 @@ void ITranslatable.Translate(ITranslator translator)
             translator.TranslateEnum(ref _projectLoadSettings, (int) _projectLoadSettings);
             translator.Translate(ref _interactive);
             translator.Translate(ref _isolateProjects);
+            translator.Translate(ref _resourceManagerSemaphoreName);
 
             // ProjectRootElementCache is not transmitted.
             // ResetCaches is not transmitted.

src/Build/BackEnd/Components/BuildComponentFactoryCollection.cs

@@ -3,6 +3,7 @@
 
 using System.Collections.Generic;
 using Microsoft.Build.BackEnd.Components.Caching;
+using Microsoft.Build.BackEnd.Components.ResourceManager;
 using Microsoft.Build.BackEnd.SdkResolution;
 using Microsoft.Build.Shared;
 
@@ -78,6 +79,8 @@ public void RegisterDefaultFactories()
 
             // SDK resolution
             _componentEntriesByType[BuildComponentType.SdkResolverService] = new BuildComponentEntry(BuildComponentType.SdkResolverService, MainNodeSdkResolverService.CreateComponent, CreationPattern.Singleton);
+
+            _componentEntriesByType[BuildComponentType.TaskResourceManager] = new BuildComponentEntry(BuildComponentType.TaskResourceManager, ResourceManagerService.CreateComponent, CreationPattern.Singleton);
         }
 
         /// <summary>

src/Build/BackEnd/Components/IBuildComponentHost.cs

@@ -128,6 +128,11 @@ internal enum BuildComponentType
         /// The SDK resolution service.
         /// </summary>
         SdkResolverService,
+
+        /// <summary>
+        /// Resource manager for tasks to use via <see cref="Microsoft.Build.Framework.IBuildEngine8.RequestCores(int)"/>.
+        /// </summary>
+        TaskResourceManager,
     }
 
     /// <summary>

src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs

@@ -338,11 +338,9 @@ private async Task<WorkUnitResult> ExecuteTask(TaskExecutionMode mode, Lookup lo
             {
                 _taskExecutionHost.CleanupForTask();
 
-#if FEATURE_APPDOMAIN
                 taskHost?.MarkAsInactive();
-#endif
 
-                // Now all task batches are done, apply all item adds to the outer 
+                // Now all task batches are done, apply all item adds to the outer
                 // target batch; we do this even if the task wasn't found (in that case,
                 // no items or properties will have been added to the scope)
                 if (buckets != null)
@@ -404,14 +402,14 @@ private async Task<WorkUnitResult> ExecuteBucket(TaskHost taskHost, ItemBucket b
                 {
                     // Change to the project root directory.
                     // If that directory does not exist, do nothing. (Do not check first as it is almost always there and it is slow)
-                    // This is because if the project has not been saved, this directory may not exist, yet it is often useful to still be able to build the project. 
+                    // This is because if the project has not been saved, this directory may not exist, yet it is often useful to still be able to build the project.
                     // No errors are masked by doing this: errors loading the project from disk are reported at load time, if necessary.
                     NativeMethodsShared.SetCurrentDirectory(_buildRequestEntry.ProjectRootDirectory);
                 }
 
                 if (howToExecuteTask == TaskExecutionMode.ExecuteTaskAndGatherOutputs)
                 {
-                    // We need to find the task before logging the task started event so that the using task statement comes before the task started event 
+                    // We need to find the task before logging the task started event so that the using task statement comes before the task started event
                     IDictionary<string, string> taskIdentityParameters = GatherTaskIdentityParameters(bucket.Expander);
                     TaskRequirements? requirements = _taskExecutionHost.FindTask(taskIdentityParameters);
                     if (requirements != null)
@@ -514,15 +512,15 @@ private async Task<WorkUnitResult> ExecuteBucket(TaskHost taskHost, ItemBucket b
         /// </summary>
         private IDictionary<string, string> GatherTaskIdentityParameters(Expander<ProjectPropertyInstance, ProjectItemInstance> expander)
         {
-            ErrorUtilities.VerifyThrowInternalNull(_taskNode, "taskNode"); // taskNode should never be null when we're calling this method. 
+            ErrorUtilities.VerifyThrowInternalNull(_taskNode, "taskNode"); // taskNode should never be null when we're calling this method.
 
             string msbuildArchitecture = expander.ExpandIntoStringAndUnescape(_taskNode.MSBuildArchitecture ?? String.Empty, ExpanderOptions.ExpandAll, _taskNode.MSBuildArchitectureLocation ?? ElementLocation.EmptyLocation);
             string msbuildRuntime = expander.ExpandIntoStringAndUnescape(_taskNode.MSBuildRuntime ?? String.Empty, ExpanderOptions.ExpandAll, _taskNode.MSBuildRuntimeLocation ?? ElementLocation.EmptyLocation);
 
             IDictionary<string, string> taskIdentityParameters = null;
 
-            // only bother to create a task identity parameter set if we're putting anything in there -- otherwise, 
-            // a null set will be treated as equivalent to all parameters being "don't care". 
+            // only bother to create a task identity parameter set if we're putting anything in there -- otherwise,
+            // a null set will be treated as equivalent to all parameters being "don't care".
             if (msbuildRuntime != String.Empty || msbuildArchitecture != String.Empty)
             {
                 taskIdentityParameters = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
@@ -821,7 +819,7 @@ private async Task<WorkUnitResult> ExecuteInstantiatedTask(ITaskExecutionHost ta
 
                     // Set the property "MSBuildLastTaskResult" to reflect whether the task succeeded or not.
                     // The main use of this is if ContinueOnError is true -- so that the next task can consult the result.
-                    // So we want it to be "false" even if ContinueOnError is true. 
+                    // So we want it to be "false" even if ContinueOnError is true.
                     // The constants "true" and "false" should NOT be localized. They become property values.
                     bucket.Lookup.SetProperty(ProjectPropertyInstance.Create(ReservedPropertyNames.lastTaskResult, taskResult ? "true" : "false", true/* may be reserved */, _buildRequestEntry.RequestConfiguration.Project.IsImmutable));
                 }
@@ -888,7 +886,7 @@ private async Task<WorkUnitResult> ExecuteInstantiatedTask(ITaskExecutionHost ta
                     }
                     else if (type == typeof(Exception) || type.GetTypeInfo().IsSubclassOf(typeof(Exception)))
                     {
-                        // Occasionally, when debugging a very uncommon task exception, it is useful to loop the build with 
+                        // Occasionally, when debugging a very uncommon task exception, it is useful to loop the build with
                         // a debugger attached to break on 2nd chance exceptions.
                         // That requires that there needs to be a way to not catch here, by setting an environment variable.
                         if (ExceptionHandling.IsCriticalException(taskException) || (Environment.GetEnvironmentVariable("MSBUILDDONOTCATCHTASKEXCEPTIONS") == "1"))