Enable arcade-powered-source-build for local and CI builds

[omajid]

This enables 'source-build', which makes it easier to build the entire shipping .NET SDK from source.

This is the first and second step of arcade-powered-source-build:
https://github.com/dotnet/source-build/blob/master/Documentation/planning/arcade-powered-source-build/README.md

See dotnet/sourcelink#692 for a similar PR, that this is based on.

For this round, this includes all .patch files from source-build directly. Some of that is easy enough to clean up and merge directly, but I am going to put that off for later, following the plan.

[omajid]

This needs an update to version of Microsoft.DotNet.Arcade.Sdk package so it can work; the current version is too old and can't be found.

error NU1102: Unable to find package Microsoft.SourceBuild.Intermediate.arcade with version (= 6.0.0-beta.20573.2) 

Probably #1172?

[omajid]

During repeated builds, I did see an error like this locally:

command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018: The "GenerateDepsFile" task failed unexpectedly. [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018: System.IO.IOException: The process cannot access the file 'command-line-api/artifacts/source-build/self/src/artifacts/bin/System.CommandLine/Release/netstandard2.0/System.CommandLine.deps.json' because it is being used by another process. [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at System.IO.FileStream.Init(FileMode mode, FileShare share, String originalPath) [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at System.IO.File.Create(String path) [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at Microsoft.NET.Build.Tasks.GenerateDepsFile.WriteDepsFile(String depsFilePath) [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at Microsoft.NET.Build.Tasks.GenerateDepsFile.ExecuteCore() [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]
command-line-api/.dotnet/sdk/3.1.301/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(194,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [command-line-api/artifacts/source-build/self/src/src/System.CommandLine/System.CommandLine.csproj]                                                                                          

Is this a known issue? Has anyone seen this before?

[omajid]

cc @crummel @dagood @dseefeld

[brettfo]

Repo has been upgraded to latest arcade and .NET 5 and I re-merged/re-started the CI.

[omajid]

CI is hitting the same errors that I was seeing locally, so it's not just me. The error is always with System.CommandLine.deps.json. Seems to be along the lines of dotnet/sdk#2902.

[dagood]

It looks like these MSBuild tasks are spinning up another build of the same csproj in a way that misses the MSBuild cache and causes a race:

command-line-api/src/System.CommandLine.Suggest.Tests/dotnet-suggest.Tests.csproj

Lines 59 to 63 in d87107b

	<MSBuild BuildInParallel="False" Projects="../System.CommandLine.Suggest/dotnet-suggest.csproj" Targets="Restore" Properties="UseAppHost=true;SelfContained=false;RuntimeIdentifier=$(Rid);ForceRestoreToEvaluateSeparately=1;Configuration=Release">
	</MSBuild>
	<MSBuild BuildInParallel="False" Projects="../System.CommandLine.Suggest/dotnet-suggest.csproj" Targets="Build;Publish" Properties="UseAppHost=true;SelfContained=false;RuntimeIdentifier=$(Rid);PublishDir=$(TestAssetsPath);Configuration=Release">
	</MSBuild>

When a project is built twice with the same global properties, this is safe: it hits some MSBuild cache and it only builds once and both dependent projects continue their builds after it's done. When the global properties are different like this, both builds execute and you get races like this.

The .Tests project should be excluded by source-build, and that should solve this for source-build, but it still looks like it would be an intermittent issue in the non-source-build build (depending on build machine perf).

(I searched the binlog for Release/netstandard2.0/System.CommandLine.deps.json and found the same task running twice in distinct nodes of the same csproj with different global properties.)

[omajid]

I am not sure what to do about the nullability errors I am now running into. Here are a few examples:

System.CommandLine/DictionaryExtensions.cs(8,44): error CS8714: The type 'TKey' cannot be used as type parameter 'TKey' in the generic type or method 'IDictionary<TKey, TValue>'. Nullability of type argument 'TKey' doesn't match 'notnull' constraint. [System.CommandLine/System.CommandLine.csproj]
System.CommandLine/DictionaryExtensions.cs(27,44): error CS8714: The type 'TKey' cannot be used as type parameter 'TKey' in the generic type or method 'IDictionary<TKey, TValue>'. Nullability of type argument 'TKey' doesn't match 'notnull' constraint. [System.CommandLine/System.CommandLine.csproj]
System.CommandLine/Help/HelpOption.cs(25,30): error CS8765: Nullability of type of parameter 'obj' doesn't match overridden member (possibly because of nullability attributes). [System.CommandLine/System.CommandLine.csproj]
System.CommandLine/Binding/Binder.cs(92,24): error CS8603: Possible null reference return. [System.CommandLine/System.CommandLine.csproj]
System.CommandLine/Invocation/ModelBindingCommandHandler.cs(83,58): error CS8604: Possible null reference argument for parameter 'value' in 'Task<int> CommandHandler.GetExitCodeAsync(object value, InvocationContext context)'. [System.CommandLine/System.CommandLine.csproj]

Is there a more appropriate fix than fixing up all the code like https://github.com/dotnet/command-line-api/pull/1204/files#diff-0ebe1e9864869d3cb89f16ad74b56273cab824fbc1e9bff94e4daa288025ae02R65-R66?

[dseefeld]

@omajid I looked at this a little more yesterday, based on the suggestions from @crummel. Regarding the nullability errors, these are warnings added because of the update to netcoreapp3.1. You can add <NoWarn Condition=" '$(DotnetBuildFromSource)' == 'true' ">$(NoWarn);CS8714;CS8765;CS8600;CS8601;CS8602;CS8603;CS8604</NoWarn> to Directory.Build.props to eliminate those.

[jonsequitur]

@omajid Please rebase this against main and let me know when you're ready for it to be merged.

[omajid]

@jonsequitur Thanks, I have squashed and rebased the PR onto main.

[omajid]

Thanks for merging this!