Add /graph:noBuild #6016

cdmihai · 2021-01-10T00:18:07Z

This new option constructs the graph but does not build the nodes.

This is useful for a few use cases:

determine if a large repo can evaluate all of its projects (e.g., is VS setup right, are all imports magically pointing to where they should, etc)
easily investigate evaluation perf. With noBuild, the only thing that MSBuild does is evaluate all projects under a single process, so it's easier to throw it under a profiler.
in the future when project caching is available, it can be used to warm up / download the caches but not do the build.
generally makes life easier for people that only want to investigate evaluation and not the build. The binlog also shows up nicely containing all the evaluation nodes.

cdmihai · 2021-01-10T00:26:45Z

@rainersigwald I added a record type with init properties and it seems the CI is using an older compiler that does not recognize these. Is it fairly easy to get CI updated, or should I replace the records with manual equals / hashcode implementations? 😢

rainersigwald · 2021-01-11T12:25:54Z

I bet the update to Arcade 5/.NET 5 SDK will handle that. @benvillalobos is pretty close on that so I wouldn't rewrite.

rainersigwald

LGTM -- should we fork for 16.9 so we have a place to land this?

src/MSBuild/Resources/Strings.resx

src/Shared/AssemblyUtilities.cs

ref/Microsoft.Build/net/Microsoft.Build.cs

rainersigwald · 2021-01-11T23:26:13Z

/azp run

azure-pipelines · 2021-01-11T23:26:21Z

Azure Pipelines successfully started running 1 pipeline(s).

Forgind · 2021-01-14T22:48:28Z

src/Build/BackEnd/BuildManager/BuildManager.cs

+
+            while (blockedNodes.Count > 0 || buildingNodes.Count > 0)
+            {
+                waitHandle.WaitOne();


I'm a little confused here. It looks like the waitHandle receives signals below, but how would it ever get there if the thread is paused indefinitely here?

This just an extracted method, not new logic, you can skip it if you wish. To answer your question, whenever there's potential new work to do the waitHandle gets signaled. Check out all the places in the method that call Set() on that waitHandle.

cdmihai · 2021-01-20T22:21:49Z

@radical, do you know how to update the compilers in the Mono CI to a new enough version to parse C# record types? CI is currently failing because of that: https://dev.azure.com/dnceng/public/_build/results?buildId=959029&view=logs&j=09f5a607-bbad-5164-6b70-f20ebe806390&t=64b2bbfe-7647-5e20-2e5f-fab3af842098

Forgind

LGTM!

Forgind · 2021-01-26T23:40:22Z

src/Build/Graph/GraphBuildRequestData.cs

@@ -7,6 +7,14 @@

 namespace Microsoft.Build.Graph
 {
+    public record GraphBuildOptions


Why make this a record if it only has one thing in it? It seems like it's making it unnecessarily complicated, since then in addition to checking whether it's null, you need to check whether .Build is true or false.

Because there will be more things in it in the future and since MSBuild APIs are set in stone until the end of time I'd rather start with a 1 member record than successive API changes that progressively add yet another public constructor with yet another parameter followed eventually by an actual struct. Basically avoiding the fiasco with the Project constructors which got eventually followed by ProjectOptions.

Forgind · 2021-01-27T00:19:36Z

src/Build/BackEnd/BuildManager/BuildManager.cs

-                using var cacheService = cacheServiceTask.Result;
+                if (submission.BuildRequestData.GraphBuildOptions.Build)
+                {
+                    var cacheServiceTask = Task.Run(() => SearchAndInitializeProjectCachePluginFromGraph(projectGraph));


Just out of curiosity—why did you reorder these? Shouldn't matter either way.

Pedantism. Put the task which I think takes the longest first. Thread can loose context right before issuing the second task so might as well have the longest running one first.

Forgind · 2021-01-27T00:23:20Z

src/Build/Graph/GraphBuildRequestData.cs

+        /// <summary>
+        /// If false, the graph is constructed but the nodes are not built.
+        /// </summary>
+        public bool Build { get; init; } = true;


I wasn't familiar with this, and it looks like it's too new for mono. Might change it to set { init; return whatever; }

Mono will get eventually updated and until then we'll just disable it. See #6059

benvillalobos

Mostly LGTM! Pending build errors & questions

benvillalobos · 2021-02-03T22:48:01Z

src/MSBuild/XMake.cs

@@ -1233,7 +1233,7 @@ string outputResultsCache

                            if (!restoreOnly)
                            {
-                                if (graphBuild)
+                                if (graphBuildOptions != null)


&& graphBuildOptions.Build?

graphBuildOptions.Build gets acted upon later on in the BuildManager, after the graph is constructed. That's the whole point, construct the graph but not build it.

benvillalobos · 2021-02-03T22:49:39Z

src/MSBuild/XMake.cs

+
+                if (parameter.Trim().Equals("NoBuild", StringComparison.OrdinalIgnoreCase))
+                {
+                    options = options with {Build = false};


Should this data structure be a record if we actually want to be able to set the value?

Why should it not be a record?

benvillalobos · 2021-02-03T22:51:49Z

src/Build.UnitTests/BackEnd/BuildManager_Tests.cs

+        [Fact]
+        public void GraphBuildShouldBeAbleToConstructGraphButSkipBuild()
+        {
+            var graph = Helpers.CreateProjectGraph(_env, new Dictionary<int, int[]> {{1, new[] {2, 3}}});


Nit: list out parameter names to make it easier to read. You did it just below this.

Random question: Any worry about this method being passed invalid data?

It's a helper method for tests, if it's passed invalid data some test somewhere will fail.

benvillalobos · 2021-02-03T22:53:48Z

src/Build/BackEnd/BuildManager/BuildManager.cs


-                IReadOnlyDictionary<ProjectGraphNode, ImmutableList<string>> targetLists = targetListTask.Result;
-                using var cacheService = cacheServiceTask.Result;
+                if (submission.BuildRequestData.GraphBuildOptions.Build)


Do we need a null check here?

The GraphBuildRequestData constructors ensures it's never null.

Forgind · 2021-02-03T23:47:12Z

And the build error is because the mono build doesn't have an updated version of roslyn. Resolved by #6059 (admittedly not ideally—by updating roslyn for mono—but by disabling it).

cdmihai · 2021-02-04T00:21:07Z

/azp run

azure-pipelines · 2021-02-04T00:21:16Z

Azure Pipelines successfully started running 1 pipeline(s).

cdmihai force-pushed the noBuild branch from 4f47ed2 to 3353e49 Compare January 10, 2021 00:19

cdmihai added the Area: Static Graph Issues with -graph, -isolate, and the related APIs. label Jan 10, 2021

rainersigwald reviewed Jan 11, 2021

View reviewed changes

src/MSBuild/Resources/Strings.resx Show resolved Hide resolved

src/Shared/AssemblyUtilities.cs Outdated Show resolved Hide resolved

ref/Microsoft.Build/net/Microsoft.Build.cs Show resolved Hide resolved

Forgind reviewed Jan 14, 2021

View reviewed changes

cdmihai force-pushed the noBuild branch 2 times, most recently from 317f2d5 to 25ba2ed Compare January 20, 2021 21:54

Forgind approved these changes Jan 27, 2021

View reviewed changes

Forgind added the merge-when-branch-open PRs that are approved, except that there is a problem that means we are not merging stuff right now. label Jan 29, 2021

cdmihai mentioned this pull request Feb 3, 2021

Fix cache exception handling #6110

Merged

benvillalobos reviewed Feb 3, 2021

View reviewed changes

cdmihai force-pushed the noBuild branch from 25ba2ed to aaea331 Compare February 3, 2021 23:47

cdmihai added 5 commits February 3, 2021 16:22

Remove unnecessary quotes

d0e960b

Add cmdline arg and OM

116442a

Extract graph build in new method

6776fe6

Implement NoBuild

c73e724

document IsExternalInit

3942a2c

cdmihai force-pushed the noBuild branch from aaea331 to 3942a2c Compare February 4, 2021 00:22

Forgind merged commit 3eb41d4 into dotnet:master Feb 8, 2021

jairbubbles mentioned this pull request Feb 12, 2021

New command line argument for evaluating properties / items #6071

Closed

cdmihai deleted the noBuild branch February 16, 2021 23:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add /graph:noBuild #6016

Add /graph:noBuild #6016

cdmihai commented Jan 10, 2021 •

edited

cdmihai commented Jan 10, 2021 •

edited

rainersigwald commented Jan 11, 2021

rainersigwald left a comment

rainersigwald commented Jan 11, 2021

azure-pipelines bot commented Jan 11, 2021

Forgind Jan 14, 2021

cdmihai Jan 14, 2021 •

edited

cdmihai commented Jan 20, 2021

Forgind left a comment

Forgind Jan 26, 2021

cdmihai Jan 27, 2021

Forgind Jan 27, 2021

cdmihai Jan 27, 2021

Forgind Jan 27, 2021

cdmihai Jan 27, 2021

benvillalobos left a comment

benvillalobos Feb 3, 2021

cdmihai Feb 3, 2021

benvillalobos Feb 3, 2021

cdmihai Feb 3, 2021

benvillalobos Feb 3, 2021

benvillalobos Feb 3, 2021

cdmihai Feb 3, 2021

benvillalobos Feb 3, 2021

cdmihai Feb 3, 2021 •

edited

Forgind commented Feb 3, 2021

cdmihai commented Feb 4, 2021

azure-pipelines bot commented Feb 4, 2021

Add /graph:noBuild #6016

Add /graph:noBuild #6016

Conversation

cdmihai commented Jan 10, 2021 • edited

cdmihai commented Jan 10, 2021 • edited

rainersigwald commented Jan 11, 2021

rainersigwald left a comment

Choose a reason for hiding this comment

rainersigwald commented Jan 11, 2021

azure-pipelines bot commented Jan 11, 2021

Choose a reason for hiding this comment

cdmihai Jan 14, 2021 • edited

Choose a reason for hiding this comment

cdmihai commented Jan 20, 2021

Forgind left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benvillalobos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdmihai Feb 3, 2021 • edited

Choose a reason for hiding this comment

Forgind commented Feb 3, 2021

cdmihai commented Feb 4, 2021

azure-pipelines bot commented Feb 4, 2021

cdmihai commented Jan 10, 2021 •

edited

cdmihai commented Jan 10, 2021 •

edited

cdmihai Jan 14, 2021 •

edited

cdmihai Feb 3, 2021 •

edited