Communication between VS -> MSBuild is pushing 6% (100MB) of allocations onto the LOH

[davkean]

We're attempting to quadruple the number of projects that the project system can open, while at same reducing solution load time. Reducing GC time is an important part of that.

In one trace where we are opening 780 projects, I'm seeing ~50 MB pushed on the large object heap (LOH) during the communication between VS and MSBuild.

It would reduce the amount of data we pushed on the LOH (and forced Gen2 collection) if we chunked this data up into arrays that were less than 85K.

GC Rollup By Generation

Gen	Count	MaxPause	MaxPeak MB	Max AllocMB/sec	TotalPause	TotalAlloc MB	Alloc MB/MSec GC	Survived MB/MSec GC	MeanPause	Induced
ALL	426	741.5	4,106.9	1,112.349	9,601.4	2,118.8	0.2	0.269	22.5	7
0	282	35.3	2,087.1	1,112.349	3,548.3	1,389.3	1.7	0.029	12.6	0
1	138	63.7	4,106.9	396.323	2,764.4	709.1	1.3	0.025	20.0	1
2	6	741.5	2,070.8	10.006	3,288.7	20.3	1.6	0.369	548.1	6

[davkean]

Here's a trace for 520 projects (above is for 780), similar problem:

[davkean]

Here's a trace that allocates 25% of all allocations during solution load on the LOH due to MSBuild:

[rainersigwald]

@benvillalobos can you check current relevance of this after the latest project system changes?

[danmoseley]

Maybe the stream passed to BinaryWriter could be some custom (subclass of?) MemoryStream that allocated in sizes that are more friendly to the LOH (?)

[ladipro]

Labeling with size:1 as the cost of the initial investigation. Will open a follow-up issue if more work is identified.

[ladipro]

Code inspection:

• Buffers allocated in NodeProviderOutOfProcBase were optimized and short-lived LOH allocations eliminated in Reduce byte array allocations when reading/writing packets #6023.
• InterningBinaryReader.ReadString was optimized in Optimize string building and interning #5663 (eliminated the StringBuilder), and Make InterningBinaryReader pool buffers #6556 and Using ArrayPool for buffers in InterningBinaryReader #6705 (implemented buffer pooling).

We should be much better off now. Let's measure.

[ladipro]

Confirmed by recording a trace opening a 1000 project solution in 17.1.0 Preview 2.0 [31915.365.main.

• ~2 MB worth of LOH char[] allocations coming from the InterningBinaryReader.
• No byte[] LOH allocations coming from MSBuild.