Azure Devops pipelines crashing in MSBuild logger, as of 5/25 VS2019 image

[rainersigwald]

---

Issue moved from dotnet/msbuild#6498

• Please respond to @Scottj1s.

---

From @Scottj1s on Friday, May 28, 2021 1:10:09 AM

Beginning on 5/25, a stage of the WinUI build which executes on hosted VS2019 agents, began failing intermittently with the stack below. Subsequent runs of the pipeline produce different combinations of success/failure.

MSBUILD : error MSB4017: The build stopped unexpectedly because of an unexpected logger failure. 
Microsoft.Build.Exceptions.InternalLoggerException: The build stopped unexpectedly because of an unexpected logger failure. ---> System.NullReferenceException: Object reference not set to an instance of an object. 
at MSBuild.Logger.BuildConfiguration.Equals(Object obj) 
at System.Collections.Generic.ObjectEqualityComparer`1.Equals(T x, T y) 
at System.Collections.Generic.List`1.Contains(T item) 
at MSBuild.Logger.ProjectTrees.AddTopLevelProject(ProjectStartedEventArgs startedEvent, BuildConfiguration platformConfiguration) 
at MSBuild.Logger.CentralLogger.HandleProjectStarted(Object sender, ProjectStartedEventArgs e) 
at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseProjectStartedEvent(Object sender, ProjectStartedEventArgs buildEvent) 

[anatolybolshakov]

Hi @Scottj1s could you please share your logs? You can also send them to my email - v-anbols@microsoft.com

[Scottj1s]

Direct emailed binlogs of failed builds.

A bit more investigation shows that the build failure is not intermittent, as supposed. It is 100% correlated with the image update on 5/25. Stages of our build pipeline that happen to land on agents with the 5/16 image always succeed. Those that are queued to an agent with the 5/25 image always fail.

[asklar]

This is now hitting React Native for Windows pipelines: https://dev.azure.com/ms/react-native-windows/_build/results?buildId=174255&view=logs&jobId=f933640c-866b-5d69-8f9e-c72a61865db3&j=f933640c-866b-5d69-8f9e-c72a61865db3&t=cc5f9f17-41e7-537a-c494-777e5536ceae

Microsoft.Build.Exceptions.InternalLoggerException: The build stopped unexpectedly because of an unexpected logger failure. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at MSBuild.Logger.BuildConfiguration.Equals(Object obj)
   at System.Collections.Generic.ObjectEqualityComparer`1.Equals(T x, T y)
   at System.Collections.Generic.List`1.Contains(T item)
   at MSBuild.Logger.ProjectTrees.AddTopLevelProject(ProjectStartedEventArgs startedEvent, BuildConfiguration platformConfiguration)
   at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseProjectStartedEvent(Object sender, ProjectStartedEventArgs buildEvent)
   --- End of inner exception stack trace ---
   at Microsoft.Build.Exceptions.InternalLoggerException.Throw(Exception innerException, BuildEventArgs e, String messageResourceName, Boolean initializationException, String[] messageArgs)
   at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseProjectStartedEvent(Object sender, ProjectStartedEventArgs buildEvent)
   at MSBuild.Logger.ForwardingLogger.ForwardEvent(Object sender, BuildEventArgs e)
   at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseProjectStartedEvent(Object sender, ProjectStartedEventArgs buildEvent)
##[error]Process 'msbuild.exe' exited with code '1'.

@anatolybolshakov Can this be prioritized as it is affecting a production pipeline?

[asklar]

CC @AgneLukoseviciute @dannyvv

[japj]

Is this a similar problem to #14863 due to logger ProjectTrees parent/child tracking?

[anatolybolshakov]

@japj hmm, not sure at the moment... But this one looks different

[anatolybolshakov]

Created PR for msbuildlogger update (blobstorage), waiting to be merged.

[anatolybolshakov]

Hi @rainersigwald the fix for this issue has been already rolled out - closing it at the moment as resolved. Please let us know if you have any other questions.