Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node Crash When Logging TaskParameters #6379

Closed
benvillalobos opened this issue Apr 28, 2021 · 2 comments · Fixed by #6382
Closed

Node Crash When Logging TaskParameters #6379

benvillalobos opened this issue Apr 28, 2021 · 2 comments · Fixed by #6382
Assignees
@KirillOsenkov
Labels
bug needs-triage Have yet to determine what bucket this goes in.

Comments

@benvillalobos
Copy link
Member

Original Feedback Ticket: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1314638 (the "Move to Github") button failed)

Issue Description

Caused by #6155 /cc: @KirillOsenkov

Stack Trace

System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.

Server stack trace:

at System.Threading.Thread.InternalCrossContextCallback(Context ctx, IntPtr ctxID, Int32 appDomainID, InternalCrossContextDelegate ftnToCall, Object[] args)

at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatch(Byte[] reqStmBuff, SmuggledMethodCallMessage smuggledMcm, SmuggledMethodReturnMessage& smuggledMrm)

at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(IMessage reqMsg)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Build.Framework.IMetadataContainer.EnumerateMetadata()

at Microsoft.Build.Framework.ITaskItemExtensions.EnumerateMetadata(ITaskItem taskItem)

at Microsoft.Build.Framework.TaskParameterEventArgs.WriteMetadata(BinaryWriter writer, ITaskItem taskItem)

at Microsoft.Build.Framework.TaskParameterEventArgs.WriteItem(BinaryWriter writer, Object item)

at Microsoft.Build.Framework.TaskParameterEventArgs.WriteItems(BinaryWriter writer, IList items)

at Microsoft.Build.Framework.TaskParameterEventArgs.WriteToStream(BinaryWriter writer)

at Microsoft.Build.Shared.LogMessagePacketBase.WriteToStream(ITranslator translator)

at Microsoft.Build.Shared.LogMessagePacketBase.Translate(ITranslator translator)

at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)

Versions & Configurations

Build Number: 16.10.31213.239
Product Version: 16.10.0-preview-2.0
@benvillalobos benvillalobos added bug needs-triage Have yet to determine what bucket this goes in. labels Apr 28, 2021
@KirillOsenkov
Copy link
Member

ouch. Taking a look.

@KirillOsenkov KirillOsenkov self-assigned this Apr 28, 2021
KirillOsenkov added a commit that referenced this issue Apr 28, 2021
@KirillOsenkov
Snapshot Items of TaskParameters from another AppDomain
b67e3d9 
Fixes #6379

We have a situation where we are in a worker node and a task runs in a separate AppDomain and logs a TaskParameterEventArgs. Since logging is asynchronous there's a risk that by the time the node packet translator accesses the TaskParameterEventArgs.Items the AppDomain is already unloaded and we crash when trying to enumerate item metadata.

Detect that we're in another AppDomain and eagerly take a snapshot of task items with all metadata.
@KirillOsenkov KirillOsenkov mentioned this issue Apr 28, 2021
Merged
@KirillOsenkov
Copy link
Member

For validation I've also tried running this logic always (regardless of appdomains) and it worked fine.

rainersigwald added a commit that referenced this issue Apr 30, 2021
@rainersigwald
Merge pull request #6382 from dotnet/dev/kirillo/6379
fa96a2a 
Fixes #6379

We have a situation where we are in a worker node and a task runs in a separate AppDomain and logs a TaskParameterEventArgs. Since logging is asynchronous there's a risk that by the time the node packet translator accesses the TaskParameterEventArgs.Items the AppDomain is already unloaded and we crash when trying to enumerate item metadata.

Detect that we're in another AppDomain and eagerly take a snapshot of task items with all metadata.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KirillOsenkov KirillOsenkov

Labels
bug needs-triage Have yet to determine what bucket this goes in.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Snapshot Items of TaskParameters from another AppDomain dotnet/msbuild
2 participants
@KirillOsenkov @benvillalobos