Builds started very often failing on Travis with the Child node "19" exited prematurely. error

[moozzyk]

Approximately half of our Linux builds we run on Travis started recently failing with:

MSBUILD : error MSB4166: Child node "19" exited prematurely. Shutting down. Diagnostic information may be found in files in the temporary files directory named MSBuild_*.failure.txt.

Here is an example of a build that failed like this: https://travis-ci.org/aspnet/SignalR/jobs/238567957

The error says to look for MSBuild_*.failure.txt files in the temporary files directory but there are not there. The only files in /tmp are:

$ ls /tmp/*
/tmp/clr-debug-pipe-3871-2510650-in   /tmp/dotnet.u9E2nmQWg
/tmp/clr-debug-pipe-3871-2510650-out  /tmp/dotnet.uFJfXhhNI
/tmp/dotnet.86YK7j2WA		      /tmp/dotnet.XVL3GkXqg
/tmp/dotnet.kn40yYkhz		      /tmp/hosts_127_0_0_1
/tmp/dotnet.prNGfwjzG		      /tmp/hosts_sans_127_0_0_1

Is the /tmp directory the correct place to find the MSBuild_*.failure.txt. files? If so, the files are not there so, how can I figure out what the issue is and why MSBuild crashed?

[smitpatel]

#2168 seems to be cause of the issue. Removing binary logger made this problem go away.
Though error message here is vary unusual.
The content of failure.txt file are like this

(TID 15) 636324668919586994 +  189661.4ms: COMMUNICATIONS ERROR (HRC) Node: 21 Process: 3482 Bytes Read: 0 Expected: 5
(TID 15) 636324668921227348 +  164.0354ms:    Child Process 3482 has exited.

Which does not provide much info either.

[rainersigwald]

Closing as duplicate of #2168.

[smitpatel]

I believe the error message should be better than this. It does not need to be directly shown to user but log files were also very un-useful.

[rainersigwald]

@smitpatel Does #1038 cover what you'd like to see? Agreed that this is hard to diagnose, but there's only so much we can do from one process when we observe that another process has crashed.

[smitpatel]

#1038 would not cover it fully.
There are 2 issues,

1. How to get the log files (MSBuild*failure.txt) from a CI server? Make "child node exited prematurely" more diagnosable from CI like Jenkins #1038 covers that by logging the content to console perhaps. We also had to jump through hoops to get the log files from the CI server. So it would be good addition to have.
2. How useful is the content of log files? This is what I am looking for. If by some means I retrieve the logs, is there sufficient information for me to understand the issue which caused it.

Agreed that this is hard to diagnose, but there's only so much we can do from one process when we observe that another process has crashed.

You are quite right here. I am not sure what information a process has when it crashes. But I would be happy if somehow it is indicated that process ran out of memory. It gives some direction in investigating what is going wrong.

[KirillOsenkov]

Should be fixed by #6155