New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test Host randomly crashing with message "##[error]The active test run was aborted. Reason: Test host process crashed" #4819
Comments
@jakubch1 this looks like a CodeCoverage issue. Testhost just crashes and there is no error reported. Please have a look. |
@ibiglari you are using dynamic native instrumentation which is not fully stable. After doing above you should not see in logs such errors:
|
@jakubch1 so if I add |
@jakubch1 I'm afraid adding Linker command line as extracted from build pipeline logs: |
Yes. Static instrumentation is better and more reliable so we have it enabled by default but if Please try to configure below flags in runsettings:
Full runsettings samples: https://github.com/microsoft/codecoverage/blob/main/docs/configuration.md |
@jakubch1 Even after updating
|
@ibiglari could you please share logs? |
@ibiglari something is wrong as I don't see
inside logs. Can you double check that this was added into Did you configure:
inside |
Added
New logs attached. |
@ibiglari currently you are instrumenting only 1 file:
(inside |
@jakubch1 No joy. Still seeing the crash. Logs attached. Also, not sure what you mean by instrumenting only 1 file? Our test project is
vstestlog.log |
@ibiglari could you please try to run it without code coverage enabled? We should check if this is even coverage related. Based on log currently your build doesn't make any instrumentation. |
@jakubch1 I find that hard to believe, as I can see code coverage in SonarCloud. However, I will try and get back to you. |
@jakubch1 Turning off code coverage leads to successful execution of the test. Logs attached |
@ibiglari thank you for info. Could you please experiment with below 4 flags:
I expect if all flags are False you don't have issue. Then you could 1 by 1 enable and check. If you can share your binaries I can also make this testing on my side. You can share privately binaries here: https://developercommunity.visualstudio.com/home |
@jakubch1 I'm afraid I had a crash with all four set to false. |
@ibiglari any idea what is "Waiting for HQ Thread to Terminate..." ? |
@jakubch1 Yes we set a mutex and wait for a thread to terminate... That's one of the log messages from the test routine. Why? |
I saw this in Test Platform logs. Could you please try this: |
@jakubch1 Now this is funny. If I add The problem is, since this error happens randomly, it might be pure luck. I have to run the queue multiple times to ascertain whether a change in parameters causes a crash or not |
@jakubch1 Disregard previous comment. Test host crashed again, and this time this message was added to execution log:
How do I make this available to my DevOps build agent? Do I need to manually include the binary? |
@ibiglari please install proc dump (https://github.com/microsoft/vstest/blob/main/docs/troubleshooting.md#collect-process-dump-using-procdump-on-windows-ie-outofmemory) and set PROCDUMP_PATH when running your tests. |
@jakubch1 Found an easier way. However, this meant the test ran for 40 minutes instead of 9 minutes, and then the pipeline timed out. Is that to be expected when using |
@jakubch1 After adjusting job's timeout, I can confirm test host does not crash when proc dump is attached to it. However, once the test method is finished, there is a delay of about 30 minutes before test host returns and the pipeline can continue. I can only surmise that proc dump somehow changes internal state of test host. Any advise? |
@ibiglari in that case please remove
and after VSTest@2 and step:
then please reproduce crash issue and provide us dumps. Please try to repro with:
|
@jakubch1 Will advise. I was installing procdump using a PowerShell task:
I guess I need to remove it, right? |
@jakubch1 Just an update, I have not been able to replicate the crash for the past week. The random nature of this crash makes it rather hard to investigate, but as soon as it happens, I will provide the logs here. |
We have an MFC application and are using
Microsoft::VisualStudio::CppUnitTestFramework
for testing.VSTest@2
step is failing almost 50% of the time with the message ##[error]The active test run was aborted. Reason: Test host process crashed after the last step of the test routine is finished, as apparent from the log messages printed byMicrosoft::VisualStudio::CppUnitTestFramework::Logger::WriteMessage()
. The test successfully finishes when being executed directly via Visual Studio 17.8.4.Pipeline Step:
Steps to reproduce
Since this is a random occurrence, re-queuing the job usually resolves the issue.
Expected behavior
No crash during test execution
Actual behavior
Random crashes after last step of the test routine
Diagnostic logs
vstestlog.datacollector.24-01-11_08-49-42_80953_4.diag.log
vstestlog.diag.log
vstestlog.host.24-01-11_08-49-51_57026_4.diag.log
Environment
The text was updated successfully, but these errors were encountered: