Azure Devops Task TimeOut/Stuck after VSTest v17.2.0 Release

[rkapoor028]

Description

IMPORTANT: if the defect is reproduced only in a workflow from within the Visual Studio IDE then do not report the issue here - instead, please report it using Visual Studio's "Send Feedback" option that can be accessed from the Help menu OR using this link https://developercommunity.visualstudio.com.

After the recent VSTest v17.2.0 release this morning, Azure Devops VsTest task is failing with the below error:

Starting test execution, please wait...
11501.6468
11504.7343
11506.4788
A total of 1 test files matched the specified pattern.
##[error]vstest.console process failed to connect to testhost process after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

VSTest@2 Task settings

Test selector : Test assemblies
Test filter criteria : (TestCategory!=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX)
Search folder : C:\a\1\s
Action when minimum tests threshold not met : donothing
Minimum tests expected to be run: 0
Run settings file : C:\a\1\s\UnitTests\NUnit.runsettings
Run in parallel : true
Run in isolation : false
Path to custom adapters : C:\a\1\s
Other console options : null
Code coverage enabled : true
Diagnostics enabled : false
Rerun failed tests: true
Rerun failed tests threshold: 50
Rerun maximum attempts: 5
SystemVssConnection exists true
Run the tests locally using vstest.console.exe

For a defect reproducable from the vstest command line, describe the issue you've observed.

Steps to reproduce

What steps can reproduce the defect?
Please share the setup, commandline for vstest.console, sample project, target
framework etc.

Expected behavior

Share the expected output

**************** Starting test execution *********************
C:\hostedtoolcache\windows\VsTest\17.1.0\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "@C:\a\_temp\3apnc0em5xm.tmp"
Microsoft (R) Test Execution Command Line Tool Version 17.1.0
Copyright (c) Microsoft Corporation.  All rights reserved.
vstest.console.exe "C:\a\1\s\Web\bin\UnitTests.dll"
/TestCaseFilter:"(TestCategory!=XXXXXXXXXXXXXXXXXXXXXXXXXXXX)"
/Settings:"C:\a\_temp\chb3dhz2pce.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\a\1\s"
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collection : Unable to find a datacollector with friendly name 'Test Impact'.
Data collection : Could not find data collector 'Test Impact'
Multiple versions of same extension found. Selecting the highest version.
  NUnit3.TestAdapter : 4.2.0.0
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in C:\a\1\s\Web\bin\UnitTests.dll

Actual behavior

What is the behavior observed?

**************** Starting test execution *********************
C:\hostedtoolcache\windows\VsTest\17.2.0\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "@C:\a\_temp\n3mbi5dbrdk.tmp"
Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.
vstest.console.exe "C:\a\1\s\Web\bin\UnitTests.dll"
/TestCaseFilter:"(TestCategory!=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX)"
/Settings:"C:\a\_temp\vnvesm1zx1o.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\a\1\s"
Starting test execution, please wait...
11501.6468
11504.7343
11506.4788
A total of 1 test files matched the specified pattern.
##[error]vstest.console process failed to connect to testhost process after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.
Microsoft (R) Coverage Collection Tool Version 16.0.30319.200


Copyright (c) Microsoft Corporation.  All rights reserved.




Results File: C:\a\_temp\.\TestResults\TestResults\AzDevOps_WinVM00018Y_2022-05-11_14_03_37.trx
Attachments:
  C:\a\_temp\TestResults\TestResults\7416a6b5-19eb-4e2a-92c4-7f2ff6ae7e82\AzDevOps_WinVM00018Y_2022-05-11.14_01_53.coverage
##[error]Test Run Aborted.
Vstest.console.exe exited with code 1.
**************** Completed test execution *********************
Test results files: C:\a\_temp\TestResults\TestResults\AzDevOps_WinVM00018Y_2022-05-11_14_03_37.trx
No Result Found to Publish 'C:\a\_temp\TestResults\TestResults\AzDevOps_WinVM00018Y_2022-05-11_14_03_37.trx'.
Created test run: 1468924
Publishing test results: 0
Publishing test results to test run '1468924'.
TestResults To Publish 0, Test run id:1468924
Published test results: 0
Publishing Attachments: 2
Failed tests: 0; Total tests: 0;
************* Disabling rerun for failed tests as some conditions were not met ************
Could not invoke Code coverage merge for Build ID 142544
Execution Result Code 1 is non zero, checking for failed results
Completed TestExecution Model...
##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
##[error]Error: The process 'C:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.203.0\Modules\DTAExecutionHost.exe' failed with exit code 1
##[error]Vstest failed with error. Check logs for failures. There might be failed tests.

Diagnostic logs

Please share test platform diagnostics logs. Instructions to collect logs are here.
The logs may contain test assembly paths, kindly review and mask those before sharing.

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console

Operation System: Windows Server 2022 DataCenter
Vstest version:

Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.
vstest.console.exe "C:\a\1\s\Web\bin\UnitTests.dll"

[rkapoor028]

@Haplois Wanted to flag this. We are facing this issue since the 17.2.0 release on our Azure Devops CI pipeline. Please suggest, if there is a workaround for this.

[shueybubbles]

I'm getting a different but just as useless behavior. It's supposed to retry on test failure, but it doesn't retry. The job gets marked as Failed in ADO but no test failures are logged.

2022-05-17T19:22:37.2492850Z Test results files: D:\a\_work\_temp\TestResults\sqlbld0_19112417c000000_2022-05-17_19_03_56.trx
2022-05-17T19:22:38.7079965Z Created test run: 1249863408
2022-05-17T19:22:38.7082970Z Publishing test results: 551
2022-05-17T19:22:38.8338085Z Publishing test results to test run '1249863408'.
2022-05-17T19:22:38.8362912Z TestResults To Publish 551, Test run id:1249863408
2022-05-17T19:22:38.8457362Z Test results publishing 551, remaining: 0. Test run id: 1249863408
2022-05-17T19:22:39.6506524Z Published test results: 551
2022-05-17T19:22:39.6511984Z Publishing Attachments: 2
2022-05-17T19:22:41.4318355Z Failed tests: 0; Total tests: 551;
2022-05-17T19:22:41.4319219Z ************* Disabling rerun for failed tests as some conditions were not met ************
2022-05-17T19:22:41.7769077Z Execution Result Code 1 is non zero, checking for failed results
2022-05-17T19:22:41.8336918Z Completed TestExecution Model...
2022-05-17T19:22:41.8996458Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2022-05-17T19:22:41.9000616Z ##[error]Error: The process 'D:\a\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.203.0\Modules\DTAExecutionHost.exe' failed with exit code 1
2022-05-17T19:22:41.9005269Z ##[error]Vstest failed with error. Check logs for failures. There might be failed tests.
2022-05-17T19:22:41.9104106Z ##[section]Finishing: PR ADO OnPrem 160 Edge merge

[luizhennemann]

I am getting the same error. Any updates on this issue?
Once I updated the packages bellow to version 17, DevOps build broke.

[Evangelink]

Hey there, sorry for the long await. I am wondering if you are still experiencing issues with newer versions of VSTest (17.5 or 17.6)?

[shueybubbles]

@Evangelink Heh heh now we're running into this one microsoft/azure-pipelines-tasks#18309

[Evangelink]

Thanks for the reply! Based on the log, issue is on AzDo side so your ticket is in the right repo.

I will move forward by closing this one.