Tests suddenly failing to run in Azure DevOps pipeline - No suitable test runtime provider was found for any source in this run

[JLRishe]

Description

I have been using the Test Assemblies task in several different pipelines related to a single project. This has been working without issue for years.

Today, I've encountered an issue where this task fails with an error "No suitable test runtime provider was found for any source in this run".

At first I was able to get the build to succeed just by re-running it, but after a few hours since the first occurrence I'm finding that the failure occurs no matter how many times I run the build.

I have not recently changed anything about my project that would affect this. And the fact that the build step succeeded intermittently after the issue started occurring, and then stopped working altogether, suggests that this is likely the result of some new change being rolled out in Azure DevOps.

Do you know of any such change being rolled out, and if so, what is the best way to deal with this situation?

Steps to reproduce

I do not have steps that would be reproducible in general. I simply initiate the pipeline and it fails 50-80% of the time.

Expected behavior

The Test Assemblies task runs the tests.

Actual behavior

Error:

##[error]No suitable test runtime provider was found for any source in this run.

Diagnostic logs

Attached:
VSTest Log.txt

Environment

Azure DevOps

Agent: windows-latest
Test Assemblies task - Test platform version: Latest

[froeschel]

Hello.
I have the exact same issue. I'm creating a new asp.net core web api project using .net 6. Then I add a test project with one simple test. After that i commit to the repository, create a new pipline and run it. It fails with exact the same error as described in the previous post.
As a work around I'm run dotnet test from cmd line. This is shown here: https://learn.microsoft.com/da-dk/azure/devops/pipelines/ecosystems/dotnet-core?view=azure-devops&tabs=dotnetfive#run-your-tests

[nohwnd]

@JLRishe do you please have log of a successful run?

[nohwnd]

The failure is on 17.6.0 vstest console, you can update it by using the TestPlatform installer task, and install Latest, which will choose 17.6.1.

And then on your VSTest task you choose: Version installed by installer.

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/visual-studio-test-platform-installer-v1?view=azure-pipelines

[JLRishe]

@nohwnd Thank you! After making that change, my pipelines seem to be working.

Hopefully @froeschel can chime in about how it goes for him, but it seems like the given solution is a good fix.

[froeschel]

If I try this approach i get an exception 'System.InvalidOperationException: The provided manager was not found in any slot.'

My pipeline looks like this:

# ASP.NET Core (.NET Framework)
# Build and test ASP.NET Core projects targeting the full .NET Framework.
# Add steps that publish symbols, save build artifacts, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:
- master

pool:
  vmImage: 'windows-latest'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

steps:
- task: NuGetToolInstaller@1

- task: NuGetCommand@2
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactStagingDirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

# Visual Studio test platform installer v1
# Acquire the test platform from nuget.org or the tool cache. Satisfies the ‘vstest’ demand and can be used for running tests and collecting diagnostic data using the Visual Studio Test task.
- task: VisualStudioTestPlatformInstaller@1
  inputs:
  # Package settings
    packageFeedSelector: 'nugetOrg' 
    versionSelector: 'latestStable'

- task: VSTest@2
  inputs:
    vsTestVersion: 'toolsInstaller'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

[nohwnd]

After making that change, my pipelines seem to be working.

Thanks for confirming that.

If I try this approach i get an exception 'System.InvalidOperationException: The provided manager was not found in any slot.'

This is a whole another issue, which happens after teshost fails to start up: The issue is described here, #4469 There should be additional errors before you see this message, are there?

[froeschel]

thanks.

yes, there are: The first error in the log is: Testhost process for source(s) 'D:\a\1\s\test-appTests\bin\Release\net6.0\testhost.dll' exited with error: A fatal error was encountered. The library 'hostpolicy.dll' required to execute the application was not found in 'C:\Program Files\dotnet'.

[nohwnd]

@froeschel Is your test project referencing Microsoft.NET.Test.SDK? If yes, which version?

[froeschel]

I used the VS functionality where you click on a method and say. Add unit tests. This creates a test project with following references.

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.10" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.10" />
    <PackageReference Include="coverlet.collector" Version="3.1.2" />
  </ItemGroup>

so not 17.6.1....

[nohwnd]

Let me try with that to see if I can repro.

[froeschel]

Great, thanks for the help! I used the standard VS template for the project. ASP.NET core Web API, which creates this weather forecast sample app (just a single controller).

[nohwnd]

Okay, I am seeing what you are seeing. A way to unblock yourself is to use a filter that is matching your test dlls naming.

Typically you have something like this:

      **\*test*.dll
      !**\*TestAdapter.dll
      !**\obj\**

But your tests probably match this naming pattern.

**\bin\**\*Tests.dll
**\bin\**\*Test.dll

[froeschel]

@nohwnd thanks again!
I have tried to update my filter based on your suggestions and it works!

[JLRishe]

@nohwnd

Hello, I'm now encountering a new error that seems to be a result of using the suggested workaround.
My pipelines that were working a few days ago with the workaround are now failing.

Is there any ETA for when the original issue that caused these problems will be fixed?.

Error:

##[error]TestExecution encountered unexpected failures System.Exception: Invalid version string for TestPlatformVersion 17.7.0-preview.23280.1+94103c3f3198fe6f1405e1cb8fe486d6cd7132fd
   at Microsoft.VisualStudio.TestService.Common.TestPlatformVersion..ctor(String versionString)
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.TestPlatformVersionFinder.IdentifyTestPlatformVersion(String vsTestConsoleExeParentFolder, Boolean debugLogs)
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.InputValidator.PopulateEnvironmentVariables()
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.InputValidator.ValidateAndFormatInputs()
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.Program.Main(String[] args)

Unhandled Exception: System.Exception: Invalid version string for TestPlatformVersion 17.7.0-preview.23280.1+94103c3f3198fe6f1405e1cb8fe486d6cd7132fd
   at Microsoft.VisualStudio.TestService.Common.TestPlatformVersion..ctor(String versionString)
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.TestPlatformVersionFinder.IdentifyTestPlatformVersion(String vsTestConsoleExeParentFolder, Boolean debugLogs)
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.InputValidator.PopulateEnvironmentVariables()
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.InputValidator.ValidateAndFormatInputs()
   at Microsoft.VisualStudio.TestService.AgentExecutionHost.Program.Main(String[] args)
##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
##[error]Error: The process 'D:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.220.0\Modules\DTAExecutionHost.exe' failed with exit code 3762504530
##[error]Vstest failed with error. Check logs for failures. There might be failed tests.

Logs:
Test Platform Installer Log.txt
TestAssemblies Log.txt

[nohwnd]

@JLRishe this was reported in #4544 and mitigated, please re-run your pipeline, it should resolve 17.6.2 if you are using the default package feed (nuget).

Those issues are not related.

[JLRishe]

@nohwnd Hello! Was the root cause of this issue resolved such that I can revert to not using the Test Platform Installer workaround?

[nohwnd]

Yes, I believe so. If you are using windows-latest image it will be on 17.7.0 at least.

[JLRishe]

@nohwnd Thank you. It seems to be working!