Test Explorer: No suitable test runtime provider was found

[bytewizer]

Tool

Visual Studio extension

Description

After installing Microsoft Visual Studio Professional 2022 (64-bit) Version 17.3.1 and updating to the latest .NET nanoFramework Extension Version: 2022.2.0.33 Test Explorer will fail with a 'No suitable test runtime provider was found' error message.

Failed Log:

Log level is set to Informational (Default).
Connected to test environment '< Local Windows Environment >'
Error opening test data: Data schema version was 30, expected 31.
Test data was cleared upon opening.
Test data store opened in 0.017 sec.
Loaded 0 test records from the solution cache in 0.222 sec.
Starting test discovery for requested test run
Using automatically detected runsettings file(s). To learn more visit https://aka.ms/vs-runsettings.
Tests in projects that reference runsettings file 'C:\Repos\nanoFramework.DependencyInjection\tests\nano.runsettings' will not be discovered in parallel because it specifies the value for 'MaxCpuCount' as '1'.
========== Starting test discovery ==========
Test run detected DLL(s) which would use different framework and platform versions. Following DLL(s) do not match current settings, which are .NETFramework,Version=v4.0 framework and X64 platform.
NFUnitTest.dll would use Framework .NETnanoFramework,Version=v1.0 and Platform X64.
Go to https://aka.ms/tp/vstest/multitargetingdoc?view=vs-2019 for more details on managing these settings.
[nanoTestAdapter]: Getting ready to discover tests...
[nanoTestAdapter]: Settings parsed
[nanoTestAdapter]:   New file processed: C:\Repos\nanoFramework.DependencyInjection\tests\bin\Debug\NFUnitTest.dll
[nanoTestAdapter]:   Adding 57 new tests
[nanoTestAdapter]: Finished adding files
No suitable test runtime provider was found:
Skipping source: C:\Repos\nanoFramework.DependencyInjection\nanoFramework.DependencyInjection\bin\Debug\nanoFramework.DependencyInjection.dll (.NETnanoFramework,Version=v1.0, X64)

Successful Log:

Log level is set to Informational (Default).
Connected to test environment '< Local Windows Environment >'
Test data store opened in 0.017 sec.
Loaded 0 test records from the solution cache in 0.213 sec.
Starting test discovery for requested test run
Using automatically detected runsettings file(s). To learn more visit https://aka.ms/vs-runsettings.
Tests in projects that reference runsettings file 'C:\Repos\nanoFramework.DependencyInjection\tests\nano.runsettings' will not be discovered in parallel because it specifies the value for 'MaxCpuCount' as '1'.
========== Starting test discovery ==========
Test run detected DLL(s) which would use different framework and platform versions. Following DLL(s) do not match current settings, which are .NETFramework,Version=v4.0 framework and X64 platform.
NFUnitTest.dll would use Framework .NETnanoFramework,Version=v1.0 and Platform X64.
Go to https://aka.ms/tp/vstest/multitargetingdoc?view=vs-2019 for more details on managing these settings.
[nanoTestAdapter]: Getting ready to discover tests...
[nanoTestAdapter]: Settings parsed
[nanoTestAdapter]:   New file processed: C:\Repos\nanoFramework.DependencyInjection\tests\bin\Debug\NFUnitTest.dll
[nanoTestAdapter]:   Adding 57 new tests
[nanoTestAdapter]: Finished adding files
========== Test discovery finished: 57 Tests found in 1.6 sec ==========
Using automatically detected runsettings file(s). To learn more visit https://aka.ms/vs-runsettings.
========== Starting test run ==========
Test run detected DLL(s) which would use different framework and platform versions. Following DLL(s) do not match current settings, which are .NETFramework,Version=v4.0 framework and X64 platform.
NFUnitTest.dll would use Framework .NETnanoFramework,Version=v1.0 and Platform X64.
Go to https://aka.ms/tp/vstest/multitargetingdoc?view=vs-2019 for more details on managing these settings.
[nanoTestAdapter]: Getting ready to run tests...
[nanoTestAdapter]: Settings parsed
[nanoTestAdapter]: Test sources enumerated
[nanoTestAdapter]: Test group is 'C:\Repos\nanoFramework.DependencyInjection\tests\bin\Debug\NFUnitTest.dll'
[nanoTestAdapter]: Setting up test runner in *** nanoCLR WIN32***
[nanoTestAdapter]: Timeout set to 120000ms
[nanoTestAdapter]: Processing assemblies to load into test runner...
[nanoTestAdapter]: Parameters to pass to nanoCLR: < -load "C:\Repos\nanoFramework.DependencyInjection\tests\bin\Debug\mscorlib.pe" -load "C:\Repos\nanoFramework.DependencyInjection\tests\bin\Debug\nanoFramework.DependencyInjection.pe" -load "C:\Repos\nanoFramework.DependencyInjection\tests\bin\Debug\nanoFramework.TestFramework.pe" -load "C:\Repos\nanoFramework.DependencyInjection\tests\bin\Debug\nanoFramework.UnitTestLauncher.pe" -load "C:\Repos\nanoFramework.DependencyInjection\tests\bin\Debug\NFUnitTest.pe">
[nanoTestAdapter]: Found nanoCLR Win32: C:\Repos\nanoFramework.DependencyInjection\packages\nanoFramework.TestFramework.1.0.209\lib\net48\nanoFramework.nanoCLR.exe
[nanoTestAdapter]: Launching process with nanoCLR (from C:\Repos\nanoFramework.DependencyInjection\packages\nanoFramework.TestFramework.1.0.209\lib\net48\nanoFramework.nanoCLR.exe)
[nanoTestAdapter]: nanoCLR started @ process ID: 7204
[nanoTestAdapter]: Parsing test results...
[nanoTestAdapter]: 
.NET nanoFramework nanoCLR WIN32 v1.8.0.440+7de2295208

How to reproduce

1. Fork the 'nanoFramework.DependencyInjection' repo and run test explorer. This should work as expected and all tests will pass.
2. Add the following code to the project. (Any major changes to the project code will break test explorer)

namespace nanoFramework.DependencyInjection
{
    internal static class BreakMe
    {
        public static bool IsBroken()
        {
            return false;
        }
    }
}

3. Run test explorer again and you will get a 'No suitable test runtime provider was found' error message.

Expected behaviour

This should work as expected and all tests will pass.

Screenshots

No response

Aditional context

Microsoft Visual Studio Professional 2022 (64-bit) Version 17.3.1
.NET nanoFramework Extension Version: 2022.2.0.33

[Ellerbach]

I can reproduce, something weird somewhere! Tried to clean the solution, remove the addition, rebuild and it's just the very first time it seems to work. Will continue to investigate a bit to try to find, at least some way to go around it.

[Ellerbach]

@bytewizer, found a way to have it working all the time. See below, give it a try and let me know.
Also, in your test nfproj, you have a reference on TestContainer, please remove that as well.
Change the target platform into the nano.runsettings to:

<TargetFrameworkVersion>net48</TargetFrameworkVersion>

seems like Framework40 or net40 is a "bit" old now and for some reasons doesn't work properly all the time.

@josesimoes we may want to test this change on other libs.

[bytewizer]

I wish I was celebrating too. Unfortunately, this fix is not working for me. I have following the exact step and copied the string above into the nano.runsettings. I have cleaned the project, deleted system created files, restarted my system and several combinations of this. However, I’m getting the exact same message. I have also tried it with and without the TestContainer removed. I’m really not sure why this is not working.

[josesimoes]

No enterily sure about that... Because that's specifying the framework of the test adapter, not the assembly under test.

[bytewizer]

I'm starting to feel like a troublemaker. Unfortunately, now when I fork the project with the updated bits, it fails the very first time I run it with the exact same message. I did double check that I see the change you made to the project are in fact the files I'm working with. I've tried to delete/clean every visual studio temp file I can find (or google recommends). I even tried to make a new test file and copied all of the code over to that test file one at a time and nothing looks to make any sort of difference on it. If you think doing a clean install of windows 10 and a fresh Visual Studio install, it's something I would consider this weekend.

[Ellerbach]

@bytewizer, before going for the Armageddon solutin, if you create a brand new project, a simple lib, add a test project, will this very simple setup works fine?

[josesimoes]

@bytewizer did you manually erase the .vs folder? That's where the details and configs go.
Also, sometimes you have to open the solution again so it picks up the correct test adapter.

[bytewizer]

@Ellerbach Ok, here is something interesting. If I create/add a test project from a solution it works fine. However, If I open Visual Studio and chose 'create a new project' and select Unit Test Project(.NET nanoFramework) I get the following error:

From this I tried to uninstall the extension and reinstall it but same results trying to both create a new test project and running the tests.

@josesimoes I'm not seeing a .vs folder as I've been deleting the 'nanoframework.DependencyInjector' folder and doing a new fork this way I'm only bringing the exact files from the repo. Also, I have tried to open and close the project but unfortunately no success.

[josesimoes]

@bytewizer that error is with the VSIX packaging. Will look ASAP!

As for the .vs solution it's local, created by VS when opening a solution. Not under version control.

[josesimoes]

@bytewizer new VS extension published with the above fixed.

[Ellerbach]

@bytewizer thanks again for your help and for the investigation! Simple things are usually great to catch big issues :-D

[josesimoes]

More details on this:

• Probably related with [rel/17.3] Do not match .NET Standard to Dotnet testhost runner microsoft/vstest#3958
• We happened to update to 17.3 on our the previous release.
• There is now a patch for this in 17.3.1

I'll be updating our test framework to reference that new version and lets see if this get's working smoothly again in Test Explorer.

[josesimoes]

nanoFramework Test Framework v2.0.42 update to reference VSTest v17.3.1.
I can run the tests locally on VS 2022 17.3.2.

Note that for auto detection of the Test Adapter the run settings file has to be named .runsettings and be placed at solution folder.

Feel free to reopen if there are any further issues.