NUnit.ConsoleRunner.NetCore executes test library under wrong runtime version

[MichalMucek]

I'm trying to execute a test library using NUnit Console 3.16.3 (NuGet, .NET Tool) on Windows Server 2019 Datacenter (10.0.17763, x64). There are multiple .NET SDK versions installed on the machine - 6.0.202, 7.0.100, 7.0.309, 8.0.100.

In use are NUnit 3.13.1, NUnit3TestAdapter 4.4.2, and SpecFlow.NUnit 3.9.74.

The TargetFramework of the library is either net7.0-windows10.0.19041.0 or net8.0-windows10.0.19041.0. Each time I execute one of the libraries, .NET 6.0.23 (greatest/latest of the other available 6.0.x) is used for runtime, which results in errors of missing assemblies for the targeted framework or the selected runtime. In the case of net7.0-windows10.0.19041.0, it can reach the point of finding the tests. It is unable to do even that for net8.0-windows10.0.19041.0.

The test library executes with no issues when I set up an environment with .NET 7 or 8 only (or when I make a few tweaks in the appropriate keys of the Windows Registry). Using global.json with a specific .NET SDK version doesn't work.

InternalTrace.9572.Org.App.Specs, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null (.NET 7).log
InternalTrace.9572 (.NET 7).log
Console output for net7.0-windows10.0.19041.0:

dotnet tool run dotnet-nunit .\Org.App.Specs.dll --trace=Debug
NUnit Console 3.16.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Thursday, January 11, 2024 11:44:21 AM

Runtime Environment
   OS Version: Microsoft Windows 10.0.17763
  Runtime: .NET 6.0.23

Test Files
    .\Org.App.Specs.dll


Errors, Failures and Warnings

1) SetUp Error : Motorola_DimetraE2ee_Specs_NUnitAssemblyHooks
System.IO.FileNotFoundException : Could not load file or assembly 'System.Collections.Specialized, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
   at System.Configuration.ConfigurationElement..ctor()
   at System.Configuration.ConfigurationElementCollection..ctor()
   at BoDi.ContainerRegistrationCollection..ctor()
   at TechTalk.SpecFlow.Configuration.ConfigurationLoader.GetDefault()
   at TechTalk.SpecFlow.Infrastructure.ContainerBuilder.CreateGlobalContainer(Assembly testAssembly, IRuntimeConfigurationProvider configurationProvider)
   at TechTalk.SpecFlow.TestRunnerManager.CreateTestRunnerManager(Assembly testAssembly, IContainerBuilder containerBuilder)
   at TechTalk.SpecFlow.TestRunnerManager.GetTestRunnerManager(Assembly testAssembly, IContainerBuilder containerBuilder, Boolean createIfMissing)
   at TechTalk.SpecFlow.TestRunnerManager.OnTestRunStart(Assembly testAssembly, IContainerBuilder containerBuilder)
   at Motorola_DimetraE2ee_Specs_NUnitAssemblyHooks.AssemblyInitialize() in C:\ApplicationRepo\Tests\App\Org.App.Specs\obj\Release\net7.0-windows10.0.19041.0\NUnit.AssemblyHooks.cs:line 20

Run Settings
    DisposeRunners: True
    InternalTraceLevel: Debug
    WorkDirectory: C:\net7.0-windows10.0.19041.0
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Failed
  Test Count: 678, Passed: 0, Failed: 678, Warnings: 0, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 0, Errors: 678, Invalid: 0
  Start time: 2024-01-11 11:44:21Z
    End time: 2024-01-11 11:44:22Z
    Duration: 0.916 seconds

Results (nunit3) saved as TestResult.xml

InternalTrace.3312.Org.App.Specs, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null (.NET 8).log
InternalTrace.3312 (.NET 8).log
Console output for net8.0-windows10.0.19041.0:

dotnet tool run dotnet-nunit .\Org.App.Specs.dll --trace=Debug
NUnit Console 3.16.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Thursday, January 11, 2024 11:44:51 AM

Runtime Environment
   OS Version: Microsoft Windows 10.0.17763
  Runtime: .NET 6.0.23

Test Files
    .\Org.App.Specs.dll


Errors, Failures and Warnings

1) Invalid : C:/net8.0-windows10.0.19041.0/Org.App.Specs.dll
Could not load type 'System.Runtime.CompilerServices.NullableContextAttribute' from assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

Run Settings
    DisposeRunners: True
    InternalTraceLevel: Debug
    WorkDirectory: C:\net8.0-windows10.0.19041.0
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2024-01-11 11:44:51Z
    End time: 2024-01-11 11:44:52Z
    Duration: 0.327 seconds

Results (nunit3) saved as TestResult.xml

[stevenaw]

Hi @MichalMucek
Unfortunately there are a few issues with the 3.16 version of the console runner. Since you have .NET8 installed could you please see if the prerelease of 3.17 works for you? That version should include .NET8 support among other fixes. https://github.com/nunit/nunit-console/releases/tag/V3.17.0-dev.3

Note that there is still a known issue related to projects requiring certain particular runtimes, with a workaround defined in #1383. If you hit the issue the workaround should work if your tests do not require a non-Windows OS.

[MichalMucek]

@stevenaw Thanks for responding.
The release version of 3.17.0 produces the same issue.

I submitted #1381 not only with the intention of making the NUnit console work in environments where the .NET SDK is not installed, but also with the hope that it will be a workaround for this issue.

[OsirisTerje]

@MichalMucek Have you worked through the comments in the #1374 and see if any of those might apply to you?

[MichalMucek]

@OsirisTerje Thanks for the tip.
I've tried tweaking the nunit3-console.runtimeconfig.json file which was located at .../.nuget/packages/nunit.consolerunner.netcore/3.17.0/tools/net6.0/any. Changing runtimeOptions.framework.version from 6.0.0 to 8.0.0 helped.

[MichalMucek]

Today, I installed NUnit.ConsoleRunner.NetCore 3.17.0 (.NET tool) on some other environment. There was a warning message (colored dark yellow), saying Format version is missing. This tool may not be supported in this SDK version. Contact the author of the tool. The used .NET SDK version was 8.0.200.

File nunit3-console.runtimeconfig.json was still pointing at framework version 6.0.0.

[OsirisTerje]

@MichalMucek I can confirm this, so should be fixed in #1396