NUnit3Runner does not work with dot net core 2.1

[soroshsabz]

ITNOA

What You Are Seeing?

When I was use NUnit3(testProject.DllFilePath, settings) with dot net core 2.1 project I got many exception.

What is Expected?

NUnit3 work properly with any type of NUnit test project

What version of Cake are you using?

1.0.0

Are you running on a 32 or 64 bit system?

64 bit system

What environment are you running on? Windows? Linux? Mac?

Windows

Are you running on a CI Server? If so, which one?

No

How Did You Get This To Happen? (Steps to Reproduce)

Create dot net core 2.1 project with NUnit3 template from visual studio 2019 and then try run cake file with below command

NUnit3(testProject.DllFilePath, settings)

Output Log

Errors, Failures and Warnings

1) Error :
NUnit.Engine.NUnitEngineException : An exception occurred in the driver while loading tests.
  ----> System.IO.FileNotFoundException : Could not load file or assembly 'nunit.framework, Version=3.12.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' or one of its dependencies. The system cannot find the file specified.
--NUnitEngineException
An exception occurred in the driver while loading tests.

Server stack trace:
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
--
FileNotFoundException
Could not load file or assembly 'nunit.framework, Version=3.12.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)

[devlead]

For .NET Core / .NET 5 projects I would recommend using the .NET CLI Test command using the DotNetCoreTest alias.

[soroshsabz]

My project is dot net core 2.1

[devlead]

The .NET Core / 5 SDK exists for

• .NET 5.0
• .NET Core 3.1
• .NET Core 2.1

You can install SDKs side by side, and they're often installed by the Visual Studio installer, so you've probably already have the SDK with dotnet command installed.

If not you can either install via Visual Studio installer or get.dot.net.

[soroshsabz]

@devlead I found if I use below command

			var settings = new VSTestSettings
			{
				HandleExitCode = _ => true,
			};

			VSTest(testProject.DllFilePath, settings)

our test is running, but I have one problem with this, that I cannot specify results path like NUnit3Settings, Did you have any idea to resolve my problem?

[soroshsabz]

@devlead I cannot use DotNetCoreTest because my project dependencies does not work with dotnet tool so I have to use alternative way. like VSTest or NUnit3 or any other things that you can say.

thanks for response

[devlead]

The VSTest alias VSTestSettings has a Logger property.

You could set it to something like Logger = "trx;LogFileName=VsTestResults.xml"

[soroshsabz]

@devlead Thanks a lot for response, but your solution does not specify result directory

as you can see https://docs.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019

/ResultsDirectory:[path]	Test results directory will be created in specified path if not exists.Example: /ResultsDirectory:

Is there anyway to set ResultDirectory?

[soroshsabz]

@devlead I use ArgumentCustomization for resolve my problem, But I think this issue is not close, because NUnit3 has Bug for working with dot net core 2.1

I think it is sound good to add ResultDirectory to VSTestSettings

thanks a lot for helping me

[soroshsabz]

@devlead HandleExitCode does not work for VSTest, What did i do can?

I set HandleExitCode = _ => true

but I got

##[error]Cake.exe : Test Run Failed.

I found one link maybe can help

https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019#runconfiguration-element

as you can see in link this tools have settings that name TreatNoTestsAsError but I don't know, is there any help for you on not?

Update: I found my problem, HandleExitCode is correct but my problem is VSTest write standard error when any test failed

[soroshsabz]

Link to #3127

[augustoproiete]

@soroshsabz Could you provide a reproducible example (Visual Studio project + build.cake file) that causes this error?

I am not able to repro the error you mention when setting HandleExitCode = _ => true

var settings = new VSTestSettings
{
    HandleExitCode = _ => true,
};

VSTest(@"C:\augustoproiete\repro-cake-3123\bin\Debug\netcoreapp3.1\UnitTestProject1.dll", settings);

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
            throw new InvalidOperationException("boom");
        }
    }
}

Compiling build script...
Resolved tool to path C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/Common7/IDE/CommonExtensions/Microsoft/TestWindow/vstest.console.exe
Executing: "C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/Common7/IDE/CommonExtensions/Microsoft/TestWindow/vstest.console.exe" "C:/augustoproiete/repro-cake-3123/bin/Debug/netcoreapp3.1/UnitTestProject1.dll"
Microsoft (R) Test Execution Command Line Tool Version 16.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Failed TestMethod1 [17 ms]
  Error Message:
   Test method UnitTestProject1.UnitTest1.TestMethod1 threw exception:
System.InvalidOperationException: boom
  Stack Trace:
      at UnitTestProject1.UnitTest1.TestMethod1() in C:\augustoproiete\repro-cake-3123\UnitTestProject1\UnitTest1.cs:line 12


Test Run Failed.
Total tests: 1
     Failed: 1
 Total time: 0.7013 Seconds

C:\augustoproiete\repro-cake-3123>echo %ERRORLEVEL%
0

[soroshsabz]

@augustoproiete As I update my comment, I found my problem, and HandleExitCode is correct

sorry for that :(

[augustoproiete]

@soroshsabz No worries. Moving this one to the GitHub Discussions area then.