Allow xunit, nunit and other frameworks to run with the new runner

[nohwnd]

Summary

MSTest runner is a new lightweight runner for MSTest tests. Please vote below if you'd like the same experience for other frameworks. And optionally share why, and which framework, please.

https://devblogs.microsoft.com/dotnet/introducing-ms-test-runner

[Rand-Random]

Maybe I am missing something, but why not?
Sure it’s more work, but isn’t it worth the additional work?

Currently, it feels like if Entity Framework only supported MSSQL and is asking do you want support for eg MySQL?

[baronfel]

Big thumbs-up here for XUnit support - this is going to be necessary to start porting large chunks of the dotnet/* repositories onto this new runner.

[Evangelink]

@baronfel We just need to provide enough new nice features for you to want to migrate to MSTest 😉

[neilcawse]

Would be very useful to support xunit for us and many customers - it would save a lot on wasted CPU cycles in CI/CD

[Daniel-Svensson]

Would love to see if it can give a similar boost to nunit tests

[pieteckhart]

+1 for XUnit

[nicolasmohamed]

+1 NUnit

[SymbioticKilla]

@baronfel We just need to provide enough new nice features for you to want to migrate to MSTest 😉

Totally agree! The migration from xUnit to MSTest wasn't a hard job. There are just some nice-to-have features in xUnit that save some code lines.

[thomhurst]

I'm in the process of writing my own framework currently.

So is the new runner locked down to just MSTest at the present time?

I'd definitely like to make use of the new runner, especially with things like performance improvements over VSTest.

[Evangelink]

So is the new runner locked down to just MSTest at the present time?

It is advertised only as MSTest runner but the code is open to any custom framework.

The only issues for non-MSTest framework at the moment are:

• runner/platform code lives in MSTest (specifically this folder: https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform)
• there isn't yet any technical documentation available for "extensions"/"framework" authors.

For this last point, if your project is OSS I can definitely help you and you can already start getting some inspiration from there draft PRs I have open:

• Initial support of Microsoft.Testing.Platform nunit/nunit3-vs-adapter#1153
• Add support for Microsoft.Testing.Platform YoloDev/YoloDev.Expecto.TestSdk#152

[thomhurst]

Thanks @Evangelink ! Those PRs are very helpful.

Question, they both seem to be a VS bridge (which makes sense if we're trying to keep backwards compatibility.)

Since my framework is still a WIP, I don't care much about this.

So, is there a non-bridged mode? If so I guess I'll have to wait for documentation, as I'd have to rewrite the VSTest stuff that scans assemblies and generates test cases.

And if there is a non-bridged mode, does that, or will it, come with support for dotnet test and test explorer? I assume it would because these are quite core to the testing experience currently, but those PRs had comments that those features were only made available by that bridging package.

Thanks!

[Evangelink]

Yes there is a non-bridged mode, this is actually the default but to move the masses (MSTest, xUnit and NUnit) we had to bring this compat layer :)

If so I guess I'll have to wait for documentation,

We will craft a dummy sample (tomorrow) that show how to create and plug a custom framework and how to return some state. This can be easily done and would be useful for when we build the technical doc.

And if there is a non-bridged mode, does that, or will it, come with support for dotnet test and test explorer? I assume it would because these are quite core to the testing experience currently, but those PRs had comments that those features were only made available by that bridging package.

Yes definitely! The support of dotnet test will be brought by Microsoft.Testing.Platform.MSBuild package and the support of VS and VS Code Test Explorers is coming. For VS Code Test Explorer, it's already available publicly in the last preview version of C# Dev Kit. For VS Test Explorer, there was a missed part so it will come in VS 17.10 Preview 2.

[thomhurst]

Amazing! Thanks for the prompt replies and look forward to any coming samples and docs!

From what I could see when navigating the code base, this looks like a very welcome improvement.

From recently creating a VSTest adapter, the code felt very legacy and convoluted, and very temperamental around "FullyQualifiedName"s or specific TestCaseProperty's!

Glad to see there's a push for a better testing experience!

[Evangelink]

Dear .NET community, thank you all for your interest and feedback on our new platform!

I'll move forward by closing this ticket and invite you to track the respective test framework tickets:

• For xUnit: Microsoft Testing Platform for xUnit xunit/visualstudio.xunit#402 (related draft PR: Add initial support for Microsoft.Testing.Platform xunit/visualstudio.xunit#403)
• For NUnit: Microsoft Testing Platform for NUnit nunit/nunit3-vs-adapter#1152 (related draft PR: Initial support of Microsoft.Testing.Platform nunit/nunit3-vs-adapter#1153)
• For Expecto: Add support for Microsoft.Testing.Platform YoloDev/YoloDev.Expecto.TestSdk#152

[thomhurst]

Can I ask where to look for the samples for a non-bridged runner when they're out? The samples folder in this repository?

[nohwnd]

@thomhurst I am adding an example that is doing the minimum amount of work to "run a test" here: #2446 There is test framework implementation, and a project using it. In reality the framework would come from a nuget package, and that nuget package would probably generate program.cs via build targets.

All the details of how to actually discover or run a test are up to your implementation. :) I've commented on the parts that did not seem clear, let me know if (ideally in a new issue), if there are more explanations needed. We will eventually document all that api, and add more examples.

[bradwilson]

@nohwnd If I'm reading the code correctly, it seems like the bridge that's being replaced here is just the translation of old ObjectModel types into the new types.

How about for developers who want to control the console output when being run via dotnet run (or even dotnet test)? Is that possible in the new system? I'm thinking about how this system gets merged with my current v3 runners which are already stand-alone executables with their own command lines and behavior that I wouldn't want to lose when moving to the new system. Also, will users be able to pass custom command line arguments to our runners when running via dotnet test?

[thomhurst]

Thanks @nohwnd I've got my runner working! (With dotnet run, and manually hooking it up. I can't get it to automatically hook up with the msbuild package)

My next thing id like to ask is how to get test filters, like how VSTest can filter on properties, but I'll raise an issue for that

[MarcoRossignoli]

@nohwnd If I'm reading the code correctly, it seems like the bridge that's being replaced here is just the translation of old ObjectModel types into the new types.

Yes. And it's needed only when you run in "runner mode"(self-contained) and you don't want to change the "current" VSTest adapter implementation. So it's done for a smooth/quick/co-existence onboarding but it's not mandatory. Pay attention that when you run in "runner mode" you cannot use anymore VSTest extensions like VSTest loggers, datacollectors, in-process datacollectors. The new platform implements a completely different extensibility points and expose a protocol for "remote" interaction (like IDEs VS/VSCode).

How about for developers who want to control the console output when being run via dotnet run (or even dotnet test)? Is that possible in the new system?

The "output device" is abstracted in the new platform. So it's possible plug a completely custom UX. By default we provide a "console" one. The extension point at the moment is "internal" because we're following the "hardening" principle. We will open when we're "good" with the design and all the expected usage. So we can "easily" support the "dotnet run" output customization.

For the "dotnet test" is a bit more tricky.

dotnet test is part of the SDK that contains the infrastructure to translate the "dotnet test" gesture in MSBuild tasks that will be invoked in a "demultiplexer" mode by "every project in a solution or directly to the project self if specified". To support it we shipped an extension point https://nuget.info/packages/Microsoft.Testing.Platform.MSBuild that plug the runner to the SDK infrastructure. The implementation we did changed completely the UX because was done mainly for CI with the "arcade" pattern where you simply see running module and result (we forward the output to a file in the result directory for further analysis). This is improving the understanding and avoid the unclear overlap of the parallel executions, but you can opt-in the console output and it's done using MSBuild loggers (that anyway is changed in 9.0 where we use the new terminal logger).
Here you can find the information about the "dotnet test" support for the new runner https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-runner-integrations#dotnet-test---mstest-runner-mode

Another tricky part are the parameters. Historically the SDK(dotnet test) supports "only" the VSTest platform with fixed options. The new platform is extensible in it and so we cannot anymore have a "fixed" set of options. For this reason as explained in the guide above you have to pass the new custom options(they depends on extensions added) using an MSBuild property -p:TestingPlatformCommandLineArguments=" --option value ".

Anyway this is the first version. We're planning to make it "first citizen" for the SDK and so move back to dotnet test --option value. But for that we need to add some code inside the SDK to fold the "different underneath test applications could support different options so how we show the help? How we show the invalid options per test application?" and so on.

I'm thinking about how this system gets merged with my current v3 runners which are already stand-alone executables with their own command lines and behavior that I wouldn't want to lose when moving to the new system.

As explained above the new platform allows custom options so it can accommodate your scenario. Here what we support https://github.com/microsoft/testfx/blob/main/test/UnitTests/Microsoft.Testing.Platform.UnitTests/CommandLine/CommandLineTests.cs and here the parser https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/CommandLine/Parser.cs

[MarcoRossignoli]

My next thing id like to ask is how to get test filters, like how VSTest can filter on properties, but I'll raise an issue for that

@thomhurst ideally the platform should not define "how to filter tests" because the adapters could express it in different way. The new platform introduces the concept of "test node with attached properties" so your tests could be a tree where every node is not related to a "method".
For this reason we've added a new "filter" options called treenode-filter https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/CommandLine/TreeNodeFilterCommandLineOptionsProvider.cs#L13 where you can express the query using a Graph Query Filtering syntax (i.e. --treenode-filter /*/*/*/*MyTest*. At the moment the filtering infrastructure is "internal" and used by our "custom adapter" for dogfooding. If you're interested in it pls open an issue and we can discuss the usage. Here you can find the draft https://github.com/microsoft/testfx/blob/main/docs/mstest-runner-graphqueryfiltering/graph-query-filtering.md

Anyway you can implement "your custom" filtering system for your tests and expose it with an custom option.