Microsoft Testing Platform for xUnit

[Evangelink]

Dear xUnit team,

As per our previous discussions, we would like to help you onboard xUnit to the new self-contained testing platform that powers MSTest Runner.

We are currently working on the technical documentation for framework authors and extensions authors so sadly for now there isn't much available for you to browse.

The new platform is already OSS and available inside of MSTest repository (at the moment).

Related issue about support NUnit on MSTest repo: microsoft/testfx#2164

MSTest runner blogpost announcement: https://devblogs.microsoft.com/dotnet/introducing-ms-test-runner/
MSTest runner doc: https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-intro?tabs=dotnetcli
Platform code: https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform
Sample code: https://github.com/microsoft/testfx/tree/main/samples/mstest-runner

[bradwilson]

How has user feedback been on the extremely abbreviated output from dotnet test? I would expect users to push back on the complete lack of any reporting information; notably, the lack of failed test information seems excessively terse by default.

• Interactive usage means dumping the log file as a separate manual step when tests fail
• If you're relying on the raw build logs to see test failure in CI, you'll either need to dump the log file on failure and/or save the log file and/or save a TRX output file.

[bradwilson]

Old output:

New output: (note that I had to cat the log file to find the failure)

[bradwilson]

Does enabling the new system preclude usage in older clients, like older VS Code/Visual Studio versions? Or does enabling the new system provide a "best of both worlds" by allowing the system using the older APIs to still function?

[MartyIX]

Out of curiosity: Does this issue and the corresponding PR (#403) represent a fix for xunit/xunit#2337?

Or is xunit/xunit#2337 about something else?

[bradwilson]

That issue is technically something else, though I did chat with the VSTest team about whether the protocol they're using here might be something I could re-use, so it would end up perhaps allowing me to reuse some of their code for my v3 cross-process protocol. It's definitely something I'm going to be investigating. 😄

[bradwilson]

save a TRX output file

Actually, this doesn't appear to be an option. The --logger switch to dotnet test appears to be entirely ignored in the new system, so I can't customize the console logger nor use the TRX logger. Is this an oversight? How are users expected to be able to get machine parseable logs from the new system?

[Evangelink]

How has user feedback been on the extremely abbreviated output from dotnet test? I would expect users to push back on the complete lack of any reporting information; notably, the lack of failed test information seems excessively terse by default.

Not liked by everyone microsoft/testfx#2162. We are working on some "interactive" mode where more things would be printed.

This default is more performant and is aligned with how we usually define things internally (for example this is matching the default output of arcade infra). We believe this is a good default for CI and we want to keep it but we understand that this is not what everyone wants and we are working on improving this.

We are also working on an improved terminal experience for terminals that supports better outputing.

Does enabling the new system preclude usage in older clients, like older VS Code/Visual Studio versions? Or does enabling the new system provide a "best of both worlds" by allowing the system using the older APIs to still function?

Thanks to the bridge, the old versions of Test Explorer (VS and VS Code) would communicate using VSTest and the new versions have some setting (default is using the runner directly through the new json protocol) to switch between modes.

Actually, this doesn't appear to be an option.

We will be updating the dotnet test documentation page but you should consider that all the options displayed there are VSTest related options. To be flowing down options to the runner you need to use TestingPlatformCommandLineArguments (as described here https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-integrations#dotnet-test---mstest-runner-mode)

so I can't customize the console logger

Correct, at the moment this is not supported but we have some infra (internal for now) to support this in the future. We still need some design and testing to ensure it's all good and can be open.

On the new platform we are making a distingo between output and logging (although we won't block if someone creates some specific logger that redirects to the output service) because they are fundamentaly different and this aggregation was causing lots of pain in VSTest platform.

nor use the TRX logger

This isn't true. You can install and use the trx extension (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-extensions#visual-studio-test-reports).

[bradwilson]

This isn't true. You can install and use the trx extension

How is this different from --logger trx (aka, why are there two ways to do it)?

[Evangelink]

Because all commands of dotnet test are VSTest oriented. Besides VSTest was using the logger concept for everything which was leading to many issues and confusions. On the new platform we are trying to have clear concepts: logger is like any regular app, console falls under an output device concept which will allow us to have different reporting (remote call, live web pages for CI, handling of devices...) and finally extensions that are subscribing to events (this is the case for TRX report).

[jkone27]

this is super useful aslo for debugging tests also cross languages in the .NET ecosystem, i hope this is available soon for Xunit

[bradwilson]

@jkone27 What is provided here that you don't already get with xUnit.net's VSTest integration? You can already debug tests in VS and VS Code. To be clear, what I see being offered here is only a potential performance advantage.