Add initial support for Microsoft.Testing.Platform #403

Evangelink · 2024-02-22T20:18:47Z

Fixes #402

Evangelink · 2024-02-22T20:19:45Z

samples/NuGet.Config

+    <clear />
+    <add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
+    <add key="feedz.io/xunit/xunit" value="https://f.feedz.io/xunit/xunit/nuget/index.json" protocolVersion="3" />
+    <add key="local" value="C:\src\visualstudio.xunit\artifacts\packages\" />


This shouldn't be kept, this is just a temp change for you to test locally (you'll need to update the path as you know).

Evangelink · 2024-02-22T20:20:13Z

src/xunit.runner.visualstudio/TestingPlatform/TestApplicationBuilderExtensions.cs

+	public static void AddXunit(this ITestApplicationBuilder testApplicationBuilder, Func<IEnumerable<Assembly>> getTestAssemblies)
+	{
+		XunitExtension extension = new();
+		testApplicationBuilder.AddRunSettingsService(extension);


This provides the runsettings support.

Evangelink · 2024-02-22T20:20:25Z

src/xunit.runner.visualstudio/TestingPlatform/TestApplicationBuilderExtensions.cs

+	{
+		XunitExtension extension = new();
+		testApplicationBuilder.AddRunSettingsService(extension);
+		testApplicationBuilder.AddTestCaseFilterService(extension);


This provides support for the vstest way of filtering tests.

Evangelink · 2024-02-22T20:21:18Z

src/xunit.runner.visualstudio/build/xunit.runner.visualstudio.desktop.props

+      DO NOT CHANGE THE GUID, IT'S A WELL KNOWN EXTENSION POINT AND THIS EXTENSION NEEDS TO BE REGISTERED AT THE END
+      WE HAVE CODE INSIDE THE TASK 'TestingPlatformEntryPoint' TO ENSURE THE ORDER OF THE REGISTRATION BASED ON THIS GUID
+    -->
+    <TestingPlatformBuilderHook Include="17E773D9-071C-4C66-97DD-57A450BDB027" Condition=" $(GenerateTestingPlatformEntryPoint) == 'true' " >


Allows Microsoft.Testing.Platform.MSBuild to detect the hook to call so we can auto-generate the entry point.

Evangelink · 2024-02-22T20:21:43Z

src/xunit.runner.visualstudio/build/xunit.runner.visualstudio.desktop.props

@@ -1,11 +1,24 @@
 <?xml version="1.0" encoding="utf-8"?>
 <Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <PropertyGroup>
+    <EnableXunitRunner Condition=" '$(EnableXunitRunner)' == '' ">false</EnableXunitRunner>


I'll let you decide on the name, this is just an example.

Evangelink · 2024-02-22T20:22:38Z

src/xunit.runner.visualstudio/xunit.runner.visualstudio.nuspec

@@ -18,8 +18,15 @@
    <dependencies>
      <group targetFramework="net462">
        <dependency id="Microsoft.TestPlatform.ObjectModel" version="$MicrosoftTestPlatformObjectModelVersion$" />
+        <dependency id="Microsoft.Testing.Extensions.Telemetry" version="$MicrosoftTestingPlatformVersion$" />
+		<dependency id="Microsoft.Testing.Extensions.VSTestBridge" version="$MicrosoftTestingPlatformVersion$" />


Not sure why the indent is all messed up, I don't see that locally:

this is mixing tabs/spaces, you'll need to configure your editor to show whistespaces

bradwilson · 2024-02-26T03:42:40Z

@Evangelink When building the sample, I get a compile failure that looks like some kind of missing assembly/package reference:

1>------ Build started: Project: xunit-runner-sample, Configuration: Debug Any CPU ------
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(12,27,12,34): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(12,92,12,117): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(13,9,13,26): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(14,9,14,26): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(15,9,15,75): error CS0012: The type 'ITestApplicationBuilder' is defined in an assembly that is not referenced. You must add a reference to assembly 'Microsoft.Testing.Platform, Version=1.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(16,34,16,41): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>Done building project "xunit-runner-sample.csproj" -- FAILED.
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(12,27,12,34): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(12,92,12,117): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(13,9,13,26): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(14,9,14,26): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(15,9,15,75): error CS0012: The type 'ITestApplicationBuilder' is defined in an assembly that is not referenced. You must add a reference to assembly 'Microsoft.Testing.Platform, Version=1.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(16,34,16,41): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>Done building project "xunit-runner-sample.csproj" -- FAILED.

If I set <EnableXunitRunner>false</EnableXunitRunner>, the compile errors go away. It looks like the hooks for MSBuild are all triggering, but something is missing with references.

bradwilson · 2024-02-26T03:46:04Z

Looks like it might be package reference related. Converting this:

<PackageReference Include="xunit.runner.visualstudio" Version="2.5.8-pre.2">
  <PrivateAssets>all</PrivateAssets>
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>

to this:

<PackageReference Include="xunit.runner.visualstudio" Version="2.5.8-pre.2" />

fixes the issue. So it looks like setting developmentDependency is probably wrong for .nuspec now. Next question to answer is: Will NuGet pick up changes to this during package upgrade and update the PackageReference appropriately?

bradwilson · 2024-02-26T03:55:54Z

All three of these variants did not upgrade the package correctly:

Package Manager in VS2022
dotnet add package xunit.runner.visualstudio --prerelease
dotnet outdated -u

That's pretty disappointing. That means we're on a failure path here unless we can somehow convince people to uninstall and then reinstall the package. 😞

bradwilson · 2024-02-26T05:23:04Z

Also, it looks like you're setting a value in the .targets file (GenerateTestingPlatformEntryPoint) that you depend on in the .props files, which seems like the wrong order. Shouldn't that top PropertyGroup from the .targets file actually be in the .props files?

bradwilson · 2024-02-29T03:36:05Z

What are the plans for dotnet vstest? It appears that it currently falls back to the old behavior.

bradwilson · 2024-02-29T03:36:44Z

Why are .NET Framework tests run in x86 by default if the test project is compiled as AnyCPU?

Run tests: 'C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\bin\Debug\net462\xunit-runner-sample.exe' [net462|x86]
Run tests: 'C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\bin\Debug\net6.0\xunit-runner-sample.dll' [net6.0|x64]

Evangelink · 2024-03-02T10:10:20Z

Hey @bradwilson,

Sorry for the delay in my replies, I am off and had some trouble finding some time to get back here. I'll resume work on 11th so will be fully available for sure at this stage.

So it looks like setting developmentDependency is probably wrong for .nuspec now

I assume that this is telling to msbuild that all deps are only compile time deps and not runtime ones but for the runner we do need the platform + adapter to be available at runtime. I see you have created a ticket on NuGet, I will try to ping some people so we can see how we can unblock this. @baronfel would you have some info or contact about that?

What are the plans for dotnet vstest? It appears that it currently falls back to the old behavior.

As the name implies this is explicitly asking for the vstest mode. This is equivalent to dotnet test without the property TestingPlatformDotnetTestSupport enabled (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-integrations#dotnet-test---mstest-runner-mode). While you reference the VSTest bridge package this will continue to work but as soon as this deps is dropped this will no longer work.

Our goal is to be able to deprecate VSTest entirely but this will most likely take some time as we will need to have enough people moved and happy on the new platform.

Why are .NET Framework tests run in x86 by default if the test project is compiled as AnyCPU?

We are not doing any change on our side, this seems to be MSBuild defaults that changed between netfx and netcore.
@MarcoRossignoli do you know? @baronfel do you know about that? Or do you have someone that might know?

Evangelink · 2024-03-02T10:20:01Z

src/xunit.runner.visualstudio/build/xunit.runner.visualstudio.desktop.props

+  </PropertyGroup>
+
+  <ItemGroup>
+    <!-- This GUID comes from Microsoft.Testing.Platform, do not modify -->


The GUID is actually just any random GUID but it needs to be unique across the various extensions registered through the hook.

Evangelink · 2024-03-02T10:35:04Z

Why are .NET Framework tests run in x86 by default if the test project is compiled as AnyCPU?

@MarcoRossignoli just pointed out this ticket dotnet/sdk#3492 that confirms this is the "default" behavior and this is a change linked to the sdk style project.

bradwilson · 2024-03-02T20:43:20Z

Sorry for the delay in my replies

No worries, this isn't a rush situation. :)

bradwilson · 2024-03-02T20:50:30Z

As the name implies this is explicitly asking for the vstest mode.

I ask because I use dotnet vstest with the --Parallel switch because it has superior behavior in that mode than just running dotnet test where all the output is interleaved and not combined (and it allows me to be more selective on what gets run). I understand that the new dotnet test output is basically so sparse that the interleaved output problem is gone, though I suspect you will get end user push back about this being too spares for interactive consumption. You should certainly consider my voice at the top of that list. 😄 Losing the ability to be more selective would be unfortunate, though in our case we do still plan to ship our multi-runners in v3 (like the console runner) that would allow us to continue to be more selective.

baronfel · 2024-03-02T20:52:49Z

@bradwilson .NET 9 preview 2 includes a rework of the test-reporting infrastructure that does show testing progress, as well as ordered test result reporting directly in MSBuild. It's built on a tiny bit of a hack while the MSBuild team works on a proper progress/ephemeral message API for the MSBuild logging infrastructure, after which we should be able to use those new message types to implement nice CLI progress reporting and such. So it's coming, we're working on it :)

Evangelink · 2024-03-03T07:31:54Z

We definitely want to work on and improve the interactive mode. As Chet is saying we have already started some work and we are also exploring some paths in parallel so we can decide what's best and most evolutive

bradwilson · 2024-03-09T22:07:54Z

@Evangelink @MarcoRossignoli @baronfel Where is the source for Microsoft.Testing.Extensions.VSTestBridge? I see dependencies on it in https://github.com/microsoft/testfx but I don't see the source.

bradwilson · 2024-03-10T00:04:45Z

Is the planned IPC for support inside Visual Studio/VS Code also going to be named pipes? If so, what is the equivalent of --internal-msbuild-node <pipename> to support that? If not, what is the planned IPC mechanism and which client class should I be using to support it?

Why are all the types in Microsoft.Testing.Platform.IPC marked as internal?

bradwilson · 2024-03-10T21:25:25Z

I'm just going to start opening issues with these questions so they're not buried here, and others can contribute (and perhaps get the benefit of the answers).

First set of issues:

microsoft/testfx#2536 ("FailedTestNodeStateProperty should include the ability to report without an exception")
microsoft/testfx#2537 ("What is the intended purpose for the hierarchy provided in TestNode?")
microsoft/testfx#2538 ("How do I report a test that's not run?")
microsoft/testfx#2539 ("Please document the IPC handshake mechanisms and open up the classes under Microsoft.Testing.Platform.IPC")

bradwilson · 2024-03-10T22:02:29Z

FYI, I have a parallel prototype effort to integrate Microsoft.Testing.Platform directly into v3's existing architecture without using the VSTest adapter: https://github.com/xunit/xunit/tree/microsoft-testing-platform

Add initial support for Microsoft.Testing.Platform

a7721a9

Evangelink marked this pull request as draft February 22, 2024 20:18

Evangelink commented Feb 22, 2024

View reviewed changes

Evangelink mentioned this pull request Feb 22, 2024

Allow xunit, nunit and other frameworks to run with the new runner microsoft/testfx#2164

Closed

bradwilson mentioned this pull request Feb 26, 2024

developmentDependency change is ignored when upgrading package NuGet/Home#13263

Open

Code style

7c7d7ab

Add OutputType of Exe when the user has requested EnableXunitRunner

80083a4

MartyIX mentioned this pull request Feb 26, 2024

Microsoft Testing Platform for xUnit #402

Open

Evangelink commented Mar 2, 2024

View reviewed changes

Evangelink added 2 commits March 2, 2024 11:47

Bump version for the sample project

001053f

Simplify condition for DisableTestingPlatformServerCapability

f856d53

MarcoRossignoli mentioned this pull request Mar 14, 2024

How do you run x64 .NET Framework tests with Microsoft.Testing.Platform.MSBuild? microsoft/testfx#2562

Closed

bradwilson mentioned this pull request Apr 5, 2024

Add runner support for v3 cross-process RPC xunit/xunit#2337

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial support for Microsoft.Testing.Platform #403

Add initial support for Microsoft.Testing.Platform #403

Evangelink commented Feb 22, 2024

Evangelink Feb 22, 2024

Evangelink Feb 22, 2024

Evangelink Feb 22, 2024

Evangelink Feb 22, 2024

Evangelink Feb 22, 2024

Evangelink Feb 22, 2024

akoeplinger Feb 23, 2024

bradwilson commented Feb 26, 2024

bradwilson commented Feb 26, 2024 •

edited

bradwilson commented Feb 26, 2024

bradwilson commented Feb 26, 2024

bradwilson commented Feb 29, 2024

bradwilson commented Feb 29, 2024 •

edited

Evangelink commented Mar 2, 2024 •

edited

Evangelink Mar 2, 2024

Evangelink commented Mar 2, 2024

bradwilson commented Mar 2, 2024

bradwilson commented Mar 2, 2024

baronfel commented Mar 2, 2024

Evangelink commented Mar 3, 2024

bradwilson commented Mar 9, 2024

bradwilson commented Mar 10, 2024 •

edited

bradwilson commented Mar 10, 2024

bradwilson commented Mar 10, 2024

Add initial support for Microsoft.Testing.Platform #403

Are you sure you want to change the base?

Add initial support for Microsoft.Testing.Platform #403

Conversation

Evangelink commented Feb 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bradwilson commented Feb 26, 2024

bradwilson commented Feb 26, 2024 • edited

bradwilson commented Feb 26, 2024

bradwilson commented Feb 26, 2024

bradwilson commented Feb 29, 2024

bradwilson commented Feb 29, 2024 • edited

Evangelink commented Mar 2, 2024 • edited

Choose a reason for hiding this comment

Evangelink commented Mar 2, 2024

bradwilson commented Mar 2, 2024

bradwilson commented Mar 2, 2024

baronfel commented Mar 2, 2024

Evangelink commented Mar 3, 2024

bradwilson commented Mar 9, 2024

bradwilson commented Mar 10, 2024 • edited

bradwilson commented Mar 10, 2024

bradwilson commented Mar 10, 2024

bradwilson commented Feb 26, 2024 •

edited

bradwilson commented Feb 29, 2024 •

edited

Evangelink commented Mar 2, 2024 •

edited

bradwilson commented Mar 10, 2024 •

edited