MSTest Runner: Provide a sample for usage in a project that uses WindowsAppSdk

[abdes]

It's not clear how the new MSTest runner fits in a context where the test project uses WindowsAppSdk or WinUI.

Such projects use a WinExe output type already and use custom setup code for the TestExecutor in the App startup code.

Please refer to: https://devblogs.microsoft.com/ifdef-windows/winui-desktop-unit-tests/ for more context on these types of test projects.

AB#2050328

[Evangelink]

Hi @abdes,

Thanks for the question! This was actually part of some of my experimentations yesterday :D

I have found and fixed a bug (see #2617) and I was able to list what's working and not working. I need to sync with WinUI team to understand the way forward.

I will post a simple example later today, maybe you could review it and give me some feedback.

[abdes]

Thanks @Evangelink for the super fast reply. I'm always available if you need to try something.

[Evangelink]

Please find enclosed a basic example of supporting MSTest runner with WinUI project MSTestRunnerWinUI.zip. As stated in the csproj, not all modes are working (it seems that there is some issue with unpackage and self-contained).

[abdes]

Please find enclosed a basic example of supporting MSTest runner with WinUI project MSTestRunnerWinUI.zip. As stated in the csproj, not all modes are working (it seems that there is some issue with unpackage and self-contained).

Here are my observations so far:

• We do get this warning during build: warning CS8892: Method 'TestingPlatformEntryPoint.Main(string[])' will not be used as an entry point because a synchronous entry point 'Program.Main(string[])' was found.
• Yes, there is an executable produced and the tests are running (checked via debugger breakpoint).
• It seems that the startup code you propose is mutually exclusive with Visual Studio TestExplorer. if MSTEST_RUNNER is being used, TestExplorer will see the tests but running them will make it spin for a long time, and none of my breakpoints gets activated anyway.
• The test runner does not produce any report on test executed/failed/skipped/etc... Not sure if this is intentional.
• You used net6, I used net8. Both seem to work the same way.
• Packaged and Unpackaged, both work, as long as you set <WindowsAppSDKSelfContained>True</WindowsAppSDKSelfContained>, otherwise, not able to locate the XAML assembly.
• Minor: using the test runner will produce an exception during startup because it calls some APIs that would normally be called only when running inside Visual Studio:

                    // Use the Setup API to find the installation folder for currently running VS instance.
                    if (new SetupConfiguration() is ISetupConfiguration setupConfiguration)
                    {
                        var currentConfiguration = setupConfiguration.GetInstanceForCurrentProcess();
                        var currentInstallationPath = currentConfiguration.GetInstallationPath();
                        s_vsInstallPath = Path.Combine(currentInstallationPath, @"Common7\IDE");
                    }

[mit456]

Hello @Evangelink and @abdes ,

I am experiencing the same issue explained here and therefore thought of trying out MSTest Runner property in WinUI3 application, but I not able to get preview version 3.4.0-preview.24171.3 from NuGet, can you please check the below .csproj content and let me know if I am making mistake?
Looking forward to some help.

<Project Sdk="Microsoft.NET.Sdk">
	<PropertyGroup>

		<OutputType>WinExe</OutputType>
		<TargetPlatformIdentifier>windows</TargetPlatformIdentifier>
		<TargetPlatformVersion>10.0.22621.0</TargetPlatformVersion>
		<TargetFramework>net8.0-windows</TargetFramework>
		<TargetPlatformMinVersion>10.0.22621.0</TargetPlatformMinVersion>
		<RootNamespace>sampleapp</RootNamespace>
		<ApplicationManifest>app.manifest</ApplicationManifest>
		<Platforms>x86;x64;ARM64</Platforms>
		<RuntimeIdentifier>win-x64</RuntimeIdentifier>
		<PublishProfile>win10-$(Platform).pubxml</PublishProfile>
		<EnableMsixTooling>true</EnableMsixTooling>
		<UseWinUI>true</UseWinUI>
		<TestResultDirectory>$(MSBuildProjectDirectory)\TestResults</TestResultDirectory>
		<IsTestProject>true</IsTestProject>
	</PropertyGroup>
	<!-- It seems that playing with these settings results in test app not able to run but from my observations
		this is not linked to MSTest runner (platform) but more linked to VS not able to start the application.
		Given my limited knowledge of WinUI, it's highly possible that I am only missing some settings or 
		configurations to have these options working.
	         -->
	<WindowsAppSDKSelfContained>True</WindowsAppSDKSelfContained>
	<!--<WindowsPackageType>None</WindowsPackageType>-->

	<EnableMSTestRunner>true</EnableMSTestRunner>

	<!-- Defining the "Msix" ProjectCapability here allows the Single-project MSIX Packaging
                      Tools extension to be activated for this project even if the Windows App SDK Nuget
                       package has not yet been restored.
                    -->
	<ItemGroup Condition="'$(DisableMsixProjectCapabilityAddedByProject)'!='true' and '$(EnableMsixTooling)'=='true'">
		<ProjectCapability Include="Msix" />
	</ItemGroup>

	<ItemGroup>
		<PackageReference Include="ClosedXML" Version="0.102.2" />
		<PackageReference Include="coverlet.collector" Version="6.0.2">
			<PrivateAssets>all</PrivateAssets>
			<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
		</PackageReference>
		<PackageReference Include="coverlet.msbuild" Version="6.0.2">
			<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
			<PrivateAssets>all</PrivateAssets>
		</PackageReference>
		<PackageReference Include="Microsoft.Windows.SDK.BuildTools" Version="10.0.22621.3233" />
		<Manifest Include="$(ApplicationManifest)" />
		<PackageReference Include="Microsoft.WindowsAppSDK" Version="1.5.240311000" />
		<PackageReference Include="MSTest.TestAdapter">
			<Version>3.4.0-preview.24171.3</Version>
		</PackageReference>
		<PackageReference Include="MSTest.TestFramework">
			<Version>3.4.0-preview.24171.3</Version>
		</PackageReference>
		<PackageReference Include="Microsoft.TestPlatform.TestHost">
			<Version>17.9.0</Version>
			<ExcludeAssets>build</ExcludeAssets>
		</PackageReference>
		<PackageReference Include="Microsoft.Data.Sqlite.Core" Version="8.0.3" />
		<PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.3" />
		<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="8.0.3" />
		<PackageReference Include="NSubstitute" Version="5.1.0" />
		<PackageReference Include="NSubstitute.Analyzers.CSharp" Version="1.0.17">
			<PrivateAssets>all</PrivateAssets>
			<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
		</PackageReference>
	</ItemGroup>

	<!--
                  Defining the "Msix" ProjectCapability here allows the Single-project MSIX Packaging
                  Tools extension to be activated for this project even if the Windows App SDK Nuget
                  package has not yet been restored.
                -->
	<ItemGroup Condition="'$(DisableMsixProjectCapabilityAddedByProject)'!='true' and '$(EnableMsixTooling)'=='true'">
		<ProjectCapability Include="Msix" />
	</ItemGroup>

	<!--  Defining the "HasPackageAndPublishMenuAddedByProject" property here allows the Solution 
                          Explorer "Package and Publish" context menu entry to be enabled for this project even if 
                          the Windows App SDK Nuget package has not yet been restored.
	          -->
	<PropertyGroup Condition="'$(DisableHasPackageAndPublishMenuAddedByProject)'!='true' and '$(EnableMsixTooling)'=='true'">
		<HasPackageAndPublishMenu>true</HasPackageAndPublishMenu>
	</PropertyGroup>
</Project>

[abdes]

@mit456 here is a Nuget.config file that will let you get the preview versions of MSTest:

<?xml version="1.0" encoding="utf-8"?>

<configuration>
    <!--
        Used to specify the default Sources for list, install and update.
        See: nuget.exe help list
        See: nuget.exe help install
        See: nuget.exe help update
    -->
    <packageSources>
        <clear />
        <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
        <add key="test-tools" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/test-tools/nuget/v3/index.json" />
    </packageSources>
    <!-- Define mappings by adding package patterns beneath the target source. -->
    <packageSourceMapping>
        <!-- key value for <packageSource> should match key values from <packageSources> element -->
        <packageSource key="test-tools">
            <package pattern="MSTest.*" />
            <package pattern="Microsoft.Testing.*" />
        </packageSource>
        <!-- key value for <packageSource> should match key values from <packageSources> element -->
        <packageSource key="nuget.org">
            <package pattern="*" />
        </packageSource>
    </packageSourceMapping>

    <!-- Used to disable package sources  -->
    <disabledPackageSources />

    <config>
        <!--
            Used to specify default source for the push command.
            See: nuget.exe help push
        -->
        <add key="defaultPushSource" value="https://api.nuget.org/v3/index.json" />
    </config>

    <packageRestore>
        <!-- Allow NuGet to download missing packages -->
        <add key="enabled" value="True" />

        <!-- Automatically check for missing packages during build in Visual Studio -->
        <add key="automatic" value="True" />
    </packageRestore>

    <packageManagement>
        <!-- Format is PackageReference -->
        <add key="format" value="1" />

        <!-- Do not show a prompt to chose the package format -->
        <add key="disabled" value="False" />
    </packageManagement>
</configuration>

Your .csproj seem OK. If you still run into problems, please look carefully at the sample provided by @Evangelink . Note that as of now, the setup works fine within Visual Studio Test Explorer, but I did not find any way (yet), to make it work from the command line, either via the new executable runner or via dotnet test command, if the test cases use WinUI.

  Error Message:
   Exception thrown while executing test. If using extension of TestMethodAttribute then please contact vendor. Error message: UITestMethodAttribute.DispatcherQueue should not be null. To use UITestMethodAttribute within a WinUI Desktop App, remember to set the static UITestMethodAttribute.DispatcherQueue during the test initialization., Stack trace:    at Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute.Execute(ITestMethod testMethod) in /_/src/TestFramework/TestFramework.Extensions/Attributes/WinUI_UITestMethodAttribute.cs:line 70
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.ExecuteTest(TestMethodInfo testMethodInfo) in /_/src/Adapter/MSTest.TestAdapter/Execution/TestMethodRunner.cs:line 388

Even if we properly set the DispatcherQueue in the App.OnLaunched or in a Test Initializer.

[mit456]

@abdes Thank you very much for fast reply. Well, if that's the case then I am also able to execute it using Visual Studio Test Explorer but when executing it from command line dotnet test, it is giving exact some error as above.

Please let me know if I can be of any help. Otherwise, I will keep an eye on the issue.

[Evangelink]

For the time being, Test Explorer (both for VS and VS Code) will always run tests using VSTest platform and so won't use MSTest runner. We have provided an improvement to this but this is waiting to be shipped into VS.

but when executing it from command line dotnet test, it is giving exact some error as above.

By default dotnet test will also fallback to VSTest platform. To enable the new runner mode with dotnet test, you will need to follow these instructions: https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#dotnet-test---microsofttestingplatform-mode

The support for WinUI with MSTest runner is not yet official as this is not something we have tested or worked on. Following @abdes suggestion, I have started to do some POC to see what's working or not. This is still considered as experimental but I do hope to find some time to sync with WinUI team and work on the failing parts so we can provide a proper experience.

[mit456]

Hello @Evangelink

Thank you for clarifying things for me. I did try to enable Microsoft.Testing.Platform but getting on dotnet test

C:\Program Files\dotnet\sdk\8.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ConflictResolution.targets(112,5): error NETSDK1152: Found multiple publish output files with the same relative path: C:\Users\mithu_f8sh03l\.nuget\packages\mstest.testadapter\3.2.2\bu
ild\net6.0\winui\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll, C:\Users\mithu_f8sh03l\.nuget\packages\mstest.testadapter\3.2.2\build\net8.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll.

But if I remove <EnableMSTestRunner>true</EnableMSTestRunner> which I think I should not do it, else test framework will not work, and that's what happening, if EnableMSTestRunner is removed, getting below error on dotnet test

C:\Users\mithu_f8sh03l\.nuget\packages\microsoft.testing.platform.msbuild\1.0.2\buildMultiTargeting\Microsoft.Testing.Platform.MSBuild.targets(228,5): error: Tests failed: Unhandled exception. System.DllNotFoundException: Unable to load DLL 'Microsoft.ui.xaml.dll'
 or one of its dependencies: The specified module could not be found. (0x8007007E)

My random guess is that I need to keep the EnableMSTestRunner but remove MSTest.TestAdapter from package reference. Please guide me, this question could be out of context of this thread but asking out of my curiosity.

[Evangelink]

There is indeed some bug with MSTest 3.2 for the support of WinUI. This is fixed in this PR #2617 that targets main (so v3.4) that I haven't backported to 3.3 that I will release this week because the support is not yet complete.

If you follow the steps described above for the nuget.config and use 3.4.0-preview.24171.3 or newer, you should not have this error.

[testplatform-bot]

✅ Successfully linked to Azure Boards work item(s):

• User Story 2050328