Feature: Added --filter option to VsTestConsoleWrapper

[corentinaltepe]

This PR is a work in progress and is clearly not ready for review. However, I need help. I am stuck with the problem described below.

Contributes to #420.
This PR depends on #1542 and #1273 which will need both to be merged first. Then this PR will be updated before ready for final review.

Progress

As suggested by @rouke-broersma:

• user passes filter to stryker as an option according to new standard: feat: Rework how stryker consumes options #1273 (wait for the PR to be merged).
• filter gets used to filter the discovered tests in the initial testrun
• filter gets used to filter the executed tests during coverage analysis (all modes), this effectively filters the used tests
• if coverage analysis is turned off, the filter still needs to be applied because coverage analysis did not perform the filtering for us

Problem

tl;dr: vstest.console.exe applied /TestCaseFilter: option works just fine, but the VsTestConsoleWrapper does not.

@dupdob: since you so kindly offered support in #420, may I ask your help here. I'm kinda stuck on this issue and running out of ideas.

I re-used a xUnit test project Utils.Test.csproj containing two dummy tests:

• Utils.Test.MathTest.Ceiling with category unit
• Utils.Test.MathTest.Floor with category integration

I managed to properly list and filter them with dotnet test and vstest.console.exe using either --filter Category=unit or /TestCaseFiler:Category=unit options.

"C:\Users\Corentin\.nuget\packages\microsoft.testplatform.portable\16.9.4\tools\net451\vstest.console.exe" -lt /TestCaseFilter:Category=unit C:\Users\Corentin\source\repos\ConteneurWeb\Tests\Common\Utils.Test\bin\Debug\net5.0\Utils.Test.dll
Microsoft (R) Test Execution Command Line Tool Version 16.9.4
Copyright (c) Microsoft Corporation.  All rights reserved.

The following Tests are available:
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET 5.0.5)
[xUnit.net 00:00:00.33]   Discovering: Utils.Test
[xUnit.net 00:00:00.38]   Discovered:  Utils.Test
    Utils.Test.MathTest.Ceiling

IVsTestConsoleWrapper.DiscoverTests() allows specifying TestPlatformOptions as a parameter, which contains a TestCaseFilter property. So it looked easy. While waiting for #1273, I hardcoded a filter "Category=unit" in VsTestRunner:

public IDictionary<Guid, VsTestDescription> DiscoverTests(string runSettings = null)
        {
            ...
            using (var waitHandle = new AutoResetEvent(false))
            {
                var handler = new DiscoveryEventHandler(waitHandle, _messages);
                var generateRunSettings = GenerateRunSettings(null, false, false, null);

                var vsTestOptions = new TestPlatformOptions { TestCaseFilter = "Category=unit" };
                _vsTestConsole.DiscoverTests(_sources, runSettings ?? generateRunSettings, vsTestOptions, handler);
                ...
            }
            return _vsTests;
        }

However, the console wrapper always returns the two tests, whereas I expected it to return only one (the one matching the category filter). I ran the very same binaries called by the wrapper in a command line, and they worked just fine when specifying /TestCaseFilter:.

Please note VsTest.Console.exe's documentation note:

The MSTest adapter in Visual Studio also works in legacy mode (equivalent to running tests with mstest.exe) for compatibility. In legacy mode, it can't take advantage of the TestCaseFilter feature. The adapter can switch to legacy mode when a testsettings file is specified, forcelegacymode is set to true in a runsettings file, or by using attributes such as HostType.

To run automated tests on an ARM architecture-based machine, you must use VSTest.Console.exe.

Could my issue be caused by legacy mode? I set forcelegacymode to false in runsettings to no effect.

Please find attached the logs generated by the vstest console wrapper. Notice the TestPlatformOptions are properly serialized.
VsTest-log.txt

Potentially related issues and PRs, though I am not sure I understand their context completely:

• Filtering default traits/categories in the .runsettings file microsoft/vstest#2273
• Filter during discovery xunit/visualstudio.xunit#205
• vstest.console's --TestCaseFilter is ignored when listing tests nunit/nunit3-vs-adapter#438

[rouke-broersma]

You may need to pass the filter in the runsettings we generate on line 414 in vstest runner. I think runsettings override other options.

I don't think legacy mode is relevant as this pertains to mstest.exe (mstest v1) and not vstest.

Also please target branch v1.0 so the diff only contains your changes, makes it easier to help you.

[corentinaltepe]

Thank you @rouke-broersma. This makes sense. I'll give it a try and keep you posted. I've also re-targeted the PR to v1.0 branch, but the diff is still difficult to read because the PR is based off #1542.

[dupdob]

@corentinaltepe :

1. Yes, use V1 as a reference it will ease reviewing your changes
2. Just in case, this link may be useful to you: https://github.com/microsoft/vstest-docs
3. Judging by the issues you referred to (thanks by the way), test discovery did not support filtering a year ago. I understand a relevant PR has been merged in xUnit, but I am afraid you need to verify your VsTest/MSTest runner version supports this.
4. Have you looked at the log for the vstest test discovery session, can you post it here?

Some general remarks:

1. Welcome to the other side of the mirror regarding VsTest integration. This will be a fun ride, I hope.
2. One of the main problem with this integration is that VsTest does not report TimeOut situations explicitly (or we failed to see it).
3. As a consequence, Stryker needs a dedicated logic for that, verifying that it has received all expected test results (if some are missing this is likely to be a timeout
4. As a secondary consequence, Stryker needs a precise view of existing tests cases and results.

[corentinaltepe]

Logs

Please find attached the logs for both vstest.console.exe and testhost.dll with verbose/trace level. Notice that while vstest.console.exe logs the TestCaseFilter appropriately, it appears as null in testhost.dll's logs.
VsTest-log.host.21-05-02_11-50-25_96094_9.txt
VsTest-log.txt

xUnit version

xUnit and Microsoft.NET.Test.Sdk are all up to date. The relevant PR in xUnit is included in xunit.runner.visualstudio v2.4.2 and v2.4.3 (the one I currently use).

Please see here the list of dependencies for my test project Utils.Test.csproj

Project 'Utils.Test' has the following package references
   [net5.0]:
   Top-level Package                Requested   Resolved
   > coverlet.collector             1.3.0       1.3.0
   > Microsoft.NET.Test.Sdk         16.*        16.9.4
   > xunit                          2.4.1       2.4.1
   > xunit.runner.visualstudio      2.4.3       2.4.3

   Transitive Package                                                                   Resolved
   > Microsoft.CodeCoverage                                                             16.9.4
   > Microsoft.Extensions.Caching.Abstractions                                          5.0.0
   > Microsoft.Extensions.Primitives                                                    5.0.0
   > Microsoft.NETCore.Platforms                                                        1.1.0
   > Microsoft.NETCore.Targets                                                          1.1.0
   > Microsoft.TestPlatform.ObjectModel                                                 16.9.4
   > Microsoft.TestPlatform.TestHost                                                    16.9.4
   > Microsoft.Win32.Primitives                                                         4.3.0
   > NETStandard.Library                                                                1.6.1
   > Newtonsoft.Json                                                                    12.0.3
   > NuGet.Frameworks                                                                   5.0.0
   > runtime.debian.8-x64.runtime.native.System.Security.Cryptography.OpenSsl           4.3.0
   > runtime.fedora.23-x64.runtime.native.System.Security.Cryptography.OpenSsl          4.3.0
   > runtime.fedora.24-x64.runtime.native.System.Security.Cryptography.OpenSsl          4.3.0
   > runtime.native.System                                                              4.3.0
   > runtime.native.System.IO.Compression                                               4.3.0
   > runtime.native.System.Net.Http                                                     4.3.0
   > runtime.native.System.Security.Cryptography.Apple                                  4.3.0
   > runtime.native.System.Security.Cryptography.OpenSsl                                4.3.0
   > runtime.opensuse.13.2-x64.runtime.native.System.Security.Cryptography.OpenSsl      4.3.0
   > runtime.opensuse.42.1-x64.runtime.native.System.Security.Cryptography.OpenSsl      4.3.0
   > runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.Apple            4.3.0
   > runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.OpenSsl          4.3.0
   > runtime.rhel.7-x64.runtime.native.System.Security.Cryptography.OpenSsl             4.3.0
   > runtime.ubuntu.14.04-x64.runtime.native.System.Security.Cryptography.OpenSsl       4.3.0
   > runtime.ubuntu.16.04-x64.runtime.native.System.Security.Cryptography.OpenSsl       4.3.0
   > runtime.ubuntu.16.10-x64.runtime.native.System.Security.Cryptography.OpenSsl       4.3.0
   > System.AppContext                                                                  4.3.0
   > System.Buffers                                                                     4.3.0
   > System.Collections                                                                 4.3.0
   > System.Collections.Concurrent                                                      4.3.0
   > System.Console                                                                     4.3.0
   > System.Diagnostics.Debug                                                           4.3.0
   > System.Diagnostics.DiagnosticSource                                                4.3.0
   > System.Diagnostics.Tools                                                           4.3.0
   > System.Diagnostics.Tracing                                                         4.3.0
   > System.Globalization                                                               4.3.0
   > System.Globalization.Calendars                                                     4.3.0
   > System.Globalization.Extensions                                                    4.3.0
   > System.IO                                                                          4.3.0
   > System.IO.Compression                                                              4.3.0
   > System.IO.Compression.ZipFile                                                      4.3.0
   > System.IO.FileSystem                                                               4.3.0
   > System.IO.FileSystem.Primitives                                                    4.3.0
   > System.Linq                                                                        4.3.0
   > System.Linq.Expressions                                                            4.3.0
   > System.Net.Http                                                                    4.3.0
   > System.Net.Primitives                                                              4.3.0
   > System.Net.Sockets                                                                 4.3.0
   > System.ObjectModel                                                                 4.3.0
   > System.Reflection                                                                  4.3.0
   > System.Reflection.Emit                                                             4.3.0
   > System.Reflection.Emit.ILGeneration                                                4.3.0
   > System.Reflection.Emit.Lightweight                                                 4.3.0
   > System.Reflection.Extensions                                                       4.3.0
   > System.Reflection.Metadata                                                         1.6.0
   > System.Reflection.Primitives                                                       4.3.0
   > System.Reflection.TypeExtensions                                                   4.3.0
   > System.Resources.ResourceManager                                                   4.3.0
   > System.Runtime                                                                     4.3.0
   > System.Runtime.Extensions                                                          4.3.0
   > System.Runtime.Handles                                                             4.3.0
   > System.Runtime.InteropServices                                                     4.3.0
   > System.Runtime.InteropServices.RuntimeInformation                                  4.3.0
   > System.Runtime.Numerics                                                            4.3.0
   > System.Security.Cryptography.Algorithms                                            4.3.0
   > System.Security.Cryptography.Cng                                                   4.3.0
   > System.Security.Cryptography.Csp                                                   4.3.0
   > System.Security.Cryptography.Encoding                                              4.3.0
   > System.Security.Cryptography.OpenSsl                                               4.3.0
   > System.Security.Cryptography.Primitives                                            4.3.0
   > System.Security.Cryptography.X509Certificates                                      4.3.0
   > System.Text.Encoding                                                               4.3.0
   > System.Text.Encoding.Extensions                                                    4.3.0
   > System.Text.RegularExpressions                                                     4.3.0
   > System.Threading                                                                   4.3.0
   > System.Threading.Tasks                                                             4.3.0
   > System.Threading.Tasks.Extensions                                                  4.3.0
   > System.Threading.Timer                                                             4.3.0
   > System.Xml.ReaderWriter                                                            4.3.0
   > System.Xml.XDocument                                                               4.3.0
   > Utils                                                                              1.0.0
   > xunit.abstractions                                                                 2.0.3
   > xunit.analyzers                                                                    0.10.0
   > xunit.assert                                                                       2.4.1
   > xunit.core                                                                         2.4.1
   > xunit.extensibility.core                                                           2.4.1
   > xunit.extensibility.execution                                                      2.4.1

Is there something else I should do to verify my VsTest/MSTest runner version supports filtering?

Follow up on other questions

I will follow up with your other questions / suggestions, but I'll need more time for:

• specifying the filtering options in runsettings instead of the VsTestConsoleWrapper's DiscoverTests argument
• focusing on filtering actual test sessions first, and consider test discovery later, as we know filtering is supposed to work here
• removing legacymode option, as it's probably not the issue

[dupdob]

Just for clarification: my remarks regarding filtering support was focused on the Discovery phase. Filtering is supported during test run phase. Hence my recommandation to focus on test run phase instead.
Looking at the log you provided, I see that no filters was transmitted to VsTest: "TestCaseFilter": null.
So either this is the wrong log (maybe the initial test run?) or the parameters was not transmitted for some reason.

[corentinaltepe]

Thank you for your help. I have switched to #1557 since it is branched off v1.0 for an easier code review.