Filtering tests with any "PropertyAttribute"

[pkrukp]

Hi,

vstest.console.exe has /TestCaseFilter switch (and in IVsTestConsoleWrapper API there's TestCaseFilter in RunSettings, in dotnet test it's --filter AFAIK).

But it seems it's not possible to run only tests which has specific PropertyAttribute. For example given following:

        [Property("Bug", "12345")]
        [Test]
        public void Test2()
        {
            Assert.Pass();
        }

I would like to be able run something like this: vstest.console.exe NUnitTestProject1.dll /TestCaseFilter:Bug=12345. But it does not seems to work - no test is executed. Am I doing it wrong, or is there no support for this?

According to microsoft/testfx#731 it must be supported by test adapter (it is supported by xUnit, not supported by MSTest).

If it is not supported by NUnit, I'd like to ask to add such support.

Please see the above issue for some more details and simple solution with 3 UT projects (mstest, xunit, nunit): ClassLibrary1.zip
I have tested this with vstest.console.exe from microsoft.testplatform.16.7.0 nuget package, with NUnit3TestAdapter 3.15.1.

[shack05]

Hello, yes it can be done given you are using at least version 3.16 of the nunit test adapter, and a relatively new version of vstest.console.exe (I'm using v16.3.0)

Example

Using your example:
vstest.console.exe NUnitTestProject1.dll -- NUnit.Where="Bug=12345"

Note the single space after the two hyphens.

Explanation

As of v 3.16, the NUnit adapter supports test selection using NUnit's own test selection language by including a 'Where' property in the runsettings file.

This can be coupled with vstest's ability to provide runsettings arguments via the command line.

Further reading

https://blog.prokrams.com/2019/12/16/nunit3-filter-dotnet/

I hope that helps.

[pkrukp]

I see. That's good it can be done.

The advantage is, as I understand, that it can use full power of NUnit test selection language.

The disadvantage I see it that it's specific for NUnit. It won't work with xUnit or MSTest as I understand.
So user won't be able to use the common "/TestCaseFilter" functionality of vstest.

From this point of view, maybe it'd be still possible to pass those properties to vstest? As I understand, NUnit already passes some properties (e.g. "Category") as Traits to vstest. I hope that passing "Property" would not be a big change.

[shack05]

Hopefully somebody else can correct me if I am wrong, but the NUnit adapter already passes test properties to VSTest as traits which is why the properties can be seen in the test explorer window in Visual Studio (I think the relevant code is the AddTraitsFromTestNode extension method in TraitsFeature).

So it's a matter of whether vstest updates the spec for /TestCaseFilter to support test properties.

FYI the vstest-docs repo has a doc about test filters here.

[OsirisTerje]

@shack05 Correct and thanks :-)

[pkrukp]

Hmm so I should file something against vstest repo? Or should this issue be moved to vstest repo (if that's possible)?

[OsirisTerje]

We can't move the issue to another org, so raise it at vstest, but we'll keep it open here. Once they have confirmed it is on their side, I'll mark this as External and VS Issue here. Nice if you add a link here too to the upcoming vstest issue.

[pkrukp]

Just to confirm - I created this issue because guys from vstest said it must be implemented in test adapter (microsoft/testfx#731 (comment)). But what you're saying is that it is already implemented in test adapter but not yet supported in vstest "spec for /TestCaseFilter" ?

[OsirisTerje]

@pkruk2 I need to check this out (aka debug) to see what happens here. I can confirm it doesn't work, and I see there is an exception (using the 4.0.0-alpha.1 adapter) that says it doesn't find the "Bug" in the selection criteria. We should be able to fix this, given the vstest guys says they're sending the correct stuff down to us.
Also, from earlier posts I see this have been working earlier, but I am not sure when this stopped working, so - we do have a regression bug here.

I also see it works in Visual Studio:

So there is something in the way that vstest and dotnet test sends the arguments.

[OsirisTerje]

@pkrukp @shack05 Can you verify if this fixes the issue for you:
NUnit3TestAdapter.4.0.0-dev.762.zip

[shack05]

Hello @OsirisTerje,
That package fixes it for me.
Thanks, that's a very useful feature.