SetupFixture methods are executed even if no tests from their namespace are selected to run

[ethanmoffat]

We have a [SetUpFixture] defined outside any namespace to serve as assembly-level test initialization for some 3rd party DLLs we depend on. The tests that run in this assembly are marked with [Explicit] and a specific [Category]. We want our build server not to run the tests in this category unless they are explicitly selected (requiring the Category attribute), and we want to force local test runs to explicitly select these tests since they are long-running integration tests (requiring the Explicit attribute). Code samples are included below.

When running all tests on our build server, the tests marked [Explicit] are correctly ignored, but the [SetUp] under our [SetUpFixture] for this assembly is executed. It throws an exception because of a missing external dependency in our build environment, which causes the build to fail even though no tests were going to be executed for this assembly. I attempted adding [Explicit] to the [SetUpFixture], with no change in behavior. I do not want to add this missing dependency to this build because this build should not be executing these tests OR their setups.

This issue is the same as the one reported here. I wasn't able to find any documentation on if this was changed for NUnit 3.0, or anything in the nunit-discuss list.

We currently use NUnit 2.6.4 in our solution; I updated to 3.4.1 and did not see a change in behavior. Our tests are executed locally and in our TFS build via the NUnitTestAdapter.

Code samples (identifiers have been changed):

[SetUpFixture, Explicit, Category("SpecialCategory"), ExcludeFromCodeCoverage]
public class AssemblySetUpAndTearDown
{
   [SetUp]
   public void AssemblyInitialize()
   {
      ContainerObject.Instance.Initialize(); //this throws the exception
   }

   [TearDown]
   public void CleanupAssembly()
   {
      ContainerObject.Instance.Dispose();
   }
}

namespace Some.Test.Assembly
{
   [TestFixture, Explicit, Category("SpecialCategory"), ExcludeFromCodeCoverage]
   public class LongRunningIntegrationTests
   {
      [Test]
      public void Test1() { }

      [Test]
      public void Test2() { }

      //etc. etc.
   }
}

[CharliePoole]

First, let me clarify why it is that ExplicitAttribute can't apply to a SetUpFixture...

Explicit means "Don't run this unless it is explicitly selected." There is no way to explicitly select a SetUpFixture. It is possible to explicitly select the namespace in which the setup fixture lives, by passing that namespace as the name of a test. However, in the case of the assembly-level, there is no namespace to indicate.

It would also be possible to select a setup fixture explicitly if we honored Categories on such fixtures. However, we don't... see issue #39.

Finally, it would be possible for us to ignore the onetime setup and teardown in any fixture - setup or test - if no tests were being run. However, a few years back we had an extended community debate about this and the conclusion reached was that setups and teardowns would be run even if a fixture contained no tests at all. Of course, we could restart that discussion here.

The fundamental problem I see here, however, is that the framework is responsible for running tests in an assembly. Any filtering it does takes place within that assembly. The engine is responsible for deciding what assemblies to run and telling the framework to run them. This use case seems to fall into a crack between the two components.

So... what do to about this issue, which seems to be a real one? I'll open it up for discussion. Do we need to resolve #39, which has some complications to it, before we can resolve this? Or is there some simpler solution we could put in place?

[ethanmoffat]

Thanks for the quick response - that makes perfect sense why the Explicit attribute doesn't apply to the SetUpFixture. I should probably have clarified, that was just one thing I was trying, and I didn't want to leave that step out as one of my troubleshooting steps. I thought maybe the category would apply, but couldn't find anything in the documentation specifying whether or not it was honored.

I don't want to re-open a debate that's a couple years old - I think I agree that it makes sense to still run a SetUp/TearDown even if no tests are contained within the fixture, it just gets more complex with the addition of Explicit and Category.

An easy first step might be to add a note to the wiki documentation (maybe on the SetUpFixture page?) about this decision. That might improve the clarity, since this specific case seems to be unexpected behavior without the proper context. Either my google-fu was failing me or I wasn't sure what to search for, but I wasn't able to find the linked issue #39 or the linked bug in that issue, which provided the context needed to understand why NUnit behaves that way.

From there, I think implementing #39 would give me one solution to what I'm looking for. I'm unaware of the complications though. From what I can tell, I don't think that the work described in #39 is required in order to add the functionality requested here, but I'll leave that decision to the more experienced.

[CharliePoole]

Can you try an experiment? Put your setupfixture in a namespace that includes all your excluded tests. Does it work any differently in a namespace than it does at the top level?

[ethanmoffat]

Can you try an experiment? Put your setupfixture in a namespace that includes all your excluded tests. Does it work any differently in a namespace than it does at the top level?

I gave that a try, unfortunately the behavior is the same. When the tests are not explicitly selected, the exception is still thrown by the SetUp method regardless of whether the SetUpFixture is in the same namespace as my tests, some other namespace, or not in any namespace.

[CharliePoole]

Too bad. The easiest solution would be if the setup fixture didn't run at all if it had no selected tests under it. That's what I was hoping.

[CharliePoole]

As a stopgap measure, I added a warning about this to the pages for Explicit and Category attribute.

[616b2f]

In a new Project I had the same issue, where I wanted to exclude some long running IntegrationTests and could not completely because they had a SetUp defined which would always run regardless if the whole class where excluded or not.

Are there any known workarounds for this behavior?

[mikkelbu]

Are there any known workarounds for this behavior?

@616b2f Can you move the tests to another assembly, so that you don't have unit tests and integration tests in the same Assembly

[616b2f]

Are there any known workarounds for this behavior?

@616b2f Can you move the tests to another assembly, so that you don't have unit tests and integration tests in the same Assembly

They are already but they are in the same solution and it's really bad DX to separate them out because you then have to switch solutions only to see if after refactoring anything is okay and also tools don't consider the integration test project as a target, so you have to do it separately.

Overall this feels like a natural thing to be able to do in a testing framework IMHO.

[OsirisTerje]

@616b2f This is probably something that the adapter could handle.
Fyi: In upcoming version 4.6 there will be an option to completely ignore Explicit tests, if that is what you're using, see Adapter Issue 1111.

The behavior that a setup fixture in a namespace is executed if none of the tests there are to be executed might also possibly be better fixed in the adapter or the engine.

If you're dealing with explicit tests, it is important to notice the run mode. It can be either Explicit Run or Non-Explicit Run. Increase verbosity to see this, command line --logger="Console;verbosity=normal" -- NUnit.Verbosity=5. The IDE would show the mode regardless of these.

You would not want to run Explicit tests.

You don't write how you run these tests, so to go through some of the ways.

Explicit

If all your tests in that assembly are Explicit:

• Ensure your SetupFixture is in the default root namespace, and not outside namespaces (as an assembly SetupFixture)
• Add one dummy test in another toplevel namespace, name doesn't matter It doesn't have to do anything, just do an Assert.Pass().

The results should show, for that project, one test run (the dummy one) and importantly , zero skipped. And the mode should be Non-Explicit run.

If you run using the command line (which you also would do in CI builds, using a yaml file) , you need to add some options:

dotnet test --logger="Console;verbosity=normal" -- NUnit.Verbosity=1 NUnit.Where="Name!='Explicit'"

Resulting in:

Note that the Where filter should then be used for all filtering in the command line. Any standard filter would override the NUnit filter. The syntax for the Where filter can be found in the NUnit docs.

Category

If you use a category, and all tests are in that category, they are not executed (and nor the SetupFixture) in the IDE, even if there is no extra dummy test. The filter in the VSTest Explorer is then -Trait:"OhNo", (Category is "OhNo").

For the command line though, it will be executed without the dummy test, but adding that, it will not.

Ensure a better Yaml

For CI builds with a mixture of integration tests and unit tests, it is wise to use a naming standard for the test projects, and use that to separate the build steps.

If you name the unit test projects to have the suffix .Tests.csproj, and integration tests to have .IntegrationTests.csproj, and then use seperate dotnet test lines for each suffix, and even completely ignore one of them in a build. You could then even name the project with explicit integration tests for .ExplicitTests.csproj.

This is also a good way of making different kinds of builds, which each one covering one specific test requirement.

Repro code

See Repro code