Exception in OneTimeSetUp has no stack trace

[provegard]

When an exception is thrown in code called from a OneTimeSetUp method, I get:

Exception doesn't have a stacktrace

Code to demonstrate the problem:

public class Tests
{
    [OneTimeSetUp]
    public void Setup()
    {
        throw new Exception("oops");
    }

    [Test]
    public void Test1()
    {
        Assert.Pass();
    }
}

Versions:

• .NET Core 3.0
• NUnit 3.12.0
• NUnit3TestAdapter 3.15.1
• Microsoft.NET.Test.SDK 16.3.0

[CharliePoole]

Please clarify "I get..." That is... How are you running the tests and where do you see this message?

[provegard]

I run them in one of two ways:

1. Visual Studio 2019 with ReSharper. The "Exception doesn't have a stacktrace" message is printed as the test failure ouput:

2. Using dotnet vstest with TRX reporting. In this case the report contains:

<UnitTestResult ...>
  <Output>
    <ErrorInfo>
      <Message>OneTimeSetUp: System.Exception : oops</Message>
    </ErrorInfo>
  </Output>
</UnitTestResult>

Changing to SetUp results in the full stack trace being printed, so it appears to be a problem with OneTimeSetUp.

[CharliePoole]

If an exception is thrown in OneTimeSetUp, then the tests are not run. Consequently no exception is ever thrown for the individual test case. Instead, it's thrown for the suite or fixture.

Because TestExplorer (and I believe Resharper as well) never reports errors on the fixture, we report the message as if the exception was thrown for the test case itself - otherwise, you would never see it. We do not report the stack trace simply because that would be a lot of repetition and you can generally find where the exception was thrown pretty easily, if not by inspection, then in the debugger.

If you run tests using nunit3-console, we actually report errors at the fixture level as well, so you can see the stack trace there. The same is true if you run using the TestCentric GUI.

IOW, at least in my view, this is a limitation of using Test Explorer, which only treats test methods as tests, not test suites.

[provegard]

We do not report the stack trace simply because that would be a lot of repetition and you can generally find where the exception was thrown pretty easily, if not by inspection, then in the debugger.

Normally, yes. In my case, I was modifying an application to run on both Windows and Linux, and on Linux I got exceptions from GDI-related code ("Parameter is invalid" - not easy to figure that out :) ). I did not have a full IDE available there, only the dotnet CLI tooling. As I mentioned, I could change to SetUp and see the stack trace, but it would have been nicer to be able to pinpoint the root cause quicker.

Occasionally, I have intermittent test failures on the build server, and in those cases the suppressed stack trace makes it very hard to know what's going on.

IOW, at least in my view, this is a limitation of using Test Explorer, which only treats test methods as tests, not test suites.

Forgive me if I misunderstand, but if I interpret you correctly it's NUnit that suppresses the stack trace, and since I observe that an exception has occurred both in Test Explorer/ReSharper and in a test report, it seems to me that the problem lies with NUnit. Or rather, that the only place where a fix is possible is within NUnit.

[provegard]

Btw, I agree that including the stack trace for each test is a lot of repetetion. However, if I use SetUp that is exactly what happens - I see the exception will full stack trace for each test. With OneTimeSetUp, I see the exception without stack trace for each test. I'd argue that stack trace repetition is better than no stack trace.

[CharliePoole]

Let me try again...

In your sample code, NUnit considers Tests to be a test. It also considers Test1 to be a test contained in Tests.

The implementation of Test1 is in the [SetUp], [Test] and [TearDown] methods. If an exception is thrown in one of those methods, NUnit reports it as an error in Test1. Since the same [SetUp] method is used for all test cases within a fixture, it's possible for the exception to be thrown more than once so, of course, it is reported more than once.

The implementation code for Tests is contained in the [OneTimeSetUp] and [OneTimeTearDown] methods. If an exception is thrown in either of those methods, NUnit reports it as an error in the test named Tests. Both message and stack trace are included. In addition, NUnit reports the same error message for each contained test (for Test1 in this case) as an indicator that you should look at the containing test (fixture Tests). Think of this report as a sort of footnote saying "See appendix." That is, the actual error didn't occur in Test1, because Test1 never executed, you need to see the results of Tests to understand why.

Unfortunately, you can't see that! The NUnit3 VS Adapter has the job of translating NUnit's view of the tests to Test Explorer's. In Test Explorer's view, only Test1 is a test. There is no obvious way for the adapter to pass back the results of Tests because it is never asked about it. That's the difficulty in translating between two entirely different views of what a "test" is.

There doesn't seem to me to be anything more that NUnit itself needs to do. The information is there and is made visible by other runners, like nunit3-console and testcentric-gui. In your case, the actual runner is the dotnet CLI, making use of the adapter. The adapter could do something, but that would probably have to vary depending on the execution environment. When running under the IDE, it could dump more info to the Output window - although that might get very cluttered. When running under vstest or a dotnet command it might write to stdout. In any case, it would have to bypass the normal reporting mechanism to get the information out about a failure that takes place in no recognized "test".

[provegard]

Ok, I understand now. Thank you for the detailed explanation! I'll report the problem against the other involved components (Test explorer/Resharper/dotnet tooling).

[CharliePoole]

Or the NUnit3 VS adapter, to which we could transfer this issue if you prefer.

[provegard]

Aha, I see now that I missed the part about a possible fix/workaround in the adapter. Yes, moving the issue there would be good. Would you like me to do it?

[CharliePoole]

Transferred!

@OsirisTerje I realize you may have other issues that relate to this one, but I transferred it in case it has added useful info.

[OsirisTerje]

Thanks :-) But, closed ?

[CharliePoole]

User closed... I meant to reopen.

[Brondahl]

FYI, you can sort-of bypass this issue, by wrapping all of your OneTimeSetUp methods in:

try
{
  //... do setup
}
catch(Exception e)
{
  throw new Exception(e.ToString(), e);
}

Kinda sucks, but it's better than what you get at the moment.

[bronumski]

I am also hitting this issue with the Test Explorer (or more specifically with ReSharper). I write a number of behavioral tests where the arrange and act happen in the one time setup and the logical assertions are the individual tests. If an exception is thrown during the arrange or the act it is a pain to debug. Work around suggestion accepted.

[radiy]

Because TestExplorer (and I believe Resharper as well) never reports errors on the fixture, we report the message as if the exception was thrown for the test case itself - otherwise, you would never see it. We do not report the stack trace simply because that would be a lot of repetition and you can generally find where the exception was thrown pretty easily, if not by inspection, then in the debugger.

Please reconsider, this is HUGE waste of time. Think about CI scenario, sometimes setup is pretty complex and when it went wrong all you get is NullReferenceException.

[bronumski]

I wan't 100% sure so I checked and Rider and ReSharper do report errors on the fixture:

[CharliePoole]

@radiy I only posted a description of how TestExplorer works and how the adapter tries to compensate for it. It's not possible to "reconsider" that - it's just a fact. Are you using TestExplorer in your CI? That seems unlikely. You may want to post info about what you are using, what doesn't work for you and what you would like changed.

To be clear, I haven't worked on the adapter for years myself, so I'm only making suggestions.

@bronumski I have no idea how they do that, but apparently they have figured out a way. As commercial companies, they're probably not going to tell us. 😞 It's possible they get the info directly from NUnit and not through the adapter.

Still, that's good information. It shows that any of these runners - test explorer included - could access and display the info you want to see. That was kind of my point. Although I don't work on the adapter any longer, I do remember how frustrating it was that it tends to be blamed for limitations in the code that calls it.

IMO, if you simply accept the fact that Test Explorer does not know about fixture-level failures and will never report them directly, you are a step ahead. Then you can start to imagine workarounds, where the adapter displays the info you need "on the side" i.e. in a fake test or in the output window or as some visible text output elsewhere in the interface.

[radiy]

@radiy I only posted a description of how TestExplorer works and how the adapter tries to compensate for it. It's not possible to "reconsider" that - it's just a fact. Are you using TestExplorer in your CI? That seems unlikely. You may want to post info about what you are using, what doesn't work for you and what you would like changed.

Looks like I was a little rushed.
I will try to describe my issue.
Of course I am not using Test Excelorer in CI, I run tests with dotnet test and collect test results from trx-files.
Development environment and test environment are different, so sometimes unexpected errors occur,
finding root case when you have only exception message in many cases impossible
and at the same time it`s trivial to find root case if you have a complete exception text.
For me, the main problem is that exceptions in the OneTimeSetUp method are neither recorded in trx-file, nor in standard output.

Do you think it`s possible to log exception only once, on the first test method.

[CharliePoole]

Sounds possible! I believe the adapter knows when it's running under vstest vs test explorer. @OsirisTerje will have to give the final answer, since he does the work. 😄

[OsirisTerje]

@radiy @CharliePoole We do get the stack trace so it should be possible to get this out somehow. If we focus on dotnet test and vstest commandline scenarios it might be even easier.

[svengeance]

@OsirisTerje I recently worked in the adapter quite a bit for a personal project. Is this an issue you'd like a contribution for? I imagine if the TestRunner bubbles these errors up to the adapter we can just write the stacktrace in addition to the message, instead of just the message?

[OsirisTerje]

@svengeance Absolutely! Feel free to add a PR :-)

[svengeance]

@OsirisTerje To be sure - what's the intended solution here? Log the stacktrace on the first test method, log it to the Test Output window, log the full stacktrace on all tests, etc?

For giggles I'm investigating manipulating the text on the Fixture itself, but I don't have high hopes for that..

[jamers99]

Nice, any progress on this? I'm sure the people over at this issue would appreciate this.

[OsirisTerje]

@jamers99 @provegard @svengeance
NUnit3TestAdapter.4.0.0-dev.751.zip
Please check the enclosed adapter. It will output the stack trace for the exceptions in the overall message part. It will not come out per test. It will be included in the trx file, and in Visual Studio Console Output window for testing.