Stopping on exception, for .net framework

[tapika]

I've took fork from xunit v2, and bugfixed it a little bit, but also added new feature - being able to stop on exception
where it occurred.

I've slightly modified original xunit source code to able to compile and debug for one platform only - but after that some of tests disappeared from unit test framework.

Wondering also what is the target with xunit v3, when you're planning to release it ?

[bradwilson]

I'm not clear how "stop on exceptions" is a test framework feature, as opposed to a debugger feature. Can you provide an example and/or intended usage?

As for schedule, we don't have one, because work just continues and I'm unsure when 3.0 will be ready for RTW. I have been making steady progress, though! Please feel free to watch my Twitch streams to follow along and interact!

[tapika]

I'm not clear how "stop on exceptions" is a test framework feature, as opposed to a debugger feature. Can you provide an example and/or intended usage?

I have implemented this feature into xunit v2 fork - unfortunately my own xunit fork is not open source code at the moment, but I could contribute my own changes if necessary.

At the moment system works for that I'm intercepting AppDomain.FirstChanceException and via separate config file I'm configuring whether or not to catch / not to catch given exception.

I have tried also to send you mail to bradwilson at github dot com - but without any reply. (maybe not using this mail?)

Once exception occurs and it's observed exception - process is halted and xunit sends mail to developer to debug that xunit session.

As for schedule, we don't have one, because work just continues and I'm unsure when 3.0 will be ready for RTW. I have been making steady progress

It would be nice to integrate my changes into xunit v3, but I haven't yet checked that source code.

I had some issues with debugging xunit in overall, would like to ask you how you're doing it. At the moment I've disabled xunit unit testing, but I'm not very proud of doing this.

[bradwilson]

At the moment system works for that I'm intercepting AppDomain.FirstChanceException and via separate config file I'm configuring whether or not to catch / not to catch given exception.

Sorry, this doesn't sound like a feature I believe we would be interested in. First chance exceptions are a normal part of the operation of the framework; trying to intercept and filter them is an extremely niche case.

I believe you're best served by continuing with your own fork.

[tapika]

At the moment system works for that I'm intercepting AppDomain.FirstChanceException and via separate config file I'm configuring whether or not to catch / not to catch given exception.

Sorry, this doesn't sound like a feature I believe we would be interested in. First chance exceptions are a normal part of the operation of the framework; trying to intercept and filter them is an extremely niche case.

I believe you're best served by continuing with your own fork.

I agree that if code is simple, small, then no need for heavy exception filtering engine - that is heavy artillery indeed. But keeping my own xuint fork branch also does not makes sense, as its not me who mainly contrubutes to xunit test framework. Would you consider to have some sort of plugin api, where exception filtering would be done on my own plugin side, but xunit would offer that plugin api? This way I dont need to release xunit binaries (and collide your vs my xunit versioning), and maybe plugin api could be used by someone else later on?

[bradwilson]

I'm unclear what this plugin API would be. Can you provide an example of what you envision, particularly giving code samples of how and where such a plugin would reside, and what additional behavior you'd expect from us?

[tapika]

That is a good question.

Theoretically we need to enable / disable exception handling functionality itself, at the moment I have added extra command line argument

test_IntegrationTests.exe -diagnostics -xml TestResults\test_IntegrationTests.xml -exceptionconfigfile ...\configExceptions.xml

(Where test_IntegrationTests.exe executes xunit.console.exe:s Main.)

This is theoretically needed because on developers machine this special kind of exception handling is not enabled, but on build machine exception handling is enabled.

In theory exception handling needs to be enabled either globally (I'm doing this at the moment), or locally per assembly.

Need some sort of one time initialization function which will parse config file (and throws exception if it's not valid)
and then would hook current AppDomain.

AppDomain.CurrentDomain.FirstChanceException += FirstChanceException;

I by myself hooking this by adding extra function
src/common/AssemblyResolution/AssemblyHelper_Desktop.cs / EnableInternalExceptionCatching

and then calling it from
src\xunit.runner.utility\Frameworks\v2\Xunit2Discoverer.cs

#if NETFRAMEWORK
            var runnerUtilityAssemblyLocation = Path.GetDirectoryName(typeof(AssemblyHelper).Assembly.GetLocalCodeBase());
            assemblyHelper = AppDomain.CreateObjectFrom<AssemblyHelper>(typeof(AssemblyHelper).Assembly.Location, typeof(AssemblyHelper).FullName, runnerUtilityAssemblyLocation);
            assemblyHelper.MessageSink = DiagnosticMessageSink;

            string file = configuration.ExceptionConfigFile;
            if (!string.IsNullOrEmpty(file))
            {
                if (!File.Exists(file))
                {
                    DiagnosticMessageSink.OnMessage(new DiagnosticMessage($"Warning: Configuration file {file} does not exists, starting without exceptions enabled."));
                }
                else
                { 
                    assemblyHelper.EnableInternalExceptionCatching(file);
                }
            }
#endif

Also I needed media where to print exception stack traces / messages, and used IMessageSink DiagnosticMessageSink for that purpose.

What I have noticed afterwards is that exception might occur in multithreading context, and since media has it's own lock mechanisms (e.g. print to console) - and this dramatically delays output, but also execution of tests - exception starts to occur less frequently.

It would make sense that output media is fully asynchronous and does not block the execution.

Exception handling (my own side) should be simple - if have preconfigured "string" in exception type, name or call stack -then
send an e-mail and halt the execution.

Mail could be simple:

From: xunit.console-dontreply@mycompany.com
To: my mail
Subject: Unhandled exception

Observed exception occurred on host: BUILDER-02
Process id: 4536
Please attach debugger to resume testing.
More information:

[exception #2234]: System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception. 

Then developer basically will try to attach debugger and analyze what does not work.

Theoretically last message should be possible to flush out, as it will be an message shown in build system (e.g. azure devops)
and people will know that something very tragical happened. :)

[tapika]

I have done also various non related items, like for example added capability to list tests available in test assembly.

Could commit that separately as you request it.