Exceptions thrown during FactAttribute.Skip or derived constructor cause test to be silently skipped

[jasonmalinowski]

Asking first as a discussion question since it's entirely possible this falls in the category "don't do that" and there's a better way to achieve what we want...

Roslyn has a ConditionalFact attribute which allows us to specify certain conditions for certain tests. For example, tests that test behaviors for certain operating systems, bitnesses, locales, etc. It's implemented by deriving FactAttribute, and in the constructor it checks some "skip condition" types and if we should skip sets FactAttribute.Skip.

I recently discovered that one of our condition types were throwing exceptions, which meant that our ConditionalFactAttribtue constructor was going to throw. It seems in this case xunit no longer was discovering our types and silently skipping the tests, in that it doesn't even count as "skipped" in the first place. Figuring this was a fun .NET behavior where the attribute can't be loaded at all if it throws (and not something xunit could really fix), I tried instead overriding FactAttribute.Skip to run our custom code, but it seems exceptions thrown from there also cause the test not to be discovered. For example:

using Xunit.Sdk;

namespace TestProject2
{
    public class UnitTest1
    {
        [ThrowInSkipFact]
        public void Test1()
        {

        }
    }

    public class ThrowInSkipFact : FactAttribute
    {
        public override string Skip
        {
            get => throw new Exception();
            set => base.Skip = value;
        }
    }
}

Running this with 'dotnet test' just says no tests could be discovered.

So two questions:

1. Is this a bug in xUnit that it seems the exception is swallowed somewhere?
2. How should we be implementing a skip attribute like this?

[bradwilson]

This feels like a bug to me. Feel free to open a new issue (or convert this).