4.4.0 breaking change for TestAssemblyLoadContext

[dave-yotta]

Looks like it may be specific to roslyn generated types:

System.InvalidCastException : [A]Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineScriptGlobals cannot be cast to [B]Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineScriptGlobals. Type A originates from 'Microsoft.CodeAnalysis.Scripting, Version=3.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' in the context '"" NUnit.Engine.Internal.TestAssemblyLoadContext #2' at location '/alloy-test/AlloyEngineTest/bin/x64/Debug/net7.0/Microsoft.CodeAnalysis.Scripting.dll'. Type B originates from 'Microsoft.CodeAnalysis.Scripting, Version=3.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' in the context 'Default' at location '/alloy-test/AlloyEngineTest/bin/x64/Debug/net7.0/Microsoft.CodeAnalysis.Scripting.dll'.

When reporting a bug, please provide the following information to speed up triage:

• NUnit and NUnit3TestAdapter versions: 4.4.0 / 17.5.0
• Visual Studio edition and full version number (see Help About): Pro 17.4.4 (but error is on dotnet test command without VS installed)
• A short repro, preferably attached or pointing to a git repo or gist: Looks like an object passed to a Script execution that comes from the test load context, while the script was compiled in the Default context.
• What .net platform and version is being targeted: 7
• If TFS/VSTS issue, what version, hosted or on-premises, and what build task you see this in: it's dotnet test running inside a docker container mcr.microsoft.com/dotnet/sdk:7.0

[paulhickman-a365]

I think I'm having the same issue. I've created a test solution at https://github.com/paulhickman-a365/Nunit440Bug

I'm using a .Net 6 project with VS2022 17.4.4. If you run the test with 4.4.0 if fails, but 4.3.1 is fine.

Foo, the class being tested is:

using Microsoft.Extensions.DependencyInjection;

namespace OtherAssembly
{
    public class Foo
    {
        public static void Bar(IServiceCollection serviceCollection)
        {

        }
    }
}

The test which loads the assembly that Foo is in dynamically is:

public class FooTest
    {
        [Test]
        public void SystemWebServicesTest() => RunTest(
            "..\\..\\..\\..\\OtherAssembly\\bin\\Debug\\net6.0",
            "OtherAssembly.dll"
        );


        private void RunTest(
            string assemblyFolder,
            string assemblyName)
        {
            //If we are running a release build of the tests, look for release builds of the rest of the code
            if (Environment.CurrentDirectory.Contains("\\Release\\", StringComparison.CurrentCulture))
            {
                assemblyFolder = assemblyFolder.Replace("\\Debug\\", "\\Release\\");
            }

            string assemblyPath = Path.Combine(assemblyFolder, assemblyName);
            if (!File.Exists(assemblyPath))
            {
                Assert.Fail($"{assemblyPath} does not exist. Have you built the full solution?.");
                return;
            }

            var context = new AssemblyLoadContext("test", true);
            context.EnterContextualReflection();
            context.Resolving += (context, childAssemblyName) =>
            {
                var tryPath = Path.Combine(assemblyFolder, childAssemblyName.Name + ".dll");
                if (File.Exists(tryPath))
                {
                    return context.LoadFromAssemblyPath(Path.Combine(Environment.CurrentDirectory, tryPath));
                }
                return null;
            };

            try
            {
                var assembly = context.LoadFromAssemblyPath(Path.Combine(Environment.CurrentDirectory, assemblyPath));
                var foo = assembly.GetTypes().FirstOrDefault(t => t.Name == "Foo");
                var bar = foo.GetMethod("Bar", BindingFlags.Static | BindingFlags.Public | BindingFlags.NonPublic);
                var services = new ServiceCollection();
                bar.Invoke(null, new[] { services }); //4.4.0 test adaptor fails here, 4.3.1 works
                context.Unload();
            }
            finally
            {
                AssemblyLoadContext.Default.EnterContextualReflection();
            }

        }
    }

The error message is:

 System.ArgumentException : Object of type 'Microsoft.Extensions.DependencyInjection.ServiceCollection' cannot be converted to type 'Microsoft.Extensions.DependencyInjection.IServiceCollection'.

[OsirisTerje]

@paulhickman-a365 @dave-yotta Does this happen on both Windows and Linux ? We have another issue reported today that seems to only happen on Linux. They look related. #1065

Thanks for the repro @paulhickman-a365 , I'll have a look at it.

Can you also just dump the stack traces you see ?

[Methuselah96]

This happens on both Windows and Linux for me.

[OsirisTerje]

@Methuselah96 What code do you have where this happens? Can you provide a short repro ?

[OsirisTerje]

@paulhickman-a365 Trying your repro on Windows, and it works there.

[paulhickman-a365]

@paulhickman-a365 Trying your repro on Windows, and it works there.

I accidentally pushed the repo on 4.3.1. I've updated the project reference to 4.4.0 now. I'm using Windows Server 2022.

[OsirisTerje]

@paulhickman-a365 Thanks! Then it failed on Windows!

[OsirisTerje]

@paulhickman-a365 The adapter 4.4.0 uses NUnit.Engine 3.16.3. This version of the engine uses Microsoft.Extensions.DependencyModel version 3.1.0 (in order to be compatible with net core 3.1, which we still support). If I change the references you have to the Microsoft.Extensions.DependencyInjection.Abstractions to the same version 3.1.0 (since I assume these are a "package deal"), then your code show compiler errors that look similar to the runtime errors.

in FooTest.cs

Since adapter 4.3.1 doesn't use this Microsoft package, it works. So, it looks like we need to rethink package version or possible the way this is solved in the engine

[daviddenis-stx]

Same issue here, SF reports a lot of failed tests with System.InvalidCastException (inhouse injection system) it happens when casting instances created using reflexion to a certain type. Like "var castedInstance = (T)instance". Works perfectly in 4.3.1. Kaboom in 4.4.0.

[paulhickman-a365]

It looks like the class ServiceCollection is in the assembly Microsoft.Extensions.DependencyInjection for versions <=5 but was moved to Microsoft.Extensions.DependencyInjection.Abstractions for version 6+

See dotnet/runtime#52284

I'm not sure I'm my issue is the same as those others are reporting anymore as it seems specific to ServiceCollection

[OsirisTerje]

@paulhickman-a365 It may explain it though, as I do see a lot of different crashes, and they all boil down to the loading of assemblies, one way or another. We have relied on using the same packages for all, keeping them at a low version, but also to avoid taking dependencies on too many external packages. I suspect this is another one of those where it doesn't work. So, we either have to package in for different frameworks, or not use this package at all. The latter seems more tempting.

PS: @daviddenis-stx What is SF ? Does the cast error happens also when not using your inhouse system - if so, can you provide a repro?

[daviddenis-stx]

Sorry, Software Factory. Upgrading the nuget package made the factory unit tests go red in any part using some degree of reflection :) NUnit3TestAdapter 4.3.1 green results, NUnit3TestAdapter 4.4.0 red results.

It fails in the region below, basically we just created an instance of something using type lookup, then calling the constructor (the instance is created correctly, that is it finds the assembly, the constructor, and invokes it nicely). Then it tries to cast the received "object" to T and fails with InvalidCastException. For example an instance of class MyThing : IMyThing will fail on the cast line, throwing an exception "Cannot cast to IMyThing". I suspect there's something weird with the reflexive loading of assemblies and the loading of types in 4.4.0 ?

        T cast;
        try
        {
            cast = (T)instance;

            // Log
            Logger.Debug($"CreateInstance: Instance was casted to {typeof(T)}");
        }
        catch (InvalidCastException invalidCastException)
        {
            // Log
            Logger.Error($"CreateInstance: Instance failed to be casted to {typeof(T)} with exception", invalidCastException);
            return Status.Failure<T>($"Instance failed to be casted to {typeof(T)} with exception", invalidCastException);
        }

[daviddenis-stx]

Perhaps the bump to engine 3.16.2 ? I can see this thing is quite new https://github.com/nunit/nunit-console/commits/485153dd0b414f71e2432aadc1028afce114e7cf/src/NUnitEngine/nunit.engine.core/Internal/TestAssemblyLoadContext.cs

[OsirisTerje]

@daviddenis-stx Yes, I think that's where it happens, and you see that the MS package was introduced there.