17.3.2 fails with MethodAccessException

[js8080]

Description

I have a unit test project referencing Microsoft.NET.Test.Sdk v 17.3.2:

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
  </ItemGroup>

When I run dotnet test on that project, an exception is output:

Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Testhost process for source(s) '/app/MyProject.UnitTests.dll' exited with error: Unhandled exception. System.MethodAccessException: Attempt by method 'Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()' to access method 'Microsoft.VisualStudio.TestPlatform.CoreUtilities.Helpers.EnvironmentVariableHelper..ctor()' failed.
   at Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()
   at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Run(String[] args)
   at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Main(String[] args)
. Please check the diagnostic logs for more information.

Steps to reproduce

See above. Using .NET 6:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

Expected behavior

Version 17.2.0 does NOT encounter this error for the exact same project.

Actual behavior

Error output as shown above.

Diagnostic logs

Environment

WSL 2 using Ubuntu: lsb_release -a output:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04 LTS
Release:        22.04
Codename:       jammy

[MarcoRossignoli]

Does it repro also with 17.4.0 https://www.nuget.org/packages/Microsoft.NET.Test.Sdk/17.4.0?

[Evangelink]

I just tested using dotnet new mstest and updating the sdk to be 17.3.2 and I don't have any issue.

[cakirpro]

Does it repro also with 17.4.0 https://www.nuget.org/packages/Microsoft.NET.Test.Sdk/17.4.0?

Same problem also occurs using 17.4.1

Microsoft (R) Test Execution Command Line Tool Version 17.4.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.
vstest.console.exe "I:\a\1\s\Tests\net6.0-windows\*****.Unittest.dll"
/Settings:"I:\a\_temp\5ddomdv1gbd.tmp.runsettings"
/EnableCodeCoverage
/InIsolation
/Logger:"trx"
/TestAdapterPath:"I:\a\1\s\Tests"
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
2.5104
##[error]Testhost process for source(s) 'I:\a\1\s\Tests\net6.0-windows\*****.Unittest.dll' exited with error: Unhandled exception. System.MethodAccessException: Attempt by method 'Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()' to access method 'Microsoft.VisualStudio.TestPlatform.CoreUtilities.Helpers.EnvironmentVariableHelper..ctor()' failed.
##[error]   at Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()
##[error]   at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Run(String[] args)
##[error]   at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Main(String[] args)
##[error]. Please check the diagnostic logs for more information.

[nohwnd]

@cakirpro could you possibly send the whole content of your csproj file, please?

[Evangelink]

Closing issue as there is no activity.

[pavel-mikula-sonarsource]

Hi,
I've just seen the same error after version bumps of
Microsoft.NET.Test.Sdk to 17.7.1
MSTest.TestAdapter to 3.1.1
altcover to 8.6.68

PR: SonarSource/sonar-dotnet#7873

I can run UTs locally - those run without altcover. Also the __Instrumented in the path is a product of altcover.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Testhost process for source(s) 'C:\ProgramData\vsts-agent\1\s\analyzers\tests\SonarAnalyzer.UnitTest\bin\Release\net7.0\__Instrumented_SonarAnalyzer.UnitTest\SonarAnalyzer.UnitTest.dll' exited with error: Unhandled exception. System.MethodAccessException: Attempt by method 'Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()' to access method 'Microsoft.VisualStudio.TestPlatform.CoreUtilities.Helpers.EnvironmentVariableHelper..ctor()' failed.
   at Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor() in /_/src/Microsoft.TestPlatform.Execution.Shared/UILanguageOverride.cs:line 22
   at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Run(String[] args) in /_/src/testhost.x86/Program.cs:line 54
   at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Main(String[] args) in /_/src/testhost.x86/Program.cs:line 37

[pavel-mikula-sonarsource]

..it happened only on net7.0 TFM, it did not happen on net48 TFM.
I tried to revert altcover to the previous version 8.6.61 and it did not fix the problem.

[pavel-mikula-sonarsource]

When running without altcover, it works. So altcover does something that is not liked by the SDK or MsTest :/

Edit:
Also tried with /p:AltCoverAssemblyFilter='Moq|Humanizer|AltCover|Microsoft.VisualStudio.TestPlatform.*|.*Test', didn't help
Microsoft.NET.Test.Sdk 17.6.3 works fine

[pavel-mikula-sonarsource]

Microsoft.NET.Test.Sdk 17.7.2 still produces the error in logs, doesn't fail the UT run and doesn't produce coverage file as a side effect.

[hulken76]

The error is reproducible updating Microsoft.NET.Test.Sdk to version 17.8.0 (works with version 17.6.0) and it is only when AltCover is used.

bash:
dotnet test Tests.csproj --configuration Release /p:AltCover=true /p:CopyLocalLockFileAssemblies=true /p:AltCoverReport=coverage.opencover.xml /p:AltCoverAssemblyExcludeFilter="xunit|Moq|moq4|Dapper|Humanizer"

if /p:AltCover=true is removed it don't throw error but of course then no cover report is produced.

error:
'Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()' to access method 'Microsoft.VisualStudio.TestPlatform.CoreUtilities.Helpers.EnvironmentVariableHelper..ctor()' failed. at Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor() in /_/src/Microsoft.TestPlatform.Execution.Shared/UILanguageOverride.cs:line 22 at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Run(String[] args) in /_/src/testhost.x86/Program.cs:line 54 at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Main(String[] args) in /_/src/testhost.x86/Program.cs:line 37 . Please check the diagnostic logs for more information.

Microsoft.NET.Test.Sdk 17.8.0
.Net 7
AltCover 8.6.68

[plillevold]

The error is reproducible updating Microsoft.NET.Test.Sdk to version 17.8.0 (works with version 17.6.0) and it is only when AltCover is used.

bash: dotnet test Tests.csproj --configuration Release /p:AltCover=true /p:CopyLocalLockFileAssemblies=true /p:AltCoverReport=coverage.opencover.xml /p:AltCoverAssemblyExcludeFilter="xunit|Moq|moq4|Dapper|Humanizer"

if /p:AltCover=true is removed it don't throw error but of course then no cover report is produced.

error: 'Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor()' to access method 'Microsoft.VisualStudio.TestPlatform.CoreUtilities.Helpers.EnvironmentVariableHelper..ctor()' failed. at Microsoft.VisualStudio.TestPlatform.Execution.UiLanguageOverride..ctor() in /_/src/Microsoft.TestPlatform.Execution.Shared/UILanguageOverride.cs:line 22 at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Run(String[] args) in /_/src/testhost.x86/Program.cs:line 54 at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Main(String[] args) in /_/src/testhost.x86/Program.cs:line 37 . Please check the diagnostic logs for more information.

Microsoft.NET.Test.Sdk 17.8.0 .Net 7 AltCover 8.6.68

Same error is reproducible with
Microsoft.NET.Test.Sdk 17.9.0
.Net 8
AltCover 8.6.125

Thanks @hulken76 for pointing out that it works with SDK version 17.6.0. Can confirm that downgrading to this version fixed the issue for us as well.

[MarcoRossignoli]

if /p:AltCover=true is removed it don't throw error but of course then no cover report is produced.

@SteveGilham could it be an instrumentation issue?

@plillevold can you setup AltCover to filter out all libraries starting with Microsoft.VisualStudio.* ?

[SteveGilham]

This looks like AltCover issue 198.
The problem boils down to

• the test-runner .exe invokes methods in the supporting assemblies to which it is granted access by [InternalsVisibleTo]
• the instrumentation process rewrite of the assemblies of necessity changes to strong-names of the assemblies (unless you can provide the original strongnaming key via the --key=[path to key] argument), thus breaking the visibility.

The latest AltCover release (8.6.125) addresses this as best it can by adding new [InternalsVisibleTo] attributes to any rewritten assembly using whatever key (if any) is provided via the --strongNameKey=[path to key]. This will of course not work in the case where only the executable is rewritten, and not the called-into assembly, but it does cover the case where both are rewritten, as well as leaving the case of 'neither' undisturbed.

In general, coverage data gathered from the test runner (as from any other third-party libraries e.g. FSharp.Core.dll) is not going to be of interest, being external to the system under test, and is likely voluminous to boot, so it is best excluded from coverage at the outset. I make no assumptions about what the test configuration is like, so leave the decisions about filtering the assemblies in the output directory to the end user.

On a machine with access to the source (developer or build-with-validation machine), then the --localSource flag is the simplest. In more general contexts (e.g. operational test), using a filter like --assemblyFilter=?[company/product/project name] to exclude all third party assemblies.

For the record, the base exclusion I use in the AltCover build and validate is

        AssemblyFilter =
          [ @"\.DataCollector"
            "Sample"
            "Microsoft"
            "testhost" ]

where I exclude some of my own infrastructure, all my example code for analysis, anything Microsoft, and the test runner itself. Of course, if someone developing at Microsoft wants to use AltCover, I would not want to flatly exclude their use of the tool by hard-coding these exclusions.

[plillevold]

if /p:AltCover=true is removed it don't throw error but of course then no cover report is produced.

@SteveGilham could it be an instrumentation issue?

@plillevold can you setup AltCover to filter out all libraries starting with Microsoft.VisualStudio.* ?

In fact, we were already excluding Microsoft in the assembly filter:

/p:AltCoverAssemblyFilter="xunit|FakeItEasy|FSharp|BouncyCastle|AltCover|Spekt|Microsoft|Humanizer"

@SteveGilham put me on the track of the culprit, though, as we were not excluding testhost. Adding it to the assembly filter resolved the issue 👍

[MarcoRossignoli]

I'm going to close the issue so.