Is there a good example of excluding modules from code coverage using vstest.console.exe only? #2038

shueybubbles · 2019-05-30T17:04:25Z

I have seen examples of using a .runsettings file to exclude modules, but when I try to use a runsettings file with my tests and the vstest.console from the microsoft.testplatform nuget package it gives me an error about not being able to launch codecoverage.exe.
I see that vstest.console.exe accepts Runsettings parameters; is there a way to list modules to exclude in that command line?

vagisha-nidhi · 2019-05-31T06:31:00Z

@shueybubbles
Sharing example of using .runsettings to exclude modules: https://docs.microsoft.com/en-gb/visualstudio/test/customizing-code-coverage-analysis?view=vs-2019#sample and this is how you configure unit tests with .runsettings file.

Also, how are you trying to collect code coverage? We recommend folks to always use vstest.console to gather coverage. Can you also share logs? https://github.com/microsoft/vstest-docs/blob/master/docs/diagnose.md contains how to collect logs for test paltform.

shueybubbles · 2019-05-31T15:26:46Z

Here's a sample command line using a .runsettings file. Notice I'm trying to use the Microsoft.TestPlatform nuget package so I don't have to install VS on build agents.

"D:\git\SqlManagementObjects\packages\Microsoft.TestPlatform.16.1.0\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "D:\git\SqlManagementObjects\target\distrib\Debug\net461\ConnectionInfoUnitTests.dll"
 /TestCaseFilter:"TestCategory=Unit" /Settings:D:\git\SqlManagementObjects\src\UnitTest\unittest.runsettings /logger:console /logger:trx;LogFileName=Testrun_ConnectionInfoUnitTests.trx /ResultsDirectory:"D:\git\SqlManagement
 Objects\target\distrib\Debug\net461\UnitTestResults"

the test immediately pops up an error dialog
testhost.x86.exe - Fatal error
CLR error: 80004005.
The program will now terminate.
OK

with this output

Microsoft (R) Test Execution Command Line Tool Version 16.1.0
  Copyright (c) Microsoft Corporation.  All rights reserved.

  Starting test execution, please wait...
  M i c r o s o f t   ( R )   C o v e r a g e   C o l l e c t i o n   T o o l   V e r s i o n   1 6 . 0 . 3 0 3 1 9 . 0


   C o p y r i g h t   ( c )   M i c r o s o f t   C o r p o r a t i o n .     A l l   r i g h t s   r e s e r v e d .




   2.7439
  Data collector 'Code Coverage' message: Failed to launch CodeCoverage.exe. No code coverage will be collected..
  vstest.console process failed to connect to testhost process after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

  Test Run Aborted.
  Results File: D:\git\SqlManagementObjects\target\distrib\Debug\net461\UnitTestResults\Testrun_ConnectionInfoUnitTests.trx

Alternately, if I remove the .runsettings file and use --collect, no problem:

 "D:\git\SqlManagementObjects\packages\Microsoft.TestPlatform.16.1.0\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "D:\git\SqlManagementObjects\target\distrib\Debug\net461\ConnectionInfoUnitTests.dll"
  /TestCaseFilter:"TestCategory=Unit" --collect:"Code Coverage" /logger:console /logger:trx;LogFileName=Testrun_ConnectionInfoUnitTests.trx /ResultsDirectory:"D:\git\SqlManagementObjects\target\distrib\Debug\net461\UnitTestRe
  sults"
  Microsoft (R) Test Execution Command Line Tool Version 16.1.0
  Copyright (c) Microsoft Corporation.  All rights reserved.

  Starting test execution, please wait...
  M i c r o s o f t   ( R )   C o v e r a g e   C o l l e c t i o n   T o o l   V e r s i o n   1 6 . 0 . 3 0 3 1 9 . 0


   C o p y r i g h t   ( c )   M i c r o s o f t   C o r p o r a t i o n .     A l l   r i g h t s   r e s e r v e d .




   2.8319
    √ VerifyInteractiveModeConnectionSettings [436ms]
    √ SqlConnectionInfo_supports_ApplicationIntent [34ms]
    √ TypeConverter_DatabaseEngineEdition_ConvertsEnumToStringAndBack [25ms]
    √ TypeConverter_DatabaseEngineType_ConvertsEnumToStringAndBack [< 1ms]
    √ When_AccessToken_is_null_SqlConnectionObject_has_null_AccessToken [36ms]
    √ When_AccessToken_is_not_null_SqlConnectionObject_has_valid_AccessToken [218ms]
    √ When_AccessToken_is_in_config_SqlConnectionObject_has_valid_AccessToken [2ms]
    √ ServerConnection_copies_AccessToken_from_source_ServerConnection [4ms]
    √ ServerConnection_constructor_throws_if_AccessToken_is_null_and_UseIntegratedSecurity_is_false [< 1ms]
    √ When_AccessToken_changes_SqlConnectionObject_has_new_AccessToken_value [2ms]
  M i c r o s o f t   ( R )   C o v e r a g e   C o l l e c t i o n   T o o l   V e r s i o n   1 6 . 0 . 3 0 3 1 9 . 0


   C o p y r i g h t   ( c )   M i c r o s o f t   C o r p o r a t i o n .     A l l   r i g h t s   r e s e r v e d .




     √ When_constructed_from_SqlConnection_with_sql_auth_LoginSecure_is_false [1ms]
    √ When_SqlConnection_uses_SqlCredential_ServerConnection_copies_it [5ms]
    √ When_assigned_ConnectionString_after_construction_Clone_is_correct [1ms]
    √ GetServerInformation_executes_one_query_and_accepts_three_result_tables [199ms]
    √ When_ServerVersion_is_less_than_14_HostPlatform_is_hard_coded_Windows [3ms]

  Attachments:
    D:\git\SqlManagementObjects\target\distrib\Debug\net461\UnitTestResults\19631726-cdda-4591-af67-432b4bbdbf68\davidshi_DAVIDSHI-WIN10 2019-05-31 11_24_02.coverage

  Test Run Successful.
  Total tests: 15
       Passed: 15
   Total time: 3.8732 Seconds
  Results File: D:\git\SqlManagementObjects\target\distrib\Debug\net461\UnitTestResults\Testrun_ConnectionInfoUnitTests.trx

The only issue I have with this usage is the .coverage file has data for DLLs I don't care about like moq and NUnit which I want to exclude.

vagisha-nidhi · 2019-06-03T10:15:24Z

@shueybubbles
This is what I have tried, and code coverage seems to be working fine along with the runsettings provided.

PS E:\agent\_work\_tool\VsTest\16.1.1\x64\tools\net451\Common7\IDE\Extensions\TestPlatform> .\vstest.console.exe "C:\Users\vanidhi\source\repos\UnitTestProject26\UnitTestProject26\bin\Debug\netcoreapp2.0\UnitTestProject26.dll" --settings:"E:\codecoverage.runsettings"
Microsoft (R) Test Execution Command Line Tool Version 16.1.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
  √ TestMethod1 [9ms]

Attachments:
  E:\agent\_work\_tool\VsTest\16.1.1\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\TestResults\efbf9ae2-fb85-4e28-826e-9070341b5508\vanidhi_VANIDHI-DESK 2019-06-03 15_31_17.coverage

This is the exact runsettings I provided: https://docs.microsoft.com/en-gb/visualstudio/test/customizing-code-coverage-analysis?view=vs-2019#sample-runsettings-file
I am getting the same result with Test Platform version 16.1.0.

Can you please share your runsettings file as well as diag logs with us?
You can collect the diag logs by adding --diag:"E:\log.txt" to the command line invocation.

shueybubbles · 2019-06-03T14:31:25Z

It works for me with the same file, but as soon as I remove a few of the excludes sections (my binaries use the same public key listed , for example) it fails with this error in the log.host

TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183983555, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolving assembly.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183985018, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolved from cache.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183986209, testhost.dll, CurrentDomain_AssemblyResolve: Resolving assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183987640, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolving assembly.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183988858, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolved from cache.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183989996, testhost.dll, CurrentDomain_AssemblyResolve: Resolving assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183991080, testhost.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.065, 11942183995574, testhost.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Warning: 0 : 3464, 4, 2019/06/03, 10:36:45.065, 11942184001821, testhost.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.VisualStudio.TraceDataCollector, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at System.Reflection.AssemblyExtensions.GetTypes(Assembly assembly)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

TpTrace Verbose: 0 : 19196, 4, 2019/06/03, 10:15:33.531, 11929468585820, testhost.dll, TestExecutionRecorder.RecordStart: Starting test: Microsoft.SqlServer.ConnectionInfoUnitTests.ConnectionSettingsTests.SqlConnectionInfo_supports_ApplicationIntent.
TpTrace Error: 0 : 19196, 4, 2019/06/03, 10:15:33.583, 11929469110382, testhost.dll, TestCaseEventsHandler.RaiseTestCaseStart: Exception occurred while calling handler of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager for TestCaseStartEventArgs: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveRawMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.DataCollectionTestCaseEventSender.SendTestCaseStart(TestCaseStartEventArgs e)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at Microsoft.VisualStudio.TestPlatform.Utilities.MulticastDelegateUtilities.SafeInvoke(Delegate delegates, Object sender, EventArgs args, String traceDisplayName)
TpTrace Information: 0 : 19196, 4, 2019/06/03, 10:15:33.749, 11929470768379, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources: Resolving assembly.
TpTrace Information: 0 : 19196, 4, 2019/06/03, 10:15:33.749, 11929470771554, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources: Failed to load assembly.

shueybubbles · 2019-06-03T14:51:34Z

ok. I figured out that the settings file parser is not resilient to improperly formed regular expression on excludes

IE this leads to the failure:

<ModulePath>*moq.*</ModulePath>

Note the missing leading character class.

 .*moq.* works fine.

So it's a user error but the tool should be more resilient.

vagisha-nidhi · 2019-06-03T15:00:21Z

It works for me with the same file, but as soon as I remove a few of the excludes sections (my binaries use the same public key listed , for example) it fails with this error in the log.host

TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183983555, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolving assembly.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183985018, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolved from cache.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183986209, testhost.dll, CurrentDomain_AssemblyResolve: Resolving assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183987640, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolving assembly.
TpTrace Information: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183988858, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.ArchitectureTools.PEReader: Resolved from cache.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183989996, testhost.dll, CurrentDomain_AssemblyResolve: Resolving assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.064, 11942183991080, testhost.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Verbose: 0 : 3464, 4, 2019/06/03, 10:36:45.065, 11942183995574, testhost.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader'.
TpTrace Warning: 0 : 3464, 4, 2019/06/03, 10:36:45.065, 11942184001821, testhost.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.VisualStudio.TraceDataCollector, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at System.Reflection.AssemblyExtensions.GetTypes(Assembly assembly)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.ArchitectureTools.PEReader, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

TpTrace Verbose: 0 : 19196, 4, 2019/06/03, 10:15:33.531, 11929468585820, testhost.dll, TestExecutionRecorder.RecordStart: Starting test: Microsoft.SqlServer.ConnectionInfoUnitTests.ConnectionSettingsTests.SqlConnectionInfo_supports_ApplicationIntent.
TpTrace Error: 0 : 19196, 4, 2019/06/03, 10:15:33.583, 11929469110382, testhost.dll, TestCaseEventsHandler.RaiseTestCaseStart: Exception occurred while calling handler of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager for TestCaseStartEventArgs: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveRawMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveMessage()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.DataCollectionTestCaseEventSender.SendTestCaseStart(TestCaseStartEventArgs e)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at Microsoft.VisualStudio.TestPlatform.Utilities.MulticastDelegateUtilities.SafeInvoke(Delegate delegates, Object sender, EventArgs args, String traceDisplayName)
TpTrace Information: 0 : 19196, 4, 2019/06/03, 10:15:33.749, 11929470768379, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources: Resolving assembly.
TpTrace Information: 0 : 19196, 4, 2019/06/03, 10:15:33.749, 11929470771554, testhost.dll, AssemblyResolver.OnResolve: Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources: Failed to load assembly.

This issue is a different issue. #2008. We will be giving fix for this soon.

ok. I figured out that the settings file parser is not resilient to improperly formed regular expression on excludes

IE this leads to the failure:
<ModulePath>*moq.*</ModulePath>

Note the missing leading character class.

 .*moq.* works fine.
So it's a user error but the tool should be more resilient.

Seems like you are able to exclude modules in code coverage now.
Closing the thread. Feel free to open another issue if needed.

PilotBob · 2021-03-08T23:16:35Z

I'm pulling my hair out trying to get this to work. I am using the sample .runsettings file from the previously linked path.

I am running:
vstest.console.exe "**\bin***Test.dll" --settings:c:\source\CodeCoverage.runsettings --testcasefilter:Backend!=Database --diag:"c:\source\log.txt"

(do I need --enablecodecoverage? Seems to be the same with or without it)

Then:
$TestResultsPath = resolve-path "c:\source\TestResults***.coverage" | Select -ExpandProperty Path
CodeCoverage.exe analyze /output:DynamicCodeCoverage.coveragexml "$TestResultsPath"

The XML contains DLLs with Microsoft in the name. (which to me looks like they should be excluded).

I have added XUnit to exclude and still get XUnit listed in the XML and the report generated from it with report generator.

.*\.dll$ .*\.exe$ .*CPPUnitTestFramework.* .*XUnit.*

XUnit dlls are included in the reports as well.

I have looked at the log and it shows my runsettings with the added XUnit exclusion.

log.txt

BTW: I am running this in the mcr.microsoft.com/dotnet/framework/sdk:4.8 windows container.

Any help appreciated. I would expect not to see Microsoft.SignalR.xx or XUnit.* dlls at all.

jayaranigarg added the question label May 31, 2019

vagisha-nidhi closed this as completed Jun 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is there a good example of excluding modules from code coverage using vstest.console.exe only? #2038

Is there a good example of excluding modules from code coverage using vstest.console.exe only? #2038

shueybubbles commented May 30, 2019

vagisha-nidhi commented May 31, 2019

shueybubbles commented May 31, 2019

vagisha-nidhi commented Jun 3, 2019

shueybubbles commented Jun 3, 2019 •

edited

shueybubbles commented Jun 3, 2019 •

edited

vagisha-nidhi commented Jun 3, 2019

PilotBob commented Mar 8, 2021

Is there a good example of excluding modules from code coverage using vstest.console.exe only? #2038

Is there a good example of excluding modules from code coverage using vstest.console.exe only? #2038

Comments

shueybubbles commented May 30, 2019

vagisha-nidhi commented May 31, 2019

shueybubbles commented May 31, 2019

vagisha-nidhi commented Jun 3, 2019

shueybubbles commented Jun 3, 2019 • edited

shueybubbles commented Jun 3, 2019 • edited

vagisha-nidhi commented Jun 3, 2019

PilotBob commented Mar 8, 2021

shueybubbles commented Jun 3, 2019 •

edited

shueybubbles commented Jun 3, 2019 •

edited