[BUG] Namespace / assembly exclusions are not being obeyed from the .runsettings file.

[xantari]

Describe the bug

Namespace / assembly exclusions are not being obeyed from the .runsettings file.

To Reproduce

Example .runsettings:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
    <DataCollectionRunSettings>
        <DataCollectors>
            <!--https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md-->
            <DataCollector friendlyName="XPlat code coverage">
                <Configuration>
                    <Format>cobertura</Format>
                    <!--https://github.com/tonerdo/coverlet/blob/master/Documentation/MSBuildIntegration.md#excluding-from-coverage-->
                    <Exclude>[*]GalaxyDigital.Console.*,[*]GalaxyDigital.Api.*</Exclude>
                    <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
                </Configuration>
            </DataCollector>
            <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
                <Configuration>
                    <CodeCoverage>
                        <ModulePaths>
                            <Exclude>
                                <ModulePath>GalaxyDigital.Console.*</ModulePath>
                                <ModulePath>GalaxyDigital.Api.Client.*</ModulePath>
                            </Exclude>
                        </ModulePaths>

                        <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
                        <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
                        <CollectFromChildProcesses>True</CollectFromChildProcesses>
                        <CollectAspDotNet>False</CollectAspDotNet>
                    </CodeCoverage>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

When running the following "GalaxyDigital.Api.Client" is not excluded:

dotnet test -s GalaxyDigital.Services.Tests\CodeCoverage.runsettings --no-build --collect:"XPlat Code Coverage"

You get the following output:

Microsoft (R) Test Execution Command Line Tool Version 17.8.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 213 ms - GalaxyDigital.Services.Tests.dll (net6.0)

Calculating coverage result...
  Generating report 'TestResults\coverage.cobertura.xml'

+-------------------------------+-------+--------+--------+
| Module                        | Line  | Branch | Method |
+-------------------------------+-------+--------+--------+
| GalaxyDigital.Api.Client | 0%    | 0%     | 0%     |
+-------------------------------+-------+--------+--------+
| GalaxyDigital.Services   | 3.59% | 1.66%  | 26.98% |
+-------------------------------+-------+--------+--------+

+---------+-------+--------+--------+
|         | Line  | Branch | Method |
+---------+-------+--------+--------+
| Total   | 0.78% | 0.12%  | 2.64%  |
+---------+-------+--------+--------+
| Average | 1.79% | 0.83%  | 13.49% |
+---------+-------+--------+--------+

2024-01-31T13:22:25: Writing report file 'TestResults\html-coverage-report\index.html'
2024-01-31T13:22:26: Report generation took 4.6 seconds

Attachments:
  C:\TFS\GalaxyDigital.Services.Tests\TestResults\08806ac0-ca8a-4319-bc19-7ecff4cdd774\test_2024-01-31.13_22_20.coverage
  C:\TFS\GalaxyDigital.Services.Tests\TestResults\08806ac0-ca8a-4319-bc19-7ecff4cdd774\coverage.cobertura.xml

Notice how GalaxyDigital.Api.Client was not excluded, even though it was excluded in the runsettings file.

If you change the command line to the following:

 dotnet test -s GalaxyDigital.Services.Tests\CodeCoverage.runsettings --no-build --collect:"XPlat Code Coverage" /p:Exclude="[*]GalaxyDigital.Api.*"

Then you get this output:

Microsoft (R) Test Execution Command Line Tool Version 17.8.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 190 ms - GalaxyDigital.Services.Tests.dll (net6.0)

Calculating coverage result...
  Generating report 'TestResults\coverage.cobertura.xml'

+-------------------------------+-------+--------+--------+
| Module                        | Line  | Branch | Method |
+-------------------------------+-------+--------+--------+
| GalaxyDigital.Api.Client | 100%  | 100%   | 100%   |
+-------------------------------+-------+--------+--------+
| GalaxyDigital.Services   | 3.59% | 1.66%  | 26.98% |
+-------------------------------+-------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 3.59%  | 1.66%  | 26.98% |
+---------+--------+--------+--------+
| Average | 51.79% | 50.83% | 63.49% |
+---------+--------+--------+--------+

2024-01-31T13:26:01: Writing report file 'TestResults\html-coverage-report\index.html'
2024-01-31T13:26:01: Report generation took 0.2 seconds

Attachments:
  C:\TFS\GalaxyDigital.Services.Tests\TestResults\df90993a-c958-4fbe-8097-6ee1ae3c2959\test_2024-01-31.13_26_01.coverage
  C:\TFS\GalaxyDigital.Services.Tests\TestResults\df90993a-c958-4fbe-8097-6ee1ae3c2959\coverage.cobertura.xml

This properly excluded the namespace "GalaxyDigital.Api.Client" but required the extra command line switch that should have been coming from the runsettings and not require the command line switch.

Expected behavior

.runsetting file excludes should be obeyed when passed into the dotnet command.

Actual behavior

dotnet command requires the /p:Exclude parameter to actually exclude the assembly/namespaces.

Configuration (please complete the following information):
Please provide more information on your .NET configuration:
* Which coverlet package and version was used? coverlet.collector 6.0.0, coverlet.msbuild 6.0.0
* Which version of .NET is the code running on? .net core 6.0
* What OS and version, and what distro if applicable? Windows 10 x64
* What is the architecture (x64, x86, ARM, ARM64)? x64
* Do you know whether it is specific to that configuration? No

Additional context
Add any other context about the problem here.

❗ Please also read Known Issues

[daveMueller]

Ok there are a few thing off here. First of all coverlet.collector doesn't have a summary output on the console. This here is definitely not from our collector.

+-------------------------------+-------+--------+--------+
| Module                        | Line  | Branch | Method |
+-------------------------------+-------+--------+--------+
| GalaxyDigital.Api.Client | 100%  | 100%   | 100%   |
+-------------------------------+-------+--------+--------+
| GalaxyDigital.Services   | 3.59% | 1.66%  | 26.98% |
+-------------------------------+-------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 3.59%  | 1.66%  | 26.98% |
+---------+--------+--------+--------+
| Average | 51.79% | 50.83% | 63.49% |
+---------+--------+--------+--------+

I see that you are additionally referencing coverlet.msbuild. So it could be from here. This would also explain why the parameter /p:Exclude="[*]GalaxyDigital.Api.*" somehow has an effect. On the other hand you are calling dotnet test with a specified collector which doesn't really match to that theory.

What else is a bit weird is that you have specified two different collectors in your .runsettings file. Coverlet with the friendlyName="XPlat Code Coverage" and the internal one from vstest with the friendlyName="Code Coverage". In the result I then see two coverage files coverage.cobertura.xml and test_2024-01-31.13_26_01.coverage. This format .coverage is not supported by coverlet at all so this can't be created by us (link).

Not sure why but it doesn't seem that this report is from coverlet.

I just tried it out with a simple repro with the following coverlet.runsettings file:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat code coverage">
                <Configuration>
                    <Format>cobertura</Format>
                    <Exclude>[*]GalaxyDigital.Api.*</Exclude>
                    <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

and calling the collector like this:

dotnet test --collect:"XPlat Code Coverage" --settings coverlet.runsettings

and everything works as expected. Please also remove the coverlet.msbuild reference if you try this again. I hope this helps a bit to find the issue. If you could provide a simple repro I could analyse this more in detail.

[xantari]

So I think my issue is that the MSBuild settings in my csproj was the only way I was getting a coverage.cobertura.xml output to a known path (TestResults/coverage.coburtura.xml). Where as removing the msbuild package and the associated .csproj modifications would only output it to TestResults/{some random guid}/coverage.cobertura.xml

The extra output you were seeing in the output with the coverage was actually the reportgenerator nuget that consumes ccode coverage results (also an msbuild task).

Here is my .csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <Platforms>AnyCPU;x64</Platforms>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="coverlet.collector" Version="6.0.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="coverlet.msbuild" Version="6.0.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="IBM.EntityFrameworkCore" Version="6.0.0.300" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.InMemory" Version="6.0.26" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
    <PackageReference Include="Moq" Version="4.20.70" />
    <PackageReference Include="NUnit" Version="4.0.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
    <PackageReference Include="ReportGenerator" Version="5.2.0" />
  </ItemGroup>

  <PropertyGroup>
    <RunSettingsFilePath>$(MSBuildProjectDirectory)\CodeCoverage.runsettings</RunSettingsFilePath>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="..\GalaxyDigital.Services\GalaxyDigital.Services.csproj" />
  </ItemGroup>

  <ItemGroup>
    <Folder Include="TestResults\" />
  </ItemGroup>

  <!--https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.msbuild.tasks/coverlet.msbuild.props-->
  <PropertyGroup>
    <CollectCoverage>true</CollectCoverage>
    <CoverletOutput>TestResults\</CoverletOutput>
    <CoverletOutputFormat>cobertura</CoverletOutputFormat>
    <IncludeTestAssembly>false</IncludeTestAssembly>
    <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
    <Exclude>[*]GalaxyDigital.Console.*,[*]GalaxyDigital.Api.*</Exclude>
  </PropertyGroup>

  <Target Name="GenerateHtmlCoverageReport" AfterTargets="GenerateCoverageResultAfterTest">
    <ReportGenerator ReportFiles="@(CoverletReport)" TargetDirectory="TestResults\html-coverage-report" ReportTypes="HtmlInline" />
  </Target>

</Project>

When running this command:

dotnet test --settings CodeCoverage.runsettings --no-build --collect:"XPlat Code Coverage"

I will get a TestResults/coverage.cobertura.xml (a deterministic file output location) so that the reportgenerator can find it.

This file appears to be generated by coverlet.msbuild and using all the settings in the PropertyGroup listed above from my .csproj file. This is then passed to @(CoverletReport) for the report generator. I was able to get the namespaces excluded properly using the above .csproj settings as well.

However I am reading that I shouldn't really be using both coverlet.collector and coverlet.msbuild together.

So I tried to remove coverlet.msbuild, and removed all of these properties from the .csproj:

  <PropertyGroup>
    <CollectCoverage>true</CollectCoverage>
    <CoverletOutput>TestResults\</CoverletOutput>
    <CoverletOutputFormat>cobertura</CoverletOutputFormat>
    <IncludeTestAssembly>false</IncludeTestAssembly>
    <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
    <Exclude>[*]GalaxyDigital.Console.*,[*]GalaxyDigital.Api.*</Exclude>
  </PropertyGroup>

Now, I can no longer run the same command:

dotnet test --settings CodeCoverage.runsettings --no-build --collect:"XPlat Code Coverage"

It will error with:

dotnet : The test source file "C:\TFS\GalaxyDigital\GalaxyDigital.Services.Tests\bin\Debug\net6.0\GalaxyDigital.Services.Tests.dll"
provided was not found.

So now I modify it slightly, because it isn't picking up that this is an x64 build:

dotnet test ./bin/x64/Debug/net6.0/GalaxyDigital.Services.Tests.dll --no-build --collect:"XPlat Code Coverage" --settings CodeCoverage.runsettings

Then I get this:

Data collection : Unable to find a datacollector with friendly name 'XPlat Code Coverage'.
Data collection : Could not find data collector 'XPlat Code Coverage'
Data collection : Unable to find a datacollector with friendly name 'Code Coverage'.
Data collection : Could not find data collector 'Code Coverage'

I guess it is because it can't find the nuget path according to this: #1546

Each dev has their own nuget path, so now i'm stuck because I won't be able to share the project around with different devs due to their nuget location differences. So back to this command which does a build but puts it into /bin/Debug/net6.0 and disobeys that this is a 64-bit only application:

dotnet test --collect:"XPlat Code Coverage" --settings CodeCoverage.runsettings

This works now as it now output another compilation to the wrong bin folder (no x64 in the path).

But then the problem of the random guid paths crops up again such that the report generator cannot determine where to find the coverage.cobertura.xml file.

C:\TFS\GalaxyDigital\GalaxyDigital.Services.Tests\TestResults\28a9b03b-eaa8-4859-b0e9-6470a6aa8e2a\coverage.cobertura.xml

So, back to the beginning, the coverlet.msbuild route seems to be the only way to generate a deterministic file path for the coverage.cobertura.xml file.

[xantari]

I settled on removing all msbuild stuff from .csproj. I created a separate .bat file that users will have to run separately from a command prompt as I couldn't figure out how to get deterministic file names output (visual studio wants to continue to create random guids in the path). And coverlet not outputting itself if it is anything other then an AnyCPU build (x64 specific builds do not output the coverlet dlls for some reason and then give you the missing XPlat Code Coverage data collector errors).

Maybe I am missing something obvious to get the other methods to work...

REM "Removing previous coburtura.xml files so it doesn't get confused which one to process"
rd /s /q "TestResults\"
dotnet test --collect:"XPlat Code Coverage" --settings CodeCoverage.RunSettings
%userprofile%\.nuget\packages\reportgenerator\5.2.0\tools\net8.0\ReportGenerator.exe "-reports:TestResults\*\coverage.cobertura.xml" "-targetdir:TestResults\CoverageReport" --reporttypes:HtmlInline
start TestResults/CoverageReport/index.html

[daveMueller]

OK I think now it is a bit more clear to me what you want to do. Integrating coverlet and reportgenerator into Visual Studio. I admit I've never tried that out. Most users only calculate coverage at some stage in their pipeline.
But there is a Visual Studio extension that uses coverlet to do exactly this: https://github.com/the-dext/RunCoverletReport. Maybe you can get some information from there how to do this.
When I use coverlet locally, I'm also using a script pretty similar to what you have now.

[xantari]

Yeah, I basically wanted to continually see how I was doing in regards to branch and code coverage as I was writing unit tests. I sort of got it working iwth the coverlet msbuild stuff, and the reportgenerator msbuild stuff, but it wasn't clear to me that when you go that route the runsettings seem to be ignored and you need to put those settings in the .csproj file instead of a runsettings file.

Closing this. Thanks for your help!