Coverage report file duplicated when using --logger option

[tremblaysimon]

Please see the reported issue here coverlet-coverage/coverlet#733 about using --logger option in conjunction with --collect:"XPlat Code Coverage".

We think that it should be related more with vstest than coverlet.

Is it an expected behavior to get two generated code coverage file instead of one if we use vstest logger option?

[nohwnd]

@jakubch1 this is CC related, please have a look 🙂

[nohwnd]

Or maybe @MarcoRossignoli if this is coverlet related? Thx

[MarcoRossignoli]

I dont' think it's related to coverlet, collectors(wrote by @vagisha-nidhi) only send artifact through data sink https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/DataCollection/AttachmentManager.cs#L146

We don't handle nothing about loggers.

[DotNetSparky]

I'm experiencing the same thing (also using coverlet)... each xml file is duplicated.

[Sebbl22]

Same here.
tests.runsettings:

<?xml version="1.0" encoding="UTF-8"?>
<RunSettings>
    <RunConfiguration>
        <ResultsDirectory>.\TestResults</ResultsDirectory>
    </RunConfiguration>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat code coverage">
                <Configuration>
                    <Format>cobertura</Format>
                    <IncludeTestAssembly>false</IncludeTestAssembly>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
    <LoggerRunSettings>
        <Loggers>
            <Logger friendlyName="trx" />
        </Loggers>
    </LoggerRunSettings>
</RunSettings>

dotnet test --settings tests.runsettings

Result:

I have four projects and every coverage.cobertura.xml-file is duplicated.

[manishi2V]

Is there any solution found for this issue?

[danjcosg]

Still experiencing this

[gioce90]

I noticed the same if I use a .runsettings configuration file. But if I use the argument directly in an Azure Pipeline task:

- task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (cobertura format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj'
      arguments: '--no-restore --collect "Code Coverage" --logger trx --results-directory "TestResults/Coverage/" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
      publishTestResults: false

The problem just disappear

(This works also locally. The problem is the logger tag in the in the .runsettings file).
Note that: I don't use coverlet but the native MS coverage tool with --collect "Code Coverage"

[celluj34]

I'm having the same problem with --collect:"XPlat Code Coverage".

[bitbonk]

I have the same problem with a pretty basic command:

dotnet test C:\source\repos\MyProject\MyProject.sln --collect "XPlat Code Coverage" --logger trx --no-build --no-restore --results-directory C:\source\repos\MyProject\artifacts\tests\results

[bitbonk]

@nohwnd @MarcoRossignoli @jakubch1 What can we do about it?

[gioce90]

This issue is still open after more than one year....

[MarcoRossignoli]

The behavior is expected, trx logger currently creates a file and a directory like:

TestResult/userName_machineName_timespan<- folder
TestResult/trxfilename.trx

all attachments generated by the test session are copied inside the userName_machineName_timespan directory and used by the result summary inside Visual Studio when the trx file is open.

For this reason at the end of the test inside the subdirectories of TestResult we can find same "attachment" more than one time.

If you need to publish coverage report using PublishCodeCoverageResults@1 inside Azure DevOps you can try with

- task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'Cobertura'
      summaryFileLocation: .../TestResults/*/*.xml

If the issue affect also other scenarios one possible solution could be add a new parameter to the logger like --logger:"trx;CleanupAttachmentsArtifact=true" and remove the artifacts after the moving.

[SteveBush]

I spent the morning digging into the most recent changes to code coverage processing. Here's what I've found:

1. Two copies of a code coverage file are created in a TestResults folder at the same level as the project file. One copy is under a random GUID. The second copy is under a folder agent_host_timestamp/In/.
2. The codecov.io bash file fails to find and upload the coverage files under TestResults.

I would suggest that we rework the code coverage publishing to be like test files. We publish them into a staging folder that removes the duplicate files and can be picked up by codecov.io bash file.

Docs

https://docs.codecov.com/docs/about-the-codecov-bash-uploader

[MarcoRossignoli]

@SteveBush can you link here the documentation of codeconv.io related to the code coverage publishing workflow?

[SteveBush]

Updated with codecov.io reference and like to docs.

[MarcoRossignoli]

After the internal discussion we decided to not take any action for the moment as the behavior is as-designed and expected.

TL;DR;

1. Logger(like trx) is an extension point for what we don't have control, everyone can implement a logger extension that takes test result and move it in any place. Trx one is implemented in our code base and used(already) extensively in more than one place(AzDo @VSTest2 etc...).
2. Test platform allows extension like datacollector which can generated some artifacts(like code coverage report file) and again we don't have control on custom datacollector. It means that we cannot impose to generate a unique artifact name. For this reason for every test session we're generating a guid folder where we put the artifacts generated, simply to avoid to override same "possible" file for different test sessions(it's useful to be able to compare before/after and there's also a chance to run parallel tests with file locking issues).

For the two reason above we think that the issue is more related to "how the artifacts are searched and loaded" after the generation. Our advice is to use minimatch patterns to correctly include/exclude assets to elaborate/upload.

[SteveBush]

It might be easier if I explain my current build structure and the challenges I'm facing.

My build targets multiple frameworks. Builds artifacts are outputted to root level bin and obj directories in the following format:

bin/<project>/<buildconfiguration>/<targetframework><bin files>
obj/<project>/<buildconfiguration>/<targetframework><obj files>

I would like to use a similar format for TestResults. However, I haven't found a way to associate a test result with a targetframework or buildconfiguration MSBuild property. Also, duplicate versions of the same test output file are being produced in a random GUID folder.

Ideally, I could modify the TestResults output folder to include the project name, build configuration, and target framework information and output results to the solution root.

Current Challenges:

1. Digging through random GUID folders and deduping test result files is a pain.
2. Switching from Debug to Release builds just produces more TestResults under random GUIDs. It's hard to figure out which test is associated with which build or configuration.
3. To remove TestResult folders from the source tree, I have to manually delete them from each project or use a batch script. Build clean by default doesn't clean these up. I often switch between target frameworks and build configurations to developer and test code on different platforms and build configurations.
4. A multiple target framework build produces two sets of GUID folders for each target framework.

Ideally, I could find a way to output TestResults to the solution root in the following format:

soluton/TestResults/<project>/<buildconfiguration>/<targetframework>/<TestResults with no duplicate files>

To accomplish this, I would need to be able to TestResult output location based on MS build properties:
<MSBuildProjectName>, <TargetFramework>, and <Configuration>.

I know I can set the test results directory on the dotnet test command line or in a test.runsettings file. However, I haven't found a MSBuild test output property that I can construct using MSbuild properties:
<MSBuildProjectName>, <TargetFramework>, and <Configuration>.

Thanks for listening. Below is my current build structure and the contents of my root level Directory.Build.props file.

solution/
   Directory.Build.props (see contents below)
   bin/
        macosunittestproj
             <buildconfiguration>
                   <targetframework>
                         <bin files>
        macosintegrationtestproj
             <buildconfiguration>
                   <targetframework>
                         <bin files>
   obj/
        macosunittestproj
             <buildconfiguration>
                   <targetframework>
                         <obj files>
        macosintegrationtestproj
             <buildconfiguration>
                   <targetframework>
                         <obj files>
   src/
   test/
       macos/
             macosunittestproj/
                 TestResults
                    <Guid1>
                        <In>
                                <HostName>
                                      stevebu_BuMacBookM1_2022-06-13.10_59_04.cobertura
                    <Guid2>
                                      stevebu_BuMacBookM1_2022-06-13.11_01_05.cobertura
                    <Guid3>
                                      stevebu_BuMacBookM1_2022-06-13.10_59_04.cobertura
                    <Guid4>
                        <In>
                                <HostName>
                                      stevebu_BuMacBookM1_2022-06-13.11_01_05.cobertura
             macosintegrationtestproj/
        shared/
        windows/
             windowsunittestproj/
             windowsintegrationtestproj/    

Directory.Build.props
    <RepoRootPath>$(MSBuildThisFileDirectory)</RepoRootPath>
    <BaseIntermediateOutputPath>$(RepoRootPath)obj\$([MSBuild]::MakeRelative($(RepoRootPath), $(MSBuildProjectDirectory)))\</BaseIntermediateOutputPath>
    <BaseOutputPath Condition=" '$(BaseOutputPath)' == '' ">$(RepoRootPath)bin\$(MSBuildProjectName)\</BaseOutputPath>
    <PackageOutputPath>$(RepoRootPath)bin\Packages\$(Configuration)\</PackageOutputPath>

[MarcoRossignoli]

Thanks @SteveBush for the info.

I'll label this one as by-design and I'll close for now.

Looks like your problem is not strictly related to the "duplication issue" but more on the possibility to generate deterministic assets name that include os/tfm/architecture in it, where a possible solution could be provide the template name feature.

We have already some related issues
#1876
#2378

I'll add your description there.

cc: @nohwnd

[siganberg]

My simple solution is to perform the delete step after the dotnet build.

rm -rf test_results

dotnet build -c Release

dotnet test \
  --filter exclude!="true" \
  --no-build  \
  --logger:"trx;LogFileName=test-results.trx" \
  -c Release  \
  /p:ExcludeByAttribute=\"Obsolete,ExcludeFromCodeCoverage\"  \
  /p:MergeWith=\"../../test_results/coverage.json\"  \
  --results-directory ./test_results \
  --collect:"XPlat Code Coverage"  \
  --verbosity normal 
  
# delete duplicate coverage
rm -rf test_results/**$(date +'%Y')**