Add support for dotnet test --collect:"Code Coverage"

[harshjain2]

Description

Add support for code coverage through dotnet test on Windows OS machine.
dotnet test --collect:"Code Coverage"

[arpit-nagar]

Do you have any plans for Linux ??

(edit from @codito: there are atleast 22 more votes in #579 (comment) for code coverage support in linux. Using this comment to keep track of linux support request)

[nulltoken]

Why the "having Visual Studio 2017 Enterprise installed" limitation? Is there any plan to lift it?

I believe collecting test code coverage during a CI build (with only the .NET Core SDK being deployed) would be a very valid scenario.

[ctro]

👍 I'd very much love to have working code coverage within the confines of .NET Core.

[harshjain2]

@arpit-nagar test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise). Mac/linux support isn't in our immediate backlog. /cc @pvlakshm @sbaid to consider the feedback for prioritization.

[arpit-nagar]

@harshjain2 This is a very basic requirement. And as dotnet core is multi platform this feature should also need to be available on Linux.
Is there any way we can achieve code coverage on Linux for dotnet core project.

[andyjansson]

test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise).

@harshjain2 What's the rationale behind requiring VS enterprise for code coverage?

[rol-dave-overall]

Downloaded VS2017 15.3 and .NET Core 2.0 and after entering (dotnet test --collect:"Code Coverage") I get an error:

The test source file "C:\Application2\tests\Console.UnitTests1\Coverage" provided was not found.

Tried changing this to be (dotnet test --collect:"CodeCoverage") i.e. without the space and then get this error:

dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program'
Object reference not set to an instance of an object.

[jstallm]

It sounds to me like the status is that we cannot do code coverage for asp.net core 1.X using dotnet test at the command line. Is this correct?

[edumserrano]

I get the same as @rol-dave-overall. This is a show-stopper for us trying to get our solution build on Visual Studio Team Services. Disabling code coverage is not really an option.

Any available workarounds?

[RoCore]

actually have the same problem. the vso test with --collect Coverage or "Code Coverage" not working. the runsettings tried with both names

[FastNinja]

code coverage - this is the most basic requirement. what serious organisation will commit to using dot net core in non Windows environment if such a basic functionality is not there? how business can make a decision to release something in production where test coverage is not known?
It can be as fast and awesome - but without this basic support it pretty much unusable outside MS/Windows worlds.
The Linux community would like to embrace dot net core on board - but look it just simply can't.

[ollyjshaw]

Yes, this feature should be available for all target platforms. It should not require an installation of Visual Studio.

[DavidParks8]

Code coverage used to be a reason to upgrade to vs enterprise, but the rise of free alternatives such as OpenCover has removed the value proposition. Thus, code coverage should be added to the dotnet cli on all platforms to keep developers from using the alternative products available.

[tracyhoward]

@harshjain2 Is there a timeframe for when this will be available?
Add support for dotnet test --collect:"Code Coverage" #981

[pvlakshm]

Supporting CC with the dotnet test command line is presently not scheduled in our pans leading up to Nov 2017.
However, the following will be possible with .NET Core tests:

1. you can vstest.console.exe command line runner and generate CC information (the CC data collectors are supported on .NET45).
2. If you are working from within VS on tests targeting .NET Core, then you will need to add a reference to the Microsoft.CodeCoverage NuGet package to your test project, and update both the test and app-under-test projects to generate ‘Full’ debug info. Please see "Working with Code Coverage" here: https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md (note that this is only a temporary workaround).

We are eager to get to this once we finish addressing a few more scenarios that are already underway.

[arpit-nagar]

@pvlakshm regarding point 2.
Is it possible to get CC on windows build environment which have Test agent, and don't have VS?

[pvlakshm]

@arpit-nagar, we are working towards that.

[henryJack]

Code coverage is an essential part of CI. By all means, have a premium version available in a nice helpful way in VS but there should be, for example, a dotnet test --with-coverage=True --fail-under=95 type command for CI.

[ISemyanchuk]

Hi, I was able to run vstest.console.exe with code coverage for .net core 2.0 project locally, but on build agent, with the same setup( i think it is same :)) I've receiving next error :
"Failed to initialize client proxy: could not connect to test process."

[ElvenSpellmaker]

We are working on #2874 . When this is implemented you can get same results as with coverlet. We will be supporting cobertura format.

Wooo I look forward to that!

[fhnaseer]

Cobertura support is added to code coverage. It is available in the following version. https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.1.0-preview-20211118-03

To get cobertura report, add <Format>Cobertura</Format> in the runsettings.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
        <Configuration>
           <Format>Cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Available format options:

Coverage (default binary file format)
Cobertura
Xml (Same as codecoverage.exe /analyze report.coverage output).

[jakubch1]

In package https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.2.0-preview-20211214-01&view=overview
we have added experimental support for macOS x64. We also removed libunwind and libuuid dependencies.

Currently we support:

OS	Version	Architectures	Dependencies	Lifecycle
Alpine Linux	3.11+	x64	apk add libxml2	Alpine
CentOS	8+	x64		CentOS
Debian	10+	x64	apt-get install libxml2	Debian
Fedora	32+	x64		Fedora
Linux Mint	19.3+	x64		Linux Mint
openSUSE	tumbleweed	x64		OpenSUSE
Ubuntu	18.04+	x64	apt-get install libxml2	Ubuntu
macOS	10.15+	x64

[gioce90]

Thanks to the preview version here indicated (now I use the 17.1.0-preview-20211130-02 version) I'm now able to generate code coverage in two different way for obtains Full coverage and the Diff coverage (for Pull Requests) on Azure DevOps.
I achieved this using the new Microsoft native "Code coverage" and using two different pipelines and I want to show you how.

The problem:

If I understood well, in Azure DevOps if you want the "Code coverage" tab on the build page and you want to visualize the report directly in a friendly way (and not to download it), your pipeline have to generate files in supported formats, like cobertura.xml, as poorly described here.
But in Azure DevOps if you also want a Diff coverage policy on the Pull Request, you have to generate a build with the native .coverage file (that presented in this thread). But in this way you obtain the Diff policy but you have to say goodbye to the friendly full report on the build (because the .coverage is only downloadable and is not viewable).
Moreover it seems that if you generate together these different format files, only the .coverage file is taken in consideration by Azure DevOps "Code coverage" tab (microsoft/azure-pipelines-tasks#13632 (comment))

Solution:

My solution involves in creating two different pipelines.

The first pipeline is for the Full Coverage.
It runs for every commit on the repository. The coverage is generated with the native MS coverage but files are in the cobertura format and is easy to generate and publish a report.
The yml is this:

trigger:
  batch: true
  branches:
    include:
    - feature/*

pool: 
  vmImage: 'windows-latest'

variables:
  buildConfiguration: 'Release'
  disable.coverage.autogenerate: 'true' # for report generator

steps:

# restore step
  - task: DotNetCoreCLI@2
    displayName: 'dotnet restore'
    inputs:
      command: 'restore'
      projects: '**/*.csproj'
      configuration: $(BuildConfiguration)
      feedsToUse: 'select'
      vstsFeed: '...ourFeedId...'

# test step
  # task: test with native code coverage (cobertura format) with publish coverage excluded:
  # Usefull for full coverage, but useless for PR diff coverage.
  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (cobertura format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj'
      arguments: '--no-restore --collect "Code Coverage" --logger trx --results-directory "TestResults/Coverage/" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
      publishTestResults: false

# report generator, publish tests and code coverage step
  - task: reportgenerator@5
    inputs:
      reports: '$(Build.SourcesDirectory)/TestResults/Coverage/**/**.cobertura.xml'
      targetdir: '$(Build.SourcesDirectory)/TestResults/Coverage/Reports'
      reporttypes: 'HtmlInline_AzurePipelines_Dark;Cobertura'
      assemblyfilters: '+My.Company.**;-My.Company.**.Tests'

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: 'VSTest'
      testResultsFiles: '**/*.trx'
      mergeTestResults: true
      failTaskOnFailedTests: false

  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'cobertura'
      summaryFileLocation: $(Build.SourcesDirectory)/TestResults/Coverage/Reports/Cobertura.xml
      reportDirectory: $(Build.SourcesDirectory)/TestResults/Coverage/Reports
      failIfCoverageEmpty: false

This works really well. Please note that:

1. I can easily set Format=cobertura in the DotNetCoreCLI@2 (test) task without create a .runsettings file.
2. The task reportgenerator@5 have the assemblyfilters field with that input because in this way ONLY projects of my interest are included (all my solution's assemblies and don't other stuff like Moq, Humanizer and other added nuget stuff...) and without my Tests projects. I think that can be interests of many of you. This also fix the problem Code Coverage do not match with Azure DevOps summary azure-pipelines-tasks#13703 .
3. I also talked here: Not able to render Code Coverage results in the "Code Coverage" tab in Azure DevOps azure-pipelines-tasks#13632 (comment)

The second pipeline is for the Diff Coverage on the Pull Request.
Obviously I first set everything up as the documentation says.
The yml is the follow:

trigger:
  batch: true
  branches:
    include:
    - develop

pool:
  vmImage: 'windows-latest'

variables:
  buildConfiguration: 'Release'
  disable.coverage.autogenerate: 'true' # for report generator

steps:

# restore step
  - task: DotNetCoreCLI@2
    displayName: 'dotnet restore'
    inputs:
      command: 'restore'
      projects: '**/*.csproj'
      configuration: $(BuildConfiguration)
      feedsToUse: 'select'
      vstsFeed: '...ourFeedId...'

# test step
  # task: test with default code coverage (.coverage format) and publish coverage included:
  # Usefull for PR diff coverage, but not for full coverage.
  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (.coverage format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj'
      arguments: '--collect "Code Coverage"'
      publishTestResults: true

Now, this works but is not good enough. Is lovely to have PR Diff coverage policy (that part works well) but I find awful to have two pipeline instead of one. There have too much in common. And is strange that the native Microsoft .coverage format is not supported by Azure DevOps for report visualization on the build.
Another weird thing: in the second pipeline I still haven't found a way to properly filter the assemblies on which to run code coverage. I see that files in Test projects are also considered in my Pull Requests and, even worse, if I download the .coverage file generated I also see things like xUnit.dll, moq.dll, etc. (I also talked here)

Apart from these problems to which I hope to find a solution soon, I believe I have successfully obtained the two different code coverage. Tell me what you think or if you think there are improvements to be made.

2022/02/19 - EDIT :

1. Now I use the new stable version (17.1.0) of the Microsoft.NET.Test.Sdk nuget package.
2. I noticed that the old version of ReportGenerator get some problem with the native dotnet-coverage. With the new version (reportgenerator@5) is fine.
3. I noticed that with the use of a .runsettings file is possible to filter some assemblies for both diff and full code coverage. So now the field assemblyfilters: '+My.Company.**;-My.Company.**.Tests' on the reportgenerator@5 task is no needs anymore.

[jakubch1]

I've created two new issues to support code coverage on Linux ARM and macOS ARM64 #3426 #3427.

I'm closing this issue as other cases are already covered. Please use thumbs-up on above issues so we can prioritize it.

[ManikandanMani]

The second pipeline is for the Diff Coverage on the Pull Request. The yml is the follow:

trigger:
  batch: true
  branches:
    include:
    - develop

pool:
  vmImage: 'windows-latest'

variables:
  buildConfiguration: 'Release'
  disable.coverage.autogenerate: 'true' # for report generator

steps:

# restore step
  - task: DotNetCoreCLI@2
    displayName: 'dotnet restore'
    inputs:
      command: 'restore'
      projects: '**/*.csproj'
      configuration: $(BuildConfiguration)
      feedsToUse: 'select'
      vstsFeed: '...ourFeedId...'

# test step
  # task: test with default code coverage (.coverage format) and publish coverage included:
  # Usefull for PR diff coverage, but not for full coverage.
  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (.coverage format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj'
      arguments: '--collect "Code Coverage"'
      publishTestResults: true

Hi @gioce90, I have a slight clarification does the above pull request builds run all [old+newly added] tests or only the newly added tests would be ran when a PR is raised?

[gioce90]

Hi @ManikandanMani , this builds and run all them (old and new tests).

[simonachmueller]

Hey, @nehsharmMS mentioned in her comment that Microsoft has a new preview version of the Azure DevOps for code coverage, as I understood now it's possible to display Code Coverage tab and info when just publishing .coverage file. Is that true?

[gioce90]

Hey, @nehsharmMS mentioned in her comment that Microsoft has a new preview version of the Azure DevOps for code coverage, as I understood now it's possible to display Code Coverage tab and info when just publishing .coverage file. Is that true?

I don't know @simonachmueller , I'm not so sure about that....

[sjd2021]

I could be missing something, but would anyone be able to explain why this works in linux for one project and not another project where the only difference is MSTest vs XUnit? The error indicates that it'll only work on Windows.

[jakubch1]

For version >= 17.5.0-preview-20221021-01 dotnet test --collect "Code Coverage" is fully cross plat.

https://dev.azure.com/dnceng/public/_artifacts/feed/test-tools/NuGet/Microsoft.CodeCoverage/overview/17.5.0-preview-20221021-01

[jakubch1]

I could be missing something, but would anyone be able to explain why this works in linux for one project and not another project where the only difference is MSTest vs XUnit? The error indicates that it'll only work on Windows.

@sjd2021 Probably 1 of your projects is referencing old version of Microsoft.CodeCoverage package. Please try upgrading both to latest.

[gioce90]

Hey, @nehsharmMS mentioned in her comment that Microsoft has a new preview version of the Azure DevOps for code coverage, as I understood now it's possible to display Code Coverage tab and info when just publishing .coverage file. Is that true?

Hi @simonachmueller , now it seems possible (how I already pointed out here)