[Question] Up-to-date overview: How to use ReportGenerator with Azure DevOps?

[georg-jung]

Thanks for creating ReportGenerator! I think it provides a great overview of code coverage and I really like all the integration options.

I got however a bit confused about all the options to use it and the best way to use ReportGenerator with Azure DevOps as of now, 2024. I've seen many issues discussing this topic including #398, but many of them are outdated as of 2024. Maybe it could be helpful for new users to have one central up-to-date page with guidance (which the other places including the Azure DevOps Marketplace page could link to). I'd be happy to contribute in that direction if that makes sense, but I guess at this point the best thing I can do is asking the questions I thought about when getting started with Code Coverage in Azure DevOps.

My Goals (decreasing prio)

• having some working code coverage report at all
• having the "Code Coverage" tab with ReportGenerator's report, not the default one
• having a downloadable full-featured Code Coverage Report artifact
• short task runtime
• a quite general Azure Pipelines task definition that is easy to adapt to/copy around between multiple repos
• possibility to configure ReportGenerator, e.g. pass a license key
• having the "X.Y% covered" info on the "Summary" tab

Options I came across

• PublishCodeCoverageResults@1
• PublishCodeCoverageResults@2
• reportgenerator@5
• A combination of the above

What I think I understood

• Many shortcomings of PublishCodeCoverageResults are about V1. V2 made things better but there are even cases when one would rather use v1 than v2 (I'm not sure which exactly though).
• PublishCodeCoverageResults uses ReportGenerator under the hood. I think I didn't read it anywhere in docs/readme/website, but from reading the sources I understood that I could use a .netconfig file to configure ReportGenerator while using PublishCodeCoverageResults@2 (maybe this should be documented somewhere, because I think it's a great feature). Mentioning how to use a ReportGenerator license with PublishCodeCoverageResults@2 on https://reportgenerator.io/pro would probably be easy and could even make some people buy a license (I tried this and it works).

Questions

• Whats the minimal, optimal but full example to use ReportGenerator with Azure DevOps today? I've seen https://reportgenerator.io/usage but in my testing using only the reportgenerator task didn't produce e.g. the Code Coverage tab.

• Report type HtmlSummary sounds like it could help with PublishCodeCoverageResults is slow with many files in the generated report microsoft/azure-pipelines-tasks#4945, if it's acceptable to e.g. have just the summary in the "Code Coverage" tab but the full report in the zip artifact. Is there an easy way to set this up?

• disable.coverage.autogenerate: 'true' doesn't seem to work for me with PublishCodeCoverageResults@2. Is there a workaround for V2 too? Is it better to use V1 with reportgenerator@5?

• I have a DotNetCoreCLI@2 test task like the following. The PublishCodeCoverageResults@2 seems to first publish ReportGenerator's report, which is later overwritten. How to fix this while still publishing the trx test results?

- task: DotNetCoreCLI@2
  inputs:
    command: test
    projects: '$(solution)'
    arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
    testRunTitle: 'release'
    publishTestResults: true
  displayName: dotnet test Release

• My current production pipeline config is similar to the following. It does fulfil most of my requirements except that the contents of "Code Coverage" are replaced by an autogenerated report when the pipleine finishes. How to get that right?

variables:
  solution: './Xyz.sln'
  disable.coverage.autogenerate: 'true' # https://github.com/microsoft/azure-pipelines-tasks/issues/4945#issuecomment-823832527

jobs:
  - job: Build
    pool:
      vmImage: ubuntu-22.04
    steps:
    - script: dotnet restore "$(solution)" /p:ContinuousIntegrationBuild=true
      displayName: dotnet restore

    - script: dotnet build --no-restore -c Release "$(solution)"  /p:ContinuousIntegrationBuild=true
      displayName: dotnet build

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Debug -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'debug'
        publishTestResults: true
      displayName: dotnet test Debug

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'release'
        publishTestResults: true
      displayName: dotnet test Release

    - task: PublishCodeCoverageResults@2
      displayName: 'Publish code coverage report'
      inputs:
        summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        failIfCoverageEmpty: true

I hope I didn't miss any obvious places to research this myself. Thanks again for creating ReportGenerator!

[danielpalme]

Thanks for your detailed issue and the research behind it.

You are right, there are several options.
My experience is, that PublishCodeCoverageResults@1 or PublishCodeCoverageResults@2 are required to get the "Code Coverage" tab

PublishCodeCoverageResults@1 uses an (outdated) version of ReportGenerator internally.
If you don't specify disable.coverage.autogenerate: 'true' it will (re-)generate the coverage report based on the Coberatura file.
If you want to use the latest version of ReportGenerator you should always set disable.coverage.autogenerate: 'true'.

PublishCodeCoverageResults@2 does not use ReportGenerator internally. If you have different information, I would be interested.
Instead it generates it's own report which looks like this:

As far as I know it's not possible to publish a full report with PublishCodeCoverageResults@2, since there is no Report Directory parameter.

Regarding your questions:

• Whats the minimal, optimal but full example to use ReportGenerator with Azure DevOps today? I've seen https://reportgenerator.io/usage but in my testing using only the reportgenerator task didn't produce e.g. the Code Coverage tab.

Correct. You'll have to combine it with PublishCodeCoverageResults@1.
See also: https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension

• Report type HtmlSummary sounds like it could help with PublishCodeCoverageResults is slow with many files in the generated report microsoft/azure-pipelines-tasks#4945, if it's acceptable to e.g. have just the summary in the "Code Coverage" tab but the full report in the zip artifact. Is there an easy way to set this up?

You could generate two reports in separate directories. Then you would probably have to rename the summary.html file to index.html before publishing the it with PublishCodeCoverageResults@1

- task: reportgenerator@5
  displayName: ReportGenerator
  inputs:
    reports: 'src\target\reports\coverage\coverage.xml'
    targetdir: '$(Build.SourcesDirectory)/coveragereport'
    reporttypes: 'HtmlInline_AzurePipelines;HtmlSummary;Cobertura'
    customSettings: 'settings:createSubdirectoryForAllReportTypes=true'

# Now rename file `$(Build.SourcesDirectory)/coveragereport/HtmlSummary/summary.html` to `$(Build.SourcesDirectory)/coveragereport/HtmlSummary/index.html`

# Create artifact of directory $(Build.SourcesDirectory)/coveragereport/HtmlInline_AzurePipelines

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage results'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Build.SourcesDirectory)/coveragereport/Cobertura/Cobertura.xml'
    reportDirectory: '$(Build.SourcesDirectory)/coveragereport/HtmlSummary'
  env:
    DISABLE_COVERAGE_AUTOGENERATE: 'true'

• disable.coverage.autogenerate: 'true' doesn't seem to work for me with PublishCodeCoverageResults@2. Is there a workaround for V2 too? Is it better to use V1 with reportgenerator@5?

As mentioned above, I'm pretty sure that you have to use PublishCodeCoverageResults@1.
disable.coverage.autogenerate: 'true' does not work here. And there is no option to specify a report directory any more. Am I missing something?!

• I have a DotNetCoreCLI@2 test task like the following. The PublishCodeCoverageResults@2 seems to first publish ReportGenerator's report, which is later overwritten. How to fix this while still publishing the trx test results?

I guess that's not possible. See also your next question.

• My current production pipeline config is similar to the following. It does fulfil most of my requirements except that the contents of "Code Coverage" are replaced by an autogenerated report when the pipleine finishes. How to get that right?

This is also my experience. See my answers above. I always use PublishCodeCoverageResults@1 instead of PublishCodeCoverageResults@2.

[georg-jung]

Thanks for your detailed response!

PublishCodeCoverageResults@2 does not use ReportGenerator internally. If you have different information, I would be interested.

My example pipeline above with

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'release'
        publishTestResults: true
      displayName: dotnet test Release

    - task: PublishCodeCoverageResults@2
      displayName: 'Publish code coverage report'
      inputs:
        summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        failIfCoverageEmpty: true

- and notably without any reportgenerator@5 task - leads to the following behaviour:

• As soon as the PublishCodeCoverageResults@2 task finished (but not the whole job/pipeline), the Code Coverage tab contains a ReportGenerator report. When specifying a license key using .netconfig it even contains a pro version report.
• At the same time, a "Code Coverage Report_<id>" artifact is created.
• As soon as the pipeline(/job?) finishes, the Code Coverage tab's content is replaced with the report format that can be seen in your screenshot.
• The tasks log also contain lines like "Initializing report builders for report types: HtmlInline_AzurePipelines" that sound quite ReportGenerator related.
• In the end, I can view the PublishCodeCoverageResults@2-style report in the Code Coverage tab or download the ReportGenerator report from artifacts.

=> Thus, a V2 disable.coverage.autogenerate: 'true' might be quite interesting.

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/dotnet-core-cli-v2?view=azure-pipelines reads

#publishTestResults: true # boolean. Optional. Use when command = test. Publish test results and code coverage. Default: true.

Not sure how this might affect the behaviour.

I changed my pipeline now like this and it works as exected. The Code Coverage tab does contain the ReportGenerator report even after the end of the job (and also still accounts for .netconfig).

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'release'
        publishTestResults: true
      displayName: dotnet test Release

    # https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension
    - task: reportgenerator@5
      displayName: '[Coverage] ReportGenerator'
      inputs:
        reports: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        targetdir: '$(Agent.TempDirectory)/coveragereport'
        reporttypes: 'HtmlInline_AzurePipelines;Cobertura;Badges'
        assemblyfilters: '-xunit*'

    - task: PublishCodeCoverageResults@1
      displayName: '[Coverage] Publish results'
      inputs:
        codeCoverageTool: Cobertura
        summaryFileLocation: '$(Agent.TempDirectory)/coveragereport/Cobertura.xml'
        reportDirectory: '$(Agent.TempDirectory)/coveragereport'
      env:
        DISABLE_COVERAGE_AUTOGENERATE: 'true'

Thanks for pointing me to https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension. I actually overlooked this first (even though it's linked in the README 🙄 - I guess there are just many places providing pieces of information, some of which are more up to date while others aren't).

[danielpalme]

=> Thus, a V2 disable.coverage.autogenerate: 'true' might be quite interesting.

There is no such option. I looked at the source code: https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishCodeCoverageResultsV2

Internally PublishCodeCoverageResults@1 uses https://github.com/microsoft/azure-pipelines-coveragepublisher to create the coverage reports.

[rikrak]

FWIW I would have benefitted from such a guide. It took me some time to understand that the PublishCodeCoverageResults@2 task was flawed, and the older version is preferred in my scenario (which was similar to the one described above). The main difficulty I faced in researching the production of a Devops Pipeline code coverage report, was that I didn't realise that a lot of the examples out in the wild (using the V2 task) are quite simplistic and don't cover what I'd consider a "real-world" scenario. In addition I thought any examples using the older V1 task were out-of-date :-/

[danielpalme]

@rikrak
Yes there are several options.

But the overall process is documented here:
https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-vsts

The FAQs also answer the most common question:
https://github.com/danielpalme/ReportGenerator/wiki/FAQ#azure-devops-extension-seems-to-ignore-my-settings

[jchenathub24]

• As soon as the PublishCodeCoverageResults@2 task finished (but not the whole job/pipeline), the Code Coverage tab contains a ReportGenerator report. When specifying a license key using .netconfig it even contains a pro version report.
• At the same time, a "Code Coverage Report_" artifact is created.
• As soon as the pipeline(/job?) finishes, the Code Coverage tab's content is replaced with the report format that can be seen in your screenshot.

@georg-jung I come cross the same behaviour.

When I test steps like:

1. dotnet test: generate coverage file
2. using PublishCodeCoverageResults@2

I can see that PublishCodeCoverageResults@2 internally use ReportGenerator, it shows a footer in the report saying that. But later, the whole report is override when pipeline finished. Feel like a bug for that task.

Also, I try the disable.coverage.autogenerate: 'true', it does not change any outcome from PublishCodeCoverageResults@2.

So the options I have tested and working for me:
Option1:

1. dotnet test: generate coverage file.
2. reportgenerator: generate report file from coverage file in step 1. (ReportGenerator 5.2.1.0 by the time of this reply)
3. disable re-generate report: by setting the env variable.
4. PublishCodeCoverageResults@1: publish the report generated in step 2.

Option2:

1. dotnet test: generate coverage file.
2. PublishCodeCoverageResults@1: this task will generate report base on coverage xml file from step 1 and publish to ADO. But the report is generated by ReportGenerator 5.1.14.0

Option3:

1. dotnet test: generate coverage file.
2. PublishCodeCoverageResults@2: this can generate report and publish some report to code coverage tab, but using ADO layout.

Opeion 2 and 1 have no much different. But Option1 give your more customization option in ReportGenerator task. To me, Option 2 is easier to setup then option 1 and do the job, if you don't need too much customization on ReportGenerator task and dont mind to use ReportGenerator 5.1.14.0.

One thing is missing and I believe is useful is that the 'collapose all' and 'expand all' link. If I generate the report locally on my laptop, I can use them to quickly collapose the list. But they are not in ADO's coverage tab, meaning that I have to use scrol bar.

[A-Rai-col]

Thanks @jchenathub24 this clears my doubts about PublishCodeCoverageResults@2 ,I faced the same issue when using V2 task as it seems to use reportgenerator internally; First it generates the HtmlInline_AzurePipelines report and publishes that to the Code Coverage tab then after a couple of seconds it repopulates the tab with the default ADO layout.

I'm having to resort to using PublishCodeCoverageResults@1 as mentioned in your Option1 cause the ReportGenerator report is just so much better. Also wish there was collapse all & filter options in the report.

[danielpalme]

Also wish there was collapse all & filter options in the report.

That's because Azure DevOps blocks all JavaScript.
If you download the full report artifact, those features will become available.

[cremor]

@danielpalme I think this issue should be reopened because the ReportGenerator documentation regarding Azure DevOps tasks should be updated. A few weeks ago Azure DevOps pipeline builds that use V1 of the PublishCodeCoverageResults tasks started showing deprecation warnings. See also the announcement here: https://devblogs.microsoft.com/devops/new-pccr-task/

Important part in that announcement:

Q: I’m using Azure DevOps Server version 2022. What will happen to me?
A: In a future version of the Azure DevOps Server, we will also remove the V1 version of the publish code coverage results task. There will be a transitioning period where the V1 version will still run. During the transition period you’ll need to move from the V1 version to the V2 version, as when the transition period ends the V1 version will be removed and any pipeline containing that task will fail.

So there should be a documentation that shows how to correctly use ReportGenerator with PublishCodeCoverageResults V2.

I think if you don't care how the HTML report looks and if you don't need to set any additional ReportGenerator settings, then you can simply use PublishCodeCoverageResults@2 now, without using reportgenerator@5 at all, because ReportGenerator will still be used to read/convert and merge the coverage reports.

[danielpalme]

@cremor
I updated the documentation here:
https://github.com/danielpalme/ReportGenerator/wiki/Integration#attention

I think if you don't care how the HTML report looks and if you don't need to set any additional ReportGenerator settings, then you can simply use PublishCodeCoverageResults@2 now, without using reportgenerator@5 at all, because ReportGenerator will still be used to read/convert and merge the coverage reports.

The new version has several disadvantages regarding the report in the Code Coverage tab within Azure DevOps

• No branch and method coverage
• No details page for each class