Add Code Coverage targets

[ryanbrandenburg]

It would be nice to add targets/properties to allow code coverage report generation out of the box in Arcade. Bonus points if it produces a format that AZDO pipelines can ingest by default.

After fiddling with things a bit it seems that the dotnet exec $(TestRunnerPath) method of executing tests used by the Xunit target prevents using packages like coverlet.msbuild, so an implementation in Arcade or documentation of a simple way to DIY would be ideal.

[sharwell]

I'm hoping this comes out of recent work to bring up Coverlet on dotnet/machinelearning and dotnet/coreclr. In particular, dotnet/machinelearning has figured out a nice combination of Coverlet+ReportGenerator to produce files consumable by codecov.io.

1. Individual test projects use Coverlet for instrumentation
2. Reports produced by multiple test projects are aggregated by ReportGenerator to a single Cobertura-format report
3. The single Cobertura report is uploaded to codecov.io

[markwilkie]

Any updates @sharwell ?

[sharwell]

Not from me, but @ViktorHofer may have updates

[ViktorHofer]

Relates to #2076. After enabling full code coverage in corefx (with codecov) I can help out here.

[markwilkie]

Thanks @ViktorHofer

[missymessa]

Hi @ViktorHofer, I see that this issue is a bit old, so checking into see if any requirements have changed on this? Thanks!

[jonfortescue]

@pranavkm seems to have a draft PR for doing something like this over at #3919.

[ViktorHofer]

VSTest now supports all the feature that we need. I recommend to use dotnet test's coverlet integration instead of invoking coverlet directly. The necessary work here would be:

1. Add the option to run on VSTest instead of xunit.console.
2. When using VSTest, add the option to collect code coverage: dotnet test --collect:"XPlat Code Coverage" (https://github.com/tonerdo/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application)

Optional:

1. Add a step to aggregate all reports together with ReportGenerator
2. Upload the report to AzDO and/or codecov.io.

[ViktorHofer]

I'm currently not actively working on these things...

[missymessa]

@ViktorHofer Thanks for the update! I've implemented something similar to your suggestion in our services' CI for code coverage, too, so we'll see how much of that we can leverage for this addition to Arcade.

[Evangelink]

Happy to help providing something standard using Microsoft Code Coverage if we are interested. I have tried to create a PR on roslyn-analyzers and it seems that the changes required for coverage only is really small.

Adding

<TestRunnerAdditionalArguments>--collect:"Code Coverage;Format=Cobertura"</TestRunnerAdditionalArguments>
    <!-- TODO: On next arcade bump of Microsoft.CodeCoverage/Microsoft.NET.Test.Sdk we will have to update this line to be netstandard2.0 -->
    <!-- Because arcade is passing dlls directly to 'dotnet test' command, we need to manually provide path to coverage datacollector -->
    <TestRunnerAdditionalArguments>$(TestRunnerAdditionalArguments) --test-adapter-path $(NUGET_PACKAGES)/microsoft.codecoverage/$(MicrosoftNETTestSdkVersion)/build/netstandard1.0</TestRunnerAdditionalArguments>

in some props/target file seems to be enough. Obviously, it would be better to also provide a .runsettings.