`dotnet-coverage` does not exclude test assemblies while `dotnet run` / `dotnet test` do with Microsoft.Testing.Platform

[MattKotsenas]

I'm seeing different behaviors for the IncludeTestAssembly property depending on how I invoke a test using Microsoft.Testing.Platform. It's unclear to me if this is expected behavior, but from my understanding / perspective, these scenarios should all be using Microsoft.Testing.Platform, and thus should all behave the same.

Repro steps

1. Clone repo

git clone https://github.com/microsoft/codecoverage.git
cd codecoverage/samples/Algorithms/tests/Algorithms.Core.Tests/

2. Add TestingPlatformDotnetTestSupport

<PropertyGroup>
  <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
</PropertyGroup>

dotnet run

dotnet run --coverage --coverage-output report.cobertura.xml --coverage-output-format cobertura

Result: Tests are not included in report.cobertura.xml

dotnet test

dotnet test -- --coverage --coverage-output report.cobertura.xml --coverage-output-format cobertura

Result: Tests are not included in report.cobertura.xml

dotnet coverage

dotnet tool install -g dotnet-coverage
dotnet-coverage collect --output report.cobertura.xml --output-format cobertura "dotnet run --no-build"

Result: Test are included in report.cobertura.xml

This is Scenario04 in the samples. Given that this is also supposed to be using the MSTest runner according to https://github.com/microsoft/codecoverage/blob/main/samples/Algorithms/README.md, I'd expect this to output the same as the other two entrypoints.

[fhnaseer]

This is the expected behavior. dotnet-coverage and Microsoft.CodeCoverage.Console.exe works on the principal that they will collect coverage for the command provided in the command line (dotnet run in this scenario). Users can run a console application, GUI based application or run tests. dotnet-coverage treats them all as an application.

Microsoft.Testing.Platform uses Microsoft.Testing.Extensions.CodeCoverage which is aware that it is running tests and the default behavior is skip tests for coverage.

[MattKotsenas]

Is there somewhere where this difference is / can be documented? Or better yet some guidance about which tool is best suited for each use case?

Once you explain it to me it makes sense, but I wouldn't have expected the code coverage tool to provide different output than running the tests directly.

[fhnaseer]

The official documentation states "Enables the cross-platform collection of code coverage data of a running process." https://learn.microsoft.com/en-us/dotnet/core/additional-tools/dotnet-coverage#description
Other components (Microsoft.Testing.Extensions.CodeCoverage and Microsoft.CodeCoverage) are testing platform extension and hence have a slightly different default behavior.

[MattKotsenas]

Sure. However it is not at all clear that

"Enables the cross-platform collection of code coverage data of a running process."

Implies that it has different default behavior, nor is it clear from that single sentence which behaviors will differ.

[fhnaseer]

I think there is confusion because of comparing two different tools together. dotnet-coverage and dotnet test are two different tools. Since they both provide code coverage functionality, it does not mean they will provide same results. They have different use cases and different settings by default.

dotnet-coverage => collection of code coverage data of a running process => Here we are talking only about dotnet-coverage as a standalone tool. We are not comparing it with dotnet test or coverlet or any other application which provide code coverage and how its behavior is different than others.

It makes sense to compare Microsoft.CodeCoverage and Microsoft.Testing.Extensions.CodeCoverage as they both are extensions for testing and have the same use case. If their behavior is different, then we should document it.

[MattKotsenas]

I think there is confusion because of comparing two different tools together.

I agree. The conversation has shifted from "dotnet-coverage works differently from the test tools" to "we should document that, how, and why dotnet-coverage works differently from the test tools".

I don't think it's reasonable to assume that "Enables the cross-platform collection of code coverage data of a running process" is enough to explain to users the differences. If this is explained elsewhere and I haven't found it, please let me know. Thanks!

[fhnaseer]

Default behavior of IncludeTestAssembly has been added in docs for vstest and Microsfot.Testing.Platfrom.