[BUG] Nondeterministic variance in OpenCover offset/offsetend

[sharpjs]

Describe the bug
I am seeing intermittent, nondeterministic variance in the OpenCover offset and offsetend attributes for the same BranchPoint across different test runs, when neither the code nor the build environment have changed. To illustrate, here is a diff of branch coverage data for a pair of if statements. These images come from subsequent CI pipeline runs for the same commit, using the same CI agent image. The left side shows the expected offset and offsetend values, while the right side shows a case of variance.

This variance creates a problem when merging coverage data from multiple test runs (e.g. a unit test run and an integration test run) using ReportGenerator. ReportGenerator interprets BranchPoints with differing offset/offsetend as different branch cases. The variance inflates total coverable branch count, which decreases branch coverage. Here are excerpts from the coverage reports for the above two builds:

The left build, with no variance:

The right build, where variance inflated the branch count:

The CI pipeline for this product is configured to fail if coverage decreases. Thus the intermittent variance tends to cause failed CI builds, even when no code has changed.

I have tried the following workarounds thus far:

• Changed the pipeline to have a static minimum coverage instead of using the most recent production-quality build as baseline.
• Changed Coverlet from OpenCover to lcov output to see if that would help. Will know after a day.

To Reproduce

• Magically gain access to my team's CI system.
• Undo the workarounds we added for this problem.
• Run the main build-and-test CI pipeline ten times for the same commit.

Expected behavior

• The pipeline runs generate identical coverage data.
• None of the runs fail due to false-negative coverage decrease.

Actual behavior

• One pipeline run has its unit tests and its integration tests (two separate dotnet test invocations) disagree on the offset/offsetend for a particular if statement.
• ReportGenerator thinks that if statement has four branch points instead of two.
• The pipeline run fails due to false-negative coverage decrease.

Configuration (please complete the following information):

• .NET: Framework 4.8.1, SDK 6.0.428, SDK 8.0.417
• Coverlet: collector 6.0.4 and 8.0.0-gbc66628323
• OS: Azure DevOps Pipelines windows-2022 image, version 20260127.11
• Architecture: x64
• Specific to that config? don't know

[Bertk]

Thank you for the detailed description.

The changing offset and offsetend values are difficult to reproduce, and without a deeper analysis of the generated OpenCover files they are not easily fixable.

I found a similar ReportGenerator issue Same code version is producing different coverage results which may offer some relevant insights:

1. High memory usage
   I will have a look at the code of ReportGenerator. I have some ideas how I could possibly reduce memory usage. The number of files in the history directory should not matter that much, since only the latest 100 are parsed by default. Your code base has ~6000 classes, that's also not that much. But you have a lot of OpenCover files, that may cause the high memory consumption.

2. Different coverage results on same code base
   In your table the CoverableLine and TotalLines are constant (line1/2 and line 6/7) only CoveredLines and UncoveredLines differ.
   It seems that your source code is really the same, otherwise the TotalLines would probably differ.
   But: Are you sure that your unit tests produce deterministic results? Or do you have at least one test where coverage depends on your environment (date, time, network connection etc). That would explain the difference in CoveredLines and UncoveredLines. To be 100% sure we have to compare the 73 OpenCover files and see if they differ.
   Are those files from both runs still available?

3. Old version of file
   ReportGenerator uses the files that are listed in your OpenCover files. The full path should be visible in the reports. Are you sure a file is used that is not referenced in any of your OpenCover files?

Hopefully this provides some direction for further investigation.