feat: exclude lines that are missing from source maps

[AriPerkkio]

• Closes Comments are reported as not covered #238
• Supersedes and closes feat: exclude lines from processing #212
• Supersedes and closes Ignore lines from code coverage which are not executable #242

feat: exclude lines that are missing from source maps

• When source maps are available, use @jridgewell/trace-mapping to figure out which lines contain runtime code. If a line is not present in source maps, consider it as empty line. This will exclude Typescript typings, comments, empty lines and any other special syntax that transpiled languages exclude from source maps.
• Should this be considered as breaking change or be a configurable option? ⚠️
  • This is now opt-in and configurable via excludeEmptyLines argument. No breaking changes.

Examples:

• Vitest

  • 
  • 

• Jest

  • 

• Playwright

  • 

• c8, test code transpiled with tsc. (tsconfig.json's removeComments was disabled when taking the picture)

  • 

fix: ignore hint to exclude lines

• /* v8 ignore ... */ should exclude the lines from coverage, not mark them as covered.
  • 
    

[AriPerkkio]

Any thoughts about this approach @SimenB and @bcoe?

In Vitest we will always have source maps available so we can utilize those for analysing the source file. I believe Jest has them too? I'm not that sure about c8 though.

When source maps are not passed, the empty lines should be marked as covered as before.

[joshuanathanson]

Possible to get this merged? We're currently forking this commit so we can get proper coverage numbers in our repo. Def not ideal.

[AriPerkkio]

@joshuanathanson - just curious, what test runner and coverage tool are you using this commit with?

[joshuanathanson]

@joshuanathanson - just curios, what test runner and coverage tool are you using this commit with?

We have our own wrappers, but under the hood we're using Selenium to run the tests, along with grabbing the coverage from Chrome on the fly with Selenium CDP DevTools.

[SimenB]

This is one of those I'd love to hear @bcoe's thoughts 👀

[lo1tuma]

I think this would also fix the related issue in c8. Would be really nice to get this merged.

[joshuanathanson]

Bump on this? Revisiting our work in this area and would be nice to get this merged.

[AriPerkkio]

This feature has been included in Vitest v1.6 and enabled by default in Vitest v2 for ~2 weeks now. We haven't (yet) heard any issues about this change.

@bcoe there are many issues in c8 that would be fixed by this PR (as long as source maps are available there):

• 0% coverage for files only containing TypeScript interfaces bcoe/c8#494
• Maybe export is instrumented with 0 coverage incase of zero test project bcoe/c8#528
• c8 Code Coverage Displaying Code Misses on Multi-line Comments bcoe/c8#515
• Most of Coverage information too high for large esbuild produced bundle bcoe/c8#502, if not all
• Lines without statements (empty lines) on them are counted as statements bcoe/c8#500

[ericmorand]

Isn't there a risk that lines removed by treeshaking are marked as empty? This is maybe not an issue (after all if they are not in the built artifact, it means that they are not relevant) but it may look strange in reports that they are marked as empty.

[AriPerkkio]

@ericmorand yup, those lines cannot be covered so I would expect them to be excluded from report. This is similar how nyc and Jest (with default babel provider) works.

[AriPerkkio]

Looks like node:test started to exclude lines that are not present in source maps too: https://github.com/nodejs/node/pull/55228/files#diff-f93d30118ca2d349a10c8f93d740e1c988671a2e35ffe1213fedd6868df60e61R48

[AriPerkkio]

Closing in favor of https://github.com/AriPerkkio/ast-v8-to-istanbul.

I ended up rewriting v8-to-istanbul with AST based remapping logic. It's intended to be used as direct replacement of v8-to-istanbul with minimal API changes. It should be outputting close to 100% identical coverage maps as istanbul-lib-instrument does.

[bcoe]

@AriPerkkio cool, I'd be curious to see how much breaks in c8 dropping this in.

[AriPerkkio]

Here's example of the Vitest integration. At the moment it's supporting both v8-to-istanbul and ast-v8-to-istanbul but we will be dropping the original v8-to-istanbul in next major release.

https://github.com/vitest-dev/vitest/blob/ba6da4d070bd26d4ec40efbf6b37d431877e7f2f/packages/coverage-v8/src/provider.ts#L245-L308

Though I don't know how you would get contents of the executed Javascript file in c8. 🤔

[bcoe]

@AriPerkkio one major problem for Node's use of c8 is, if I recall, we don't always have the source code so it did something interesting and popualted:

...............
.........
........................
...
...............

I don't fully recollect how it was that I have the line length but not the actual source, but I'm pretty sure I'm remembering correctly. tldr; an AST wouldn't work on it.