Misleading subprocess coverage with relative path --cov

[ndevenish]

If --cov is passed relative, not absolute, and a subprocess is run in a different directory, then the subprocess coverage will be silently ignored. Example:

test_subprocess_coverage.py

import os
import subprocess
import sys


def do_something():
    print("A test in {}".format(os.getcwd()))


def test_subprocess_coverage(tmp_path):
    apy = os.path.abspath(__file__)
    print("Calling {}".format(apy))
    # Call this same script externally from the temporary path
    subprocess.call([sys.executable, apy, "--do"], cwd=str(tmp_path))


if __name__ == "__main__":
    if "--do" in sys.argv:
        do_something()

Output:

$ pytest test_subprocess_coverage.py --cov=.
test_subprocess_coverage.py      12      3    75%

$ pytest test_subprocess_coverage.py --cov=$(pwd)
test_subprocess_coverage.py      12      0   100%

I think I understand the reasons why, that the path is passed through in the environment relative so the child coverage has no idea what is being checked - but the fact that it's silently failing has bitten us several times. (instantiating child jobs as in-process function calls is something we are slowly working towards, but is a long ways off).

I'd imagine that just making the environment variable absolute (guessing here) would solve this for my case, but imagine might cause issues with cluster-distributed testing where the code might have been copied somewhere different? In which case maybe there is a similar bug when cluster-distributed and passing in an absolute path....

Edit: A slightly similar issue seems to have been dealt with for the .coveragerc file in #94/#95

[ionelmc]

It appears you have found a quirk in coveragepy. The "source" (whatever you pass to --cov) can be empty (None) or a module name - it would be tricky to absolutize it, and consequently duplicate or reimplement whatever is in coveragepy. Not really a fan of that idea.

Personally I consider your particular configuration an antipattern - you should avoid using paths. That configuration will be broken when you change from setup.py develop to setup.py install. Use module names or nothing (just --cov - you can filter the output via .coveragerc) instead.

[ndevenish]

If using a path is an antipattern..... It's explicitly documented to be a filesystem path. Also in pytest --help:

...
--cov=[path]          measure coverage for filesystem path (multi-allowed)

It's not at all documented as a bare option, as far as I can tell.

This particular case came up with a colleague wanting to test the coverage of some subset of a module rather than everything. At the very least the problem should be listed in the caveats of subprocess, or detected and warned about.

[ionelmc]

[something] means that something is optional. The help text will be improved.