Support pip versions up to 25.1

[sirosen]

This PR is based on #2178 and therefore includes those commits.

Part 1: Relativizing Paths

Resolves #2131

This is a much simpler approach than the one which was attempted in #1650 , and it is imperfect. There is a heuristic being used here to try to produce good outputs, and that heuristic can be tricked. More on this below.

Although I did not make use of that work directly in creating this changeset, it deserves citation as inspiration, and for having demonstrated a path to more complete handling which we won't pursue -- the avenues closed off by that change proposal and review process helped to inform the choice here to do something ultra simple.

---

First, this changeset introduces a new conditional behavior for tweaking the comes_from annotation data.
pip-compile needs to handle the fact that the comes_from field on a parsed requirement may be rewritten from a relative path to an absolute one.

The change in pip occurred in version 24.3. comes_from is populated with absolute paths. I believe this is an effect of pypa/pip#12877 , which was a change included in that release and in which paths are made absolute in an internal pip context.
As far as pip-compile is concerned, the main thing which matters is that requirements and constraints files will always appear with absolute paths on newer pip versions.

pip-compile targets usage in which a relative path is supplied as input and ought to be preserved in the annotations for its outputs. As such, even though the relative path has been translated to an absolute one, the original (relative) path should be rendered.

Because the data coming back from pip are ambiguous to some degree, as a heuristic, this change will use whether or not the original path was absolute to determine behavior. Therefore, the following files will result in correct behavior:

# reqs.in
-r other-reqs.in
# other-reqs.in
some-pkg

some-pkg came from -r other-reqs.in.

By contrast, if the CWD is /opt, and -r reqs.in is used, these data will be incorrectly rendered with relative paths:

# reqs.in
-r /opt/other-reqs.in
# /opt/other-reqs.in
some-pkg

some-pkg came from -r /opt/other-reqs.in, but will be annotated with -r other-reqs.in

Note that the relative path which is used is, in this scenario, correct with respect to the original invocation directory, but not reflective of the inputs to the program.

It is possible, if the absolute paths are really wanted in such a case, to ask for them using pip-compile -r /opt/reqs.in.

---

It is possible to enhance this behavior in the future, potentially with a more robust approach, but this changeset tries to make a simple guess which will be right in >80% of realistic usages.

Part 2: Update the pipsupported tox env

With this fix in hand + #2178, it should be possible for tests to pass on newer pip versions.

Update pipsupported to pick the latest versions of pip and setuptools.¹
For Python 3.8, a lower pip version is chosen, as 25.0 dropped support for 3.8 .

Part 3: Normalize Windows Paths in comes_from

After finally getting the tests to work consistently with the paths produced by various pip versions for comes_from data, I've updated this changeset to also add a layer of normalization which ensures that we format paths as posix paths.

The driver for this was the discovery that you can make pip produce paths with mixed path separators, e.g.:

C:\foo\bar/baz/requirements.in

Contributor checklist

• Included tests for the changes.
• PR title is short, clear, and ready to be included in the user-facing changelog.

Maintainer checklist

• Verified one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
• Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

Footnotes

1. The setuptools bound might not be needed anymore on newer pip versions, but this needs separate assessment. ↩

[webknjaz]

@sirosen I think this is nice overall. And even if imperfect, we should merge it to unblock other things. But see my inline suggestions first and decide whether they need to be in the scope of this PR or postponed.

[webknjaz]

Instead of having a lot of setup logic in the test and raw strings as params, let's try to move it out.

One cool technique that one of the pytest maintainers once shared is using dataclasses to encapsulate coupled bits of data in params.

The idea is that you define a shape like

from dataclasses import dataclass


@dataclass(frozen=True)
class Thing:
    name: str
    stuff: object

    ...

    def __str__(self) -> str:
        return self.name

And then use it like

@pytest.mark.parametrize(
    'thing',
    (
        Thing('case-name'),
        Thing('another-case'),
    ),
    id=str,
)

If you can extend that with more attributes and maybe add a method for setting up the test directory, this would remove complex logic from the test function, keeping only testing in.

[sirosen]

I've used a similar technique in the past -- maybe with named tuples? -- but this looks like a nice way to do it, and id=str feels clever and classically pythonic.

I'm happy to do this, and some of the other associated refactoring. I was trying to avoid introducing a bunch of new project infrastructure in the form of fixtures, but it's probably worth it. I'll start working through it!

[webknjaz]

Thanks! It was just that I was reading through the tests, and they seemed to do a lot of stuff, have a lot of logic and branches. It felt like we'd need tests for tests. So in my suggestion I was trying to come up with something that would keep the tests simple as the typical recommendation is to avoid any complexity in tests.

[sirosen]

I took a crack at this, trying to avoid baking in any test execution logic, and am happy with what I produced for now. As we've been discussing, I'd like to do a broader test refactor later to see where the things I did apply and where they can/should be modified.

[webknjaz]

@sirosen apparently, I suggested .as_posix() in the wrong places. It doesn't return a PurePosixPath as I'd hoped but an instance of str. So the .as_posix() invocations will have to be applied on conversion, not early.

[webknjaz]

I think the one you pass into CLI shouldn't matter. Only the paths that end up written to output would be important to convert like this.

[sirosen]

Windows CI is still failing on the new tests, and as seen in comments above I did get a little mixed up about applying as_posix() in at least one wrong place.

So this is going to take a bit more time to sort out, but I think it's worth the extra time. It gives us clear information about what the pip-compile behavior really is, which we won't have otherwise, and there's a chance that I've introduced some bug in the Windows behavior which I need to fix.

[sirosen]

I've just finished working out the issues with Windows CI. I made more changes to the way that the paths are normalized/handled and documented it in the original PR comment as "Part 3: Normalize Windows Paths in comes_from".

1c6da33 shows what it took to get CI passing on Windows without adding this extra normalization.

The mixed-separator paths (C:\foo\bar/baz/requirements.in) are sufficiently "wrong" that I decided to grow the scope of these changes to normalize things to posix-style, even though it's a bit of scope creep. @webknjaz, if you are strongly against this expansion to the PR, I've separated it out and I can revert that change ( f2436f2 ).

[webknjaz]

Got a few suggestions. Looks good otherwise.

[webknjaz]

Wonderful! Please, see if you can improve Git history before merging.

[mering]

Thanks for fixing this! When do you have the next release including this planned?