Add the compat_modifier kwarg to Pkg.test and remove the force_latest_compatible_version and allow_earlier_backwards_compatible_versions kwargs

[DilumAluthge]

Summary

This pull request:

• Adds the compat_modifier::Union{Function, Nothing} keyword argument to the Pkg.test function
• Removes the force_latest_compatible_version keyword argument from the Pkg.test function
• Removes the allow_earlier_backwards_compatible_versions keyword argument from the Pkg.test function

The default value of compat_modifier is compat_modifier = nothing.

If compat_modifier is a Function, it is called with exactly one positional argument x which has the following properties:

• x.name is the name of the dependency
• x.uuid is the UUID of the dependency
• x.has_compat::Bool is:
  • if the dependency has a [compat] entry: true
  • if the dependency does not have a [compat] entry: false
• x.versions::Set{VersionNumber} is
  • if the dependency has a [compat] entry: the set of all registered versions of the dependency that are compatible with the original [compat] entry for the dependency
  • if the dependency does not have a [compat] entry: the set of all registered versions of the dependency

If compat_modifier is a Function, it must return exactly one return value, which must be a Union{Pkg.Versions.VersionSpec, Nothing}.

See also: #2439

Description

If compat_modifier is nothing, then we do:

1. Pkg.test sets up the temporary sandbox project.
2. Pkg.test runs the resolver inside the temporary sandbox project.
3. Pkg.test runs the tests/runtests.jl file inside the temporary sandbox project.

If compat_modifier is a Function, then we do:

1. Pkg.test sets up the temporary sandbox project.
2. We assemble the list of all direct dependencies. For each direct dependency DepName, we do the following:
   1. We set has_compat to true if the project has a [compat] entry for DepName, and false otherwise.
   2. We assemble the set of all registered version numbers of DepName.
   3. Based on the value of has_compat:
      • If has_compat is true: We get the original [compat] entry for DepName, and then for each registered version number of DepName, we determine whether or not the version number is inside the original [compat] entry for DepName. At the end of this step, we have the Set{VersionNumber} containing all of the registered version numbers of DepName that are compatible with the original [compat] entry for DepName. We set versions to be this set.
      • If has_compat is false: We set versions to be the Set{VersionNumber} containing all registered version numbers of DepName.
   4. We call the compat_modifier function as follows: spec = compat_modifier((; name, uuid, has_compat, versions))
   5. We check that spec is a Union{Pkg.Versions.VersionSpec, Nothing} and throw an error if it is not.
   6. Based on the type of spec:
      • If spec is a Nothing, we do nothing.
      • If spec is a Pkg.Versions.VersionSpec, we do the following, based on the value of has_compat:
        • If has_compat is true: We delete the original [compat] entry for DepName and set the new [compat] entry for DepName to be the intersection of spec and the original [compat] entry for DepName.
        • If has_compat is false: We delete the original [compat] entry for DepName and set the new [compat] entry for DepName to be spec.
3. Pkg.test runs the resolver inside the temporary sandbox project.
4. Pkg.test runs the tests/runtests.jl file inside the temporary sandbox project.

Example Usage

Suppose that your package has a [compat] entry that looks like this:

[compat]
Foo = "0.2, 0.3, 0.4"

And suppose that the Foo.jl function has the following versions in the registry:

• 0.1.0
• 0.1.1
• 0.2.0
• 0.2.1
• 0.3.0
• 0.3.1
• 0.4.0
• 0.4.1
• 0.5.0
• 0.5.1

Observe that only the following versions of Foo.jl are compatible with your package's [compat] entry for Foo:

• 0.2.0
• 0.2.1
• 0.3.0
• 0.3.1
• 0.4.0
• 0.4.1

You can control which version of Foo.jl is used in your package's test suite by passing a function to the compat_modifier keyword argument. The function will be applied to all direct dependencies. (It will not be applied to any indirect dependencies.) Here are some examples:

Description	Allowed Versions	compat_modifier
Force latest compatible version	0.4.1	x -> Pkg.Versions.semver_spec("=$(maximum(x.versions))")
Force latest compatible "breaking family" of versions	0.4.x	x -> Pkg.Versions.semver_spec("^$(Pkg.Operations.earliest_backwards_compatible(maximum(x.versions)))")
Force earliest compatible version	0.2.0	x -> Pkg.Versions.semver_spec("=$(minimum(x.versions))")
Force earliest compatible "breaking family" of versions	0.2.x	x -> Pkg.Versions.semver_spec("^$(minimum(x.versions))")
Select a random compatible version	random	x -> Pkg.Versions.semver_spec("=$(StatsBase.sample(collect(x.versions)))")

If you only want to apply this to direct dependencies that have [compat] entries (i.e. you want to skip any direct dependency that does not have a [compat] entry), you can do the following instead:

Description	Allowed Versions	compat_modifier
Force latest compatible version	0.4.1	Pkg.Operations.force_latest_compatible_exact
Force latest compatible "breaking family" of versions	0.4.x	Pkg.Operations.force_latest_compatible_family
Force earliest compatible version	0.2.0	x -> x.has_compat ? Pkg.Versions.semver_spec("=$(minimum(x.versions))") : nothing
Force earliest compatible "breaking family" of versions	0.2.x	x -> x.has_compat ? Pkg.Versions.semver_spec("^$(minimum(x.versions))") : nothing)
Select a random compatible version	random	x -> x.has_compat ? Pkg.Versions.semver_spec("=$(StatsBase.sample(collect(x.versions)))") : nothing

For CompatHelper/Dependabot PRs, we would use the bolded row, i.e. the second row of the second table above.

Motivation

In #2439, we added the ability to do the following when running your package tests:

1. For each direct dependency, force exactly the latest compatible version.
2. For each direct dependency, force either the latest compatible version, or any version that is compatible (according to SemVer) with the latest compatible version.

The motivation for #2439 was to meet the CompatHelper/Dependabot use case.

However, users may have other restrictions that they want to place on their direct dependencies when running their package tests. Some examples include:

1. For each direct dependency, force exactly the oldest (earliest) compatible version.
2. For each direct dependency, force either the oldest (earliest) compatible version, or any version that is compatible (according to SemVer) with the oldest (earliest) compatible version.

Initially, I considered adding this functionality directly to Pkg. However, this would require adding a lot of extra keyword arguments to the Pkg.test function.

Instead, I figured it would be better to have only a single compat_modifier keyword argument to the Pkg.test function. Then, users can write a function that does exactly what they want, and pass this function to the compat_modifier kwarg.

[DilumAluthge]

@KristofferC @00vareladavid I had to add back the code for converting spec::Pkg.Versions.VersionSpec to a str::AbstractString such that Pkg.Versions.semver_spec(str) == spec. This is because the order of operations in this PR is as follows:

1. Pkg.test sets up the temporary sandbox project.
2. We modify the temporary sandbox project. During this process, some of the compat.val will fall out of sync with the corresponding compat.str.
3. Pkg.test runs the resolver inside the temporary sandbox project.

The problem is this: resolve calls up which calls write_env which calls write_project which calls destructure. And destructure will throw an error if any of the compat.val do not match the corresponding compat.str.

I'm happy to remove the "convert a VersionSpec to a string" functionality if we can figure out another way for this to work.

[DilumAluthge]

@KristofferC I have removed the Versions.semver_spec_string functionality, and I have implemented your suggestion from #2482 (comment).

[oxinabox]

In the real world o don't think this is a practical solution to testing oldest versions nor to testing random ones.
Not for packages with many dependencies that themselves have interlocking dependencies.
E.g try something that depends directly on CSV, Tables, CategoricalArrays, and DataFrames
Each of those are only compatible with a narrow range of versions of the others.

Getting either of those out of that would in practice be very hard AFAICT
If let's you modify the version spec, but when you do so you only see one at a time.

Getting the oldest version of the packages is not quite the same as setting the version spec for each package to be only the oldest permitted.
That would be something like "force_oldest".
And it would go e you the oldest if and only if there is a single mutually compatible solution that lets everything be set to oldest. Rather than something's being force to be newer to be compatible with others.
Potentially the oldest versions you directly permit of one might not actually be compatible with any set of your other dependencies.

More generally would want to test some point(/s) on the Pareto frontier of oldness.

Similarity you can't get random compatible versions by independently randomly restricting version specs.
Since the result might not be compatible.

But maybe I misunderstand.

It seems like one can't do this without running the resolver with an alternative objective.
Oldest for oldest.
And for random: something like nearness to some random set of versions

[DilumAluthge]

Instead of passing the versions one at a time to compat_modifier, we could pass a vector with all of the versions. And we could include additional information, like all of the names and UUIDs. And then the compat_modifier could do whatever logic, and would return a vector of VersionSpecs.

[DilumAluthge]

It seems like one can't do this without running the resolver with an alternative objective.

If we pass a vector (one element for each direct dependency) to the compat_modifer function, you could run an alternate resolution algorithm and return the results in the form of a vector of VersionSpecs.

[oxinabox]

Yes, and to do so the user would have to reimplement the resolver.
Which is not a reasonable thing to ask the user to do.

And if they are doing that,then we might want to step back and ask if this is the API we want.

[DilumAluthge]

Fair enough. In that case, I guess the main benefit of this PR is the CompatHelper use case, i.e. force the latest compatible versions of all direct dependencies.

So, I'll keep the current setup of passing one dependency at a time to compat_modifier.

However, I'll add name and uuid to the information that is passed to compat_modifier. So at least you can easily implement things like "use the oldest compatible version of Foo, but all other [compat] entries are unmodified."

[DilumAluthge]

Okay, now we also pass name and uuid to compat_modifier. So now you can also do things like this:

1. Force the latest compatible version of Foo.jl, but leave all other [compat] entries unmodified
2. Force the earliest compatible version of Foo.jl, but leave all other [compat] entries unmodified

Some examples:

Description	Allowed Versions	compat_modifier
Force latest compatible version	0.4.1	x -> (x.name == "Foo") ? Pkg.Versions.semver_spec("=$(maximum(x.versions))") : nothing
Force latest compatible "breaking family" of versions	0.4.x	x -> (x.name == "Foo") ? Pkg.Versions.semver_spec("^$(Pkg.Operations.earliest_backwards_compatible(maximum(x.versions)))") : nothing
Force earliest compatible version	0.2.0	x -> (x.name == "Foo") ? Pkg.Versions.semver_spec("=$(minimum(x.versions))") : nothing
Force earliest compatible "breaking family" of versions	0.2.x	x -> (x.name == "Foo") ? Pkg.Versions.semver_spec("^$(minimum(x.versions))") : nothing
Select a random compatible version	random	x -> (x.name == "Foo") ? Pkg.Versions.semver_spec("=$(StatsBase.sample(collect(x.versions)))") : nothing

[KristofferC]

Similarity you can't get random compatible versions by independently randomly restricting version specs.
Since the result might not be compatible.

I think this is an important point made by @oxinabox in that the desired end result (affect the resolver output in various ways) and the implementation (restricting compat entries) are not really in "harmony". The way you would get the lowest compatible version of everything is to tell the resolver to "like" low versions over high versions which is quite different from changing compat. Because changing compat is very hard and you might very well restrict into something that has no solution.

Force the latest compatible version of Foo.jl, but leave all other [compat] entries unmodified

This just looks like a pin, or?

---

FWIW, I personally think this exposes too much API for the demand there is on the functionality it provides and to use it properly you need to use a lot of Pkg internals.

If there is a need to override compat for CompatHelper then I think it is ok to put something very specific (and non-public) that handles that case but starting to generalize that too much might not be a good idea.

[DilumAluthge]

Force the latest compatible version of Foo.jl, but leave all other [compat] entries unmodified

This just looks like a pin, or?

Not quite. If I understand correctly, you can only pin to a single version. In contrast, this kind of functionality would restrict to any Foo version 0.4.x.

[DilumAluthge]

FWIW, I personally think this exposes too much API for the demand there is on the functionality it provides and to use it properly you need to use a lot of Pkg internals.

If there is a need to override compat for CompatHelper then I think it is ok to put something very specific (and non-public) that handles that case but starting to generalize that too much might not be a good idea.

That's fair.

We already have the following kwargs for the CompatHelper case:

1. force_latest_compatible_version
2. allow_earlier_backwards_compatible_versions

So we can just keep those two kwargs, and close this PR.

[DilumAluthge]

If there is a need to override compat for CompatHelper then I think it is ok to put something very specific (and non-public) that handles that case but starting to generalize that too much might not be a good idea.

I have implemented this suggestion in #2541.

[DilumAluthge]

Closing in favor of #2541