Allow-newer conditional on GHC-version is not respected

[jasagredo]

Describe the bug

The allow-newer constraints don't get respected when enclosed in a conditional block.

To Reproduce
Checkout this commit: https://github.com/input-output-hk/ouroboros-consensus/tree/ed184183d048af6126f9e00e32862d669e705899. And maybe you need to install some dependencies, in particular pkg-config, libsodium and liblmdb-dev in Ubuntu.

cabal build all --with-compiler=ghc-9.6.2 --dry-run

succesfully creates a build plan. However, applying this diff:

allow-newer:
  , lens
  , cardano-ledger-core
  , cardano-ledger-alonzo
  , cardano-ledger-babbage
+if impl(ghc >= 9.6)
+  allow-newer:
+    , *:base
+    , *:ghc-prim
-  , *:base
-  , *:ghc-prim

results in cabal failing to create a build plan, by saying:

[__3] rejecting: base-4.18.0.0/installed-4.18.0.0 (conflict: small-steps-test => base>=4.12 && <4.17)

So the constraint allow-newer: *:base is not being respected.

Expected behavior

I expected cabal to produce a build plan.

System information

• Ubuntu 23.04
• cabal 3.10.1.0
• ghc 9.6.2

[jasagredo]

I just checked and it seems like it doesn't have to do with having conditional and unconditional allow-newer blocks, as removing the unconditional one still makes it fail with base incompatibility:

❯ git --no-pager diff
diff --git a/cabal.project b/cabal.project
index 9c83c70a2a..350469ccf1 100644
--- a/cabal.project
+++ b/cabal.project
@@ -82,13 +82,10 @@ package strict-stm
 package text-short
   flags: +asserts
 
-allow-newer:
-  , lens
-  , cardano-ledger-core
-  , cardano-ledger-alonzo
-  , cardano-ledger-babbage
-  , *:base
-  , *:ghc-prim
+if impl(ghc >= 9.6)
+  allow-newer:
+    , *:base
+    , *:ghc-prim
 
 -- TODO: remove when a new version of strict-mvar is released
 source-repository-package

❯ cabal build all --with-compiler=ghc-9.6.2 --dry-run
Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: cardano-ledger-core-1.4.1.0 (user goal)
[__1] trying: cardano-ledger-byron-test-1.5.0.0 (dependency of
cardano-ledger-core)
[__2] trying: small-steps-test-1.0.0.0 (dependency of
cardano-ledger-byron-test)
[__3] next goal: base (dependency of cardano-ledger-core)
[__3] rejecting: base-4.18.0.0/installed-4.18.0.0 (conflict: small-steps-test
=> base>=4.12 && <4.17)
[__3] skipping: base-4.18.0.0, base-4.17.1.0, base-4.17.0.0 (has the same
characteristics that caused the previous version to fail: excluded by
constraint '>=4.12 && <4.17' from 'small-steps-test')
[__3] rejecting: base-4.16.4.0, base-4.16.3.0, base-4.16.2.0, base-4.16.1.0,
base-4.16.0.0, base-4.15.1.0, base-4.15.0.0, base-4.14.3.0, base-4.14.2.0,
base-4.14.1.0, base-4.14.0.0, base-4.13.0.0, base-4.12.0.0, base-4.11.1.0,
base-4.11.0.0, base-4.10.1.0, base-4.10.0.0, base-4.9.1.0, base-4.9.0.0,
base-4.8.2.0, base-4.8.1.0, base-4.8.0.0, base-4.7.0.2, base-4.7.0.1,
base-4.7.0.0, base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0,
base-4.4.1.0, base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2,
base-4.2.0.1, base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2,
base-3.0.3.1 (constraint from non-upgradeable package requires installed
instance)
[__3] fail (backjumping, conflict set: base, cardano-ledger-core,
small-steps-test)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: base, small-steps-test,
cardano-ledger-core, cardano-ledger-byron-test
Try running with --minimize-conflict-set to improve the error message.

[jasagredo]

Sorry for the confusion. I really don't understand what happened as I was getting those errors consistently. However, I switched the ghc un GHCup, then removed dist-newstyle, etc, and now it works. Perhaps some stale info was in dist-newstyle. Closing as I cannot even reproduce it myself.

[jasagredo]

Ok I do have a reproduction:

❯ git --no-pager diff
diff --git a/cabal.project b/cabal.project
index 9c83c70a2a..350469ccf1 100644
--- a/cabal.project
+++ b/cabal.project
@@ -82,13 +82,10 @@ package strict-stm
 package text-short
   flags: +asserts
 
-allow-newer:
-  , lens
-  , cardano-ledger-core
-  , cardano-ledger-alonzo
-  , cardano-ledger-babbage
-  , *:base
-  , *:ghc-prim
+if impl(ghc >= 9.6)
+  allow-newer:
+    , *:base
+    , *:ghc-prim

❯ cabal build all --with-compiler=ghc-9.6.2 --dry-run
... succeeds

❯ nano cabal.project
... change it to `if impl(ghc >= 9.4)`

❯ cabal build all --with-compiler=ghc-9.4.5 --dry-run
Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: cardano-ledger-core-1.4.1.0 (user goal)
[__1] trying: cardano-ledger-byron-test-1.5.0.0 (dependency of
cardano-ledger-core)
[__2] trying: small-steps-test-1.0.0.0 (dependency of
cardano-ledger-byron-test)
[__3] next goal: base (dependency of cardano-ledger-core)
[__3] rejecting: base-4.17.1.0/installed-4.17.1.0 (conflict: small-steps-test
=> base>=4.12 && <4.17)
[__3] skipping: base-4.18.0.0, base-4.17.1.0, base-4.17.0.0 (has the same
characteristics that caused the previous version to fail: excluded by
constraint '>=4.12 && <4.17' from 'small-steps-test')
[__3] rejecting: base-4.16.4.0, base-4.16.3.0, base-4.16.2.0, base-4.16.1.0,
base-4.16.0.0, base-4.15.1.0, base-4.15.0.0, base-4.14.3.0, base-4.14.2.0,
base-4.14.1.0, base-4.14.0.0, base-4.13.0.0, base-4.12.0.0, base-4.11.1.0,
base-4.11.0.0, base-4.10.1.0, base-4.10.0.0, base-4.9.1.0, base-4.9.0.0,
base-4.8.2.0, base-4.8.1.0, base-4.8.0.0, base-4.7.0.2, base-4.7.0.1,
base-4.7.0.0, base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0,
base-4.4.1.0, base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2,
base-4.2.0.1, base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2,
base-3.0.3.1 (constraint from non-upgradeable package requires installed
instance)
[__3] fail (backjumping, conflict set: base, cardano-ledger-core,
small-steps-test)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: base, small-steps-test,
cardano-ledger-core, cardano-ledger-byron-test
Try running with --minimize-conflict-set to improve the error message.

❯ rm -rf dist-newstyle 

❯ cabal build all --dry-run --with-compiler=ghc-9.4.5
... succeeds

[andreabedini]

It might be related to #8772 and #8699

Can you try with cabal head?