[Bug?]: various yarn.lock modifications after v3 -> v4 migration

[akwodkiewicz]

Self-service

• I'd be willing to implement a fix

Describe the bug

After upgrading Tarn from 3.4.1 to 4.0.1, my 50k-line long yarn.lock had around 30k modified lines.

I looked at the diff and saw the following:

1. some changed metadata (seems reasonable)

    __metadata:
   -  version: 6
   -  cacheKey: 8
   +  version: 8
   +  cacheKey: 10

2. added npm: prefix to some dependencies for some packages (ok, I guess?) (EDIT: implemented here: Normalizes dependencies #4305)

   "@jest/schemas@npm:^29.4.3":
    version: 29.4.3
    resolution: "@jest/schemas@npm:29.4.3"
    dependencies:
   -  "@sinclair/typebox": ^0.25.16
   +  "@sinclair/typebox": "npm:^0.25.16"
    checksum: ac754e245c19dc39e10ebd41dce09040214c96a4cd8efa143b82148e383e45128f24599195ab4f01433adae4ccfbe2db6574c90db2862ccd8551a86704b5bebd
    languageName: node
    linkType: hard

3. new resolutions for existing dependencies (?!?)

   I checked 3 times to make sure that these changes are not introduced by anything other than the sheer Yarn upgrade:

   -"@babel/code-frame@npm:^7.0.0, @babel/code-frame@npm:^7.10.4, @babel/code-frame@npm:^7.12.13, @babel/code-frame@npm:^7.16.0, @babel/code-frame@npm:^7.16.7, @babel/code-frame@npm:^7.21.4, @babel/code-frame@npm:^7.22.10, @babel/code-frame@npm:^7.22.5, @babel/code-frame@npm:^7.5.5, @babel/code-frame@npm:^7.8.3":
   +"@babel/code-frame@npm:^7.0.0, @babel/code-frame@npm:^7.10.4, @babel/code-frame@npm:^7.12.13, @babel/code-frame@npm:^7.16.7, @babel/code-frame@npm:^7.18.6, @babel/code-frame@npm:^7.5.5, @babel/code-frame@npm:^7.8.3":
   +  version: 7.18.6
   +  resolution: "@babel/code-frame@npm:7.18.6"
   +  dependencies:
   +    "@babel/highlight": "npm:^7.18.6"
   +  checksum: 195e2be3172d7684bf95cff69ae3b7a15a9841ea9d27d3c843662d50cdd7d6470fd9c8e64be84d031117e4a4083486effba39f9aef6bbb2c89f7f21bcfba33ba
   +  languageName: node
   +  linkType: hard
   +
   +"@babel/code-frame@npm:^7.16.0, @babel/code-frame@npm:^7.22.10, @babel/code-frame@npm:^7.22.5":
      version: 7.22.10
      resolution: "@babel/code-frame@npm:7.22.10"
      dependencies:
   -    "@babel/highlight": ^7.22.10
   -    chalk: ^2.4.2
   -  checksum: 89a06534ad19759da6203a71bad120b1d7b2ddc016c8e07d4c56b35dea25e7396c6da60a754e8532a86733092b131ae7f661dbe6ba5d165ea777555daa2ed3c9
   +    "@babel/highlight": "npm:^7.22.10"
   +    chalk: "npm:^2.4.2"
   +  checksum: 53620d831c8f2230a7d2fbe833c01c071740a642317c960d45cda9b0b2d0492e152e00ab45aad8b55329ba5de647354b95f42b546fb905c0b7acf78d3f2d3ecd
   +  languageName: node
   +  linkType: hard
   +
   +"@babel/code-frame@npm:^7.21.4":
   +  version: 7.21.4
   +  resolution: "@babel/code-frame@npm:7.21.4"
   +  dependencies:
   +    "@babel/highlight": "npm:^7.18.6"
   +  checksum: 99236ead98f215a6b144f2d1fe84163c2714614fa6b9cbe32a547ca289554770aac8c6a0c0fb6a7477b68cf17b9b7a7d0c81b50edfbe9e5c2c8f514cc2c09549
      languageName: node
      linkType: hard

   There are lots of other packages that got additional resolutions ("un-deduped")

4. Changed checksums for some packages (??)

     "y18n@npm:^5.0.5":
      version: 5.0.8
      resolution: "y18n@npm:5.0.8"
   -  checksum: 54f0fb95621ee60898a38c572c515659e51cc9d9f787fb109cef6fde4befbe1c4602dc999d30110feee37456ad0f1660fa2edcfde6a9a740f86a290999550d30
   +  checksum: 5f1b5f95e3775de4514edbb142398a2c37849ccfaf04a015be5d75521e9629d3be29bd4432d23c57f37e5b61ade592fb0197022e9993f81a06a5afbdcda9346d
      languageName: node
      linkType: hard

   This is what originally made me create this issue, but then I found there are other changes things as well.

And for some packages -- a mix of everything mentioned above:

-"yargs@npm:^17.0.0, yargs@npm:^17.2.1, yargs@npm:^17.3.0, yargs@npm:^17.3.1, yargs@npm:^17.6.2, yargs@npm:^17.7.2":
+"yargs@npm:^17.0.0, yargs@npm:^17.3.0, yargs@npm:^17.3.1":
+  version: 17.6.0
+  resolution: "yargs@npm:17.6.0"
+  dependencies:
+    cliui: "npm:^8.0.1"
+    escalade: "npm:^3.1.1"
+    get-caller-file: "npm:^2.0.5"
+    require-directory: "npm:^2.1.1"
+    string-width: "npm:^4.2.3"
+    y18n: "npm:^5.0.5"
+    yargs-parser: "npm:^21.0.0"
+  checksum: f6159923d5234c040832dd7319a1e201348342916640db9db5294a8b6cab6692860ac7d136da9441390aa7f1982830543450725944dbe59fcba3a5795c7c31f6
+  languageName: node
+  linkType: hard
+
+"yargs@npm:^17.2.1, yargs@npm:^17.7.2":
   version: 17.7.2
   resolution: "yargs@npm:17.7.2"
   dependencies:
-    cliui: ^8.0.1
-    escalade: ^3.1.1
-    get-caller-file: ^2.0.5
-    require-directory: ^2.1.1
-    string-width: ^4.2.3
-    y18n: ^5.0.5
-    yargs-parser: ^21.1.1
-  checksum: 73b572e863aa4a8cbef323dd911d79d193b772defd5a51aab0aca2d446655216f5002c42c5306033968193bdbf892a7a4c110b0d77954a7fdf563e653967b56a
+    cliui: "npm:^8.0.1"
+    escalade: "npm:^3.1.1"
+    get-caller-file: "npm:^2.0.5"
+    require-directory: "npm:^2.1.1"
+    string-width: "npm:^4.2.3"
+    y18n: "npm:^5.0.5"
+    yargs-parser: "npm:^21.1.1"
+  checksum: abb3e37678d6e38ea85485ed86ebe0d1e3464c640d7d9069805ea0da12f69d5a32df8e5625e370f9c96dd1c2dc088ab2d0a4dd32af18222ef3c4224a19471576
+  languageName: node
+  linkType: hard
+
+"yargs@npm:^17.6.2":
+  version: 17.6.2
+  resolution: "yargs@npm:17.6.2"
+  dependencies:
+    cliui: "npm:^8.0.1"
+    escalade: "npm:^3.1.1"
+    get-caller-file: "npm:^2.0.5"
+    require-directory: "npm:^2.1.1"
+    string-width: "npm:^4.2.3"
+    y18n: "npm:^5.0.5"
+    yargs-parser: "npm:^21.1.1"
+  checksum: 77e4221b49867d50ce5ded87e91ff11f439b46aa4f01d2116f65402c3ac7dfba937d5bb29d50cecf4acda5aaf848d6ff4facd50b2428098c3990c46d58d5b539
   languageName: node
   linkType: hard

Additionally, these are the changes in my .yarnrc.yml:

diff --git a/.yarnrc.yml b/.yarnrc.yml
index 0642acee15..9f159531dd 100644
--- a/.yarnrc.yml
+++ b/.yarnrc.yml
@@ -1,3 +1,9 @@
+compressionLevel: mixed
+
+defaultSemverRangePrefix: ''
+
+enableGlobalCache: false
+
 httpRetry: 10

 httpTimeout: 60000
@@ -15,14 +21,6 @@ npmScopes:
 patchFolder: ./patches

 plugins:
-  - path: .yarn/plugins/@yarnpkg/plugin-workspace-tools.cjs
-    spec: '@yarnpkg/plugin-workspace-tools'
-  - path: .yarn/plugins/@yarnpkg/plugin-interactive-tools.cjs
-    spec: '@yarnpkg/plugin-interactive-tools'
-  - path: .yarn/plugins/@yarnpkg/plugin-typescript.cjs
-    spec: '@yarnpkg/plugin-typescript'
   - .yarn/plugins/token-auth.cjs

-yarnPath: .yarn/releases/yarn-3.4.1.cjs
-
-defaultSemverRangePrefix: ''
+yarnPath: .yarn/releases/yarn-4.0.1.cjs

---

So the main question is: are all these modifications expected during a migration?

Additional questions:

1. What was the default (implicit) compressionLevel before the upgrade (now it's mixed)

2. Does the mixed compression level make the checksum different (point 4.) ?

To reproduce

N/A

Environment

yarn dlx -q envinfo --preset jest

  System:
    OS: macOS 14.0
    CPU: (10) arm64 Apple M1 Pro
  Binaries:
    Node: 18.18.2 - /private/var/folders/9v/c77m1k_90wl3225q3pl3335w0000gq/T/xfs-45053198/node
    Yarn: 4.0.1 - /private/var/folders/9v/c77m1k_90wl3225q3pl3335w0000gq/T/xfs-45053198/yarn
    npm: 9.8.1 - /opt/homebrew/opt/nvm/versions/node/v18.18.2/bin/npm
    bun: 1.0.7 - ~/.bun/bin/bun

Additional context

No response

[arcanis]

So the main question is: are all these modifications expected during a migration?

Yes

What was the default (implicit) compressionLevel before the upgrade (now it's mixed)

See the changelog:

• compressionLevel now defaults to 0 rather than mixed. It's been proved significantly faster on installs, and the size impact was reasonable enough to change the default. Note that it benefits you even if you use Zero-Installs: as per our tests, a zero-compression is actually easier to handle for Git (you can see by yourself with those examples using compressionLevel: 0 vs compressionLevel: mixed).

  • To avoid making the upgrade too disruptive, Yarn will check whether Zero-Installs are enabled the first time you run yarn install after migrating from 3.6 to 4.0. If you do, it will automatically set the old default (compressionLevel: mixed) in your .yarnrc.yml file. You can then remove it whenever you feel ready to actually change the compression settings.

Does the mixed compression level make the checksum different (point 4.) ?

The different checksums aren't due to the mixed compression level (since it's the same as in 3.6, as we inlined the old default in your project), but just because we upgraded the libzip version.

[akwodkiewicz]

but just because we upgraded the libzip version.

Ok, I was suspecting something like this, but couldn't find info about that. Thanks for the explanation.

So the main question is: are all these modifications expected during a migration?

Yes

The mystery of changed checksums is now resolved 😄 But what about the duplicated resolutions to dependencies? Is it some side-effect of the mixed compression level with a new lib, is it a new Yarn behaviour or is it a bug that "forgets" that some dependencies were already deduped?

[akwodkiewicz]

I just thought of running yarn dedupe after the migration and comparing the diffs.

I'm 99% confident that it deduplicated all the resolutions that were added during the migration. After explicit deduping the lockfile now contains only additions of npm: prefixes, changed checksums and the preamble metadata bump 👍

So it seems that migration from v3 to v4 reverted deduplication of packages I did a couple of days ago on my project.

[yarnbot]

Hi! 👋

It seems like this issue as been marked as probably resolved, or missing important information blocking its progression. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it.

[ezweave]

Why is this closed? The other, related bug is still open... 🤷