[BUG] npm behaves inconsistent when fetching package data vs downloading packages (tgz)

[bt-nia]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

npm seems to behave differently during meta data fetching and downloading the actual packages, when it comes to URLs that have authentication data already in them.

In my environment, I need to configure npm with environment variables. To replace the registry with our internal one, I'm setting the environment variable NPM_CONFIG_REGISTRY="https://publicbot:PASS@redacted.host/artifactory/api/npm/public-npm"

Here's a partial output of the command npm install -d --no-cache @metamask/eth-sig-util:

npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@noble%2fsecp256k1 464ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@noble%2fhashes 480ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/strip-hex-prefix 460ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@chainsafe%2fssz 510ms (cache miss)
npm timing idealTree:node_modules/@metamask/eth-sig-util Completed in 621ms
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@chainsafe%2fas-sha256 243ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@chainsafe%2fpersistent-merkle-tree 249ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/case 256ms (cache miss)
npm timing idealTree:node_modules/@ethereumjs/util Completed in 303ms
npm timing idealTree:node_modules/@chainsafe/ssz Completed in 26ms
npm timing idealTree:node_modules/@chainsafe/as-sha256 Completed in 0ms
npm timing idealTree:node_modules/@chainsafe/persistent-merkle-tree Completed in 0ms
npm timing idealTree:node_modules/@ethereumjs/rlp Completed in 0ms
npm timing idealTree:node_modules/bn.js Completed in 0ms
npm timing idealTree:node_modules/case Completed in 0ms
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@scure%2fbase 313ms (cache miss)
npm timing idealTree:node_modules/ethereum-cryptography Completed in 383ms
npm timing idealTree:node_modules/@noble/hashes Completed in 2ms
npm timing idealTree:node_modules/@noble/secp256k1 Completed in 0ms
npm timing idealTree:node_modules/@scure/bip32 Completed in 26ms
npm timing idealTree:node_modules/@scure/base Completed in 0ms
npm timing idealTree:node_modules/@scure/bip39 Completed in 0ms
npm timing idealTree:node_modules/ethjs-util Completed in 19ms
npm timing idealTree:node_modules/is-hex-prefixed Completed in 0ms
npm timing idealTree:node_modules/strip-hex-prefix Completed in 0ms
npm timing idealTree:node_modules/tweetnacl Completed in 0ms
npm timing idealTree:node_modules/tweetnacl-util Completed in 0ms
npm timing idealTree:buildDeps Completed in 2718ms
npm timing idealTree:fixDepFlags Completed in 11ms
npm timing idealTree Completed in 2845ms
npm timing reify:loadTrees Completed in 2849ms
npm timing reify:diffTrees Completed in 14ms
npm timing reify:retireShallow Completed in 10ms
npm timing reify:createSparse Completed in 26ms
npm timing reify:loadBundles Completed in 0ms
npm http fetch POST 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/-/npm/v1/security/advisories/bulk 482ms
npm timing auditReport:getReport Completed in 517ms
npm timing auditReport:init Completed in 0ms
npm timing reify:audit Completed in 522ms
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/strip-hex-prefix/-/strip-hex-prefix-1.0.0.tgz 329ms (cache skip)
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/is-hex-prefixed/-/is-hex-prefixed-1.0.0.tgz 353ms (cache skip)
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/@scure/bip32/-/bip32-1.1.5.tgz 339ms (cache skip)
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/@noble/secp256k1/-/secp256k1-1.7.1.tgz 336ms (cache skip)
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/@scure/base/-/base-1.1.1.tgz 332ms (cache skip)
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/@scure/bip39/-/bip39-1.1.1.tgz 330ms (cache skip)
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/@noble/hashes/-/hashes-1.2.0.tgz 332ms (cache skip)
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/ethjs-util/-/ethjs-util-0.1.6.tgz 388ms (cache skip)
npm http fetch GET 401 https://redacted.host/artifactory/api/npm/public-npm/ethereum-cryptography/-/ethereum-cryptography-1.2.0.tgz 440ms (cache skip)

Once the process goes into fetching actual packages, all requests are not performed with the authentication data inside the URL anymore (hence 401 errors). Instead, I can only make it work if I also set the NPM_CONFIG__AUTH variable to the username and password in base64 form (for npm 8, this will not work in npm 9), which should not be required.

In npm 9, setting NPM_CONFIG__AUTH does not solve this issue, but the same inconsistency can be observed.

Expected Behavior

This is how I would expect the output to be (for both npm 8 and 9) without having to set the NPM_CONFIG__AUTH environment variable:

npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@scure%2fbip32 373ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/strip-hex-prefix 362ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@noble%2fsecp256k1 389ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@noble%2fhashes 396ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@chainsafe%2fssz 423ms (cache miss)
npm timing idealTree:node_modules/@metamask/eth-sig-util Completed in 532ms
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@chainsafe%2fas-sha256 237ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@chainsafe%2fpersistent-merkle-tree 245ms (cache miss)
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/case 254ms (cache miss)
npm timing idealTree:node_modules/@ethereumjs/util Completed in 295ms
npm timing idealTree:node_modules/@chainsafe/ssz Completed in 27ms
npm timing idealTree:node_modules/@chainsafe/as-sha256 Completed in 0ms
npm timing idealTree:node_modules/@chainsafe/persistent-merkle-tree Completed in 0ms
npm timing idealTree:node_modules/@ethereumjs/rlp Completed in 0ms
npm timing idealTree:node_modules/bn.js Completed in 0ms
npm timing idealTree:node_modules/case Completed in 0ms
npm http fetch GET 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/@scure%2fbase 270ms (cache miss)
npm timing idealTree:node_modules/ethereum-cryptography Completed in 642ms
npm timing idealTree:node_modules/@noble/hashes Completed in 0ms
npm timing idealTree:node_modules/@noble/secp256k1 Completed in 0ms
npm timing idealTree:node_modules/@scure/bip32 Completed in 11ms
npm timing idealTree:node_modules/@scure/base Completed in 0ms
npm timing idealTree:node_modules/@scure/bip39 Completed in 0ms
npm timing idealTree:node_modules/ethjs-util Completed in 16ms
npm timing idealTree:node_modules/is-hex-prefixed Completed in 1ms
npm timing idealTree:node_modules/strip-hex-prefix Completed in 0ms
npm timing idealTree:node_modules/tweetnacl Completed in 0ms
npm timing idealTree:node_modules/tweetnacl-util Completed in 0ms
npm timing idealTree:buildDeps Completed in 3337ms
npm timing idealTree:fixDepFlags Completed in 11ms
npm timing idealTree Completed in 3457ms
npm timing reify:loadTrees Completed in 3462ms
npm timing reify:diffTrees Completed in 15ms
npm timing reify:retireShallow Completed in 9ms
npm timing reify:createSparse Completed in 58ms
npm timing reify:loadBundles Completed in 0ms
npm http fetch POST 200 https://publicbot:***@redacted.host/artifactory/api/npm/public-npm/-/npm/v1/security/advisories/bulk 445ms
npm timing auditReport:getReport Completed in 477ms
npm timing auditReport:init Completed in 0ms
npm timing reify:audit Completed in 484ms
npm http fetch GET 200 https://redacted.host/artifactory/api/npm/public-npm/strip-hex-prefix/-/strip-hex-prefix-1.0.0.tgz 1315ms (cache miss)
npm timing reifyNode:node_modules/strip-hex-prefix Completed in 1435ms
npm http fetch GET 200 https://redacted.host/artifactory/api/npm/public-npm/is-hex-prefixed/-/is-hex-prefixed-1.0.0.tgz 1312ms (cache miss)
npm timing reifyNode:node_modules/is-hex-prefixed Completed in 1442ms
npm http fetch GET 200 https://redacted.host/artifactory/api/npm/public-npm/tweetnacl-util/-/tweetnacl-util-0.15.1.tgz 1316ms (cache miss)
npm timing reifyNode:node_modules/tweetnacl-util Completed in 1494ms
npm http fetch GET 200 https://redacted.host/artifactory/api/npm/public-npm/case/-/case-1.6.3.tgz 1479ms (cache miss)

Steps To Reproduce

1. In this environment...
2. With this config...
3. Run '...'
4. See error...

Environment

• npm: 8.19.4, 9.5.0
• Node.js: v18.14.1, v18.14.2
• OS Name: nikolaik/python-nodejs:latest
• System Model Name: Docker
• npm config:

; node bin location = /usr/local/bin/node
; node version = v18.14.2
; npm local prefix = /home/user/app
; npm version = 9.5.0
; cwd = /home/user/app
; HOME = /root
; Run `npm config ls -l` to show all defaults.

Note: npm config is always empty, it's a default container.