[Feature] Async iterator over ls.stream

[ccollie]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Calling ls.stream over a large store results large memory hit for all entries

Expected Behavior

An iterator which returns entries as needed

[wraithgar]

I don't think this will make a difference either way, you're just changing how the stream is consumed. We think the real memory hog here is

cacache/lib/content/read.js

Line 11 in e3a2928

const MAX_SINGLE_READ_SIZE = 64 * 1024 * 1024

, and have plans to fix it soon.

[yury-kozlov]

I also stumbled on a similar issue (on windows) when npm cache contains lots of packages, I got this error:

Uncaught Error Error: EMFILE: too many open files,
Emitted 'error' event on Minipass instance at:
at (program) (internal/process/promises:265:12)

The reason for the issue is that cacache\lib\entry-index.js\lsStream function tries to asynchronously load all files and only then awaits the results.
When working with streams I would expect a different behavior: load data in portions (one-by-one or in chunks of X) instead of reading all files "in parallel".

As a workaround, I adjusted the source code locally and used regular for loops instead of those map functions:

• await Promise.all(buckets.map(async (bucket) =>
• await Promise.all(subbuckets.map(async (subbucket) =>
• await Promise.all(subbucketEntries.map(async (entry) =>