[Bug]: Memory usage significantly increases in 30.0.0-beta.6 for certain tests

[anomiex]

Version

30.0.0-beta.9

Steps to reproduce

Repeat the following using 30.0.0-beta.5 and 30.0.0-beta.6:

1. Unzip this example repo: test.zip
2. Install with your preferred package manager.
3. Start monitoring system memory usage, then run jest.

Expected behavior

Memory usage is about the same between 30.0.0-beta.5 and 30.0.0-beta.6.

Actual behavior

With 30.0.0-beta.5, free memory decreases from 19G to 18G. Tests take about 5 seconds wall clock time.

With 30.0.0-beta.6, free memory decreases from 19G to 9.2G. Tests take about 7 seconds wall clock time.

Additional context

My system I used for testing has 32G memory and 16 CPU cores, which seems to mean Jest runs 10 child processes for the 10 test files included in the example repo.

The setTimeout in the tests is not necessary to reproduce, it's just there to make it easier to get memory measurements. It also continues to reproduce if the async is removed along with the awaited promise.

The require of the @wordpress/components package seems to be what's somehow bringing the memory usage way up. I don't know whether other large packages might do the same thing.

Attempting to measure memory usage within node (e.g. with node --inspect-brk) does not seem to reflect the increased memory usage. Or maybe I was just doing it wrong.

Using --runInBand shows a more modest effect: memory usage is still up in beta.6 compared to beta.5, but since it's doing one test at a time it doesn't go up as obviously as it does with 10 workers.

Environment

System:
    OS: Linux 6.12 Debian GNU/Linux 13 (trixie) 13 (trixie)
    CPU: (16) x64 AMD Ryzen 7 7840U w/ Radeon  780M Graphics
  Binaries:
    Node: 22.16.0 - /usr/bin/node
    Yarn: 1.22.22 - /usr/local/bin/yarn
    npm: 10.9.2 - /usr/bin/npm
    pnpm: 10.11.1 - /usr/local/bin/pnpm
  npmPackages:
    jest: 30.0.0-beta.6 => 30.0.0-beta.6

[anomiex]

A colleague of mine pointed out #15556 (comment)

jest@30.0.0-beta.6 was just published. It should be faster than beta 5, but the default now uses more memory than beta 5. If you'd like the aggressive memory cleanup behavior from beta.5, add testEnvironmentOptions: { globalsCleanupMode: 'hard' } to your Jest config.

Setting that did not help with the issue here.

[cpojer]

I'm a bit confused, in your issue you are saying that beta 5 reduces memory to 18G, and beta 6 reduces it to 9.2G:

With 30.0.0-beta.5, free memory decreases from 19G to 18G. Tests take about 5 seconds wall clock time.
With 30.0.0-beta.6, free memory decreases from 19G to 9.2G. Tests take about 7 seconds wall clock time.

So.. what is the issue?

[anomiex]

Sorry, I guess that was confusing. Free memory decreases. In other words, beta.5 uses about 1G, while beta.6 uses about 10G.

[cpojer]

@anomiex did you run pnpm approve-builds to approve the unrs-resolver build (I know, this pnpm feature is super annoying).

@JounQin the major difference between beta 5 and 6 is unrs-resolver from #15619. Would you be able to take a look at this?

[JounQin]

Maybe you should also do a benchmark for the testing time between those two versions? The memory is there to be used. I didn't notice any downside performance issue when using unrs-resolver with other projects neither.

And please try to upgrade unrs-resolver at the same time, https://github.com/unrs/unrs-resolver/releases/tag/v1.7.10 bumps napi which could also improve the memory usage.

[anomiex]

did you run pnpm approve-builds to approve the unrs-resolver build (I know, this pnpm feature is super annoying).

Did that now, it makes no difference.

AFAICT the postinstall script does nothing when the appropriate dependency (e.g. @unrs/resolver-binding-linux-x64-gnu) is installed anyway.

[anomiex]

Maybe you should also do a benchmark for the testing time between those two versions?

Slowed it down by about 2 seconds with this test repo.

The memory is there to be used. I didn't notice any downside performance issue when using unrs-resolver with other projects also.

"Memory is there to be used", but OOM-ing the system isn't a good idea.

And please try to upgrade unrs-resolver at the same time, https://github.com/unrs/unrs-resolver/releases/tag/v1.7.10 bumps napi which could also improve the memory usage.

Already at 1.7.10.

[cpojer]

Could you run your tests with --logHeapUsage and report the results of two or so runs each here?

[anomiex]

As I mentioned,

Attempting to measure memory usage within node (e.g. with node --inspect-brk) does not seem to reflect the increased memory usage. Or maybe I was just doing it wrong.

The extra memory usage seems to be outside of what node itself can see.

With beta.5

 PASS  tests/test07.test.js (61 MB heap size)
 PASS  tests/test06.test.js (65 MB heap size)
 PASS  tests/test01.test.js (61 MB heap size)
 PASS  tests/test09.test.js (5.016 s, 61 MB heap size)
 PASS  tests/test05.test.js (5.035 s, 61 MB heap size)
 PASS  tests/test02.test.js (5.04 s, 61 MB heap size)
 PASS  tests/test04.test.js (5.048 s, 61 MB heap size)
 PASS  tests/test03.test.js (5.058 s, 61 MB heap size)
 PASS  tests/test08.test.js (5.08 s, 61 MB heap size)
 PASS  tests/test10.test.js (5.094 s, 65 MB heap size)

Test Suites: 10 passed, 10 total
Tests:       10 passed, 10 total
Snapshots:   0 total
Time:        5.255 s, estimated 7 s
Ran all test suites.

With beta.6

 PASS  tests/test10.test.js (6.315 s, 57 MB heap size)
 PASS  tests/test08.test.js (6.31 s, 57 MB heap size)
 PASS  tests/test02.test.js (6.406 s, 57 MB heap size)
 PASS  tests/test04.test.js (6.416 s, 57 MB heap size)
 PASS  tests/test03.test.js (6.425 s, 57 MB heap size)
 PASS  tests/test01.test.js (6.494 s, 57 MB heap size)
 PASS  tests/test07.test.js (6.545 s, 57 MB heap size)
 PASS  tests/test06.test.js (6.603 s, 57 MB heap size)
 PASS  tests/test05.test.js (6.653 s, 57 MB heap size)
 PASS  tests/test09.test.js (6.771 s, 57 MB heap size)

Test Suites: 10 passed, 10 total
Tests:       10 passed, 10 total
Snapshots:   0 total
Time:        6.972 s, estimated 7 s
Ran all test suites.

That was with unrs-resolver 1.7.11, BTW.

[anomiex]

Playing around a bit with tcmalloc, I find there's little difference in the usage reports between beta.5 and beta.6 with the default settings. But if I enable HEAP_PROFILE_MMAP=true, with beta.6 there's a new line right at the top of the chart:

  1024.0  78.2%  78.2%   1024.0  78.2% napi_register_module_v1

That indicates an allocation of 1024MB, which is presumably an mmap() since it doesn't show up without HEAP_PROFILE_MMAP.

I find that tcmalloc doesn't seem to be able to know whether mmapped memory is actually used. Even if a process mmaps 10GB, Linux doesn't actually allocate that memory to the process right away. Each page of memory is only allocated when it's used. I've confirmed that the line shows up even if the require( '@wordpress/components' ) is removed from the test (and therefore the high memory usage doesn't occur).

Since "napi" seems related to unrs-resolver, this makes me suspect that unrs-resolver has some sort of 1GB cache (per process), allocated outside of node's heap so it doesn't show up in most measurements. Whether the cache fills up (and therefore actually uses that memory) depends on what modules get loaded.

P.S. A sample tcmalloc command is

HEAPPROFILE=/tmp/some-prefix HEAP_PROFILE_MMAP=true LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_and_profiler.so.4.6.13 /usr/bin/node node_modules/jest/bin/jest.js --runInBand tests/test01.test.js

The google-pprof executable can process the file that will be created at /tmp/some-prefix.0001.heap.

[tbradsha]

I did some basic testing on macOS and notice the same increase in memory usage with the test project provided above:

• beta.5: used about 1-2GB
• beta.6: used about 9-10GB

[cpojer]

@JounQin any thoughts re unrs-resolver in multiple processes possibly using too much memory like @anomiex is pointing to?