ci: use deterministic allocator for benchmarks

[overlookmotel]

An attempt to reduce variance in benchmarks. Just testing at this stage.

[overlookmotel]

• ci: use deterministic allocator for benchmarks #4483 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on Graphite

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[codspeed-hq]

CodSpeed Performance Report

Merging #4483 will improve performances by 11.22%

_{Comparing 07-26-ci_use_deterministic_allocator_for_benchmarks (70e6c9c) with main (d8c2a83)}

Summary

⚡ 7 improvements
✅ 25 untouched benchmarks

Benchmarks breakdown

	Benchmark	main	07-26-ci_use_deterministic_allocator_for_benchmarks	Change
⚡	codegen[checker.ts]	18.3 ms	16.4 ms	+11.22%
⚡	codegen_sourcemap[checker.ts]	75 ms	67.9 ms	+10.46%
⚡	semantic[RadixUIAdoptionSection.jsx]	99.3 µs	92.5 µs	+7.34%
⚡	semantic[cal.com.tsx]	46.3 ms	44.3 ms	+4.39%
⚡	sourcemap[cal.com.tsx]	54.7 ms	51.1 ms	+7.05%
⚡	transformer[RadixUIAdoptionSection.jsx]	224.1 µs	217.5 µs	+3.02%
⚡	transformer[cal.com.tsx]	62.4 ms	60.2 ms	+3.74%

[overlookmotel]

This does seem to work in that it removes almost all variance from the benchmarks.

Results for 5 dummy commits on #4485 which is on top of this PR:

Remaining problem is that changing the allocator is showing as quite a significant speed-up on many benchmarks. Probably this allocator is faster than system allocator! Need to slow it down somehow so it matches system allocator better.

[Boshen]

Need to slow it down somehow so it matches system allocator better.

Probably doesn't matter, we are interested in relative change, not the absolute value from codspeed.

[overlookmotel]

Probably doesn't matter, we are interested in relative change, not the absolute value from codspeed.

Problem is that I think it will give us unrealistic relative results too.

Let's say we introduce a change that removes a bunch of allocations, but requires some extra work to do that (caching structs, bookkeeping etc). That is very likely to be a performance gain in real world, but if the allocator we use for benchmarks makes allocation unrealistically cheap (as this one does), benchmarks will lie to us and tell us it's a perf regression. For example, with this allocator, benchmarks probably would have told us #4213 was a perf regression, whereas in fact it gave +5% speed up.

NB: Allocations are a small part of the code overall. So if we're seeing 10% perf boost on some benchmarks from replacing the allocator, probably that means this new allocator is ~double the speed of the system one. That's a very big discrepancy.

We might have more luck with https://crates.io/crates/dlmalloc which it sounds like is a closer analogue to the default system allocator (from libc), and so may reduce this discrepancy. I'll look into that.

[overlookmotel]

Have asked for help on CodSpeed Discord: https://discord.com/channels/1065233827569598464/1065686090452828251/threads/1267428778691399690

[Boshen]

Merge or close?

[overlookmotel]

Close!

Have opened an issue in backlog with findings, so we can come back to this later. oxc-project/backlog#89