readFile in promises very slow

[xJeneKx]

Version

v21.7.1

Platform

Darwin evgeniis.local 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:43 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6000 arm64

Subsystem

No response

What steps will reproduce the bug?

slow

const fs = require('fs/promises');

const start = Date.now();
let count = 0;
for (let i = 0; i < 10000; i++) {
	fs.readFile("./text.txt", { encoding: 'utf-8' })
		.then((data) => {
			if (data !== "Hello, world") throw 1;
			count++
			if (count === 10000) {
				console.log('time: ', Date.now() - start);
			}
		})
		.catch((err) => {
			throw 1
		})
}

fast

const fs = require('fs');
const util = require('util');
const readFile = util.promisify(fs.readFile);

const start = Date.now();
let count = 0;
for (let i = 0; i < 10000; i++) {
	readFile("./text.txt", { encoding: 'utf-8' })
		.then((data) => {
			if (data !== "Hello, world") throw 1;
			count++
			if (count === 10000) {
				console.log('time: ', Date.now() - start);
			}
		})
		.catch((err) => {
			throw 1
		})
}

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

No response

What do you see instead?

The promise version is 2 times slower. My tests showed 200ms vs 100ms

Additional information

No response

[AlttiRi]

The same on Windows.

However, I need to decrease the count from 10000 to 8189.
There is the error when I use 8190+:

[Error: EMFILE: too many open files, open 'C:\test\text.txt'] {
  errno: -4066,
  code: 'EMFILE',
  syscall: 'open',
  path: 'C:\\test\\text.txt'
}
Node.js v18.18.1

[UlisesGascon]

I suggest to transfer this issue to the performance team repo... what do you think @anonrig?

[lemire]

You can go faster (maybe 2x faster) by going synchronous.

const fs = require('fs');
const readFileSync = fs.readFileSync;

const start = Date.now();
let count = 0;
for (let i = 0; i < 10000; i++) {
        var data = readFileSync("./text.txt", { encoding: 'utf-8' })
}
console.log('time: ', Date.now() - start)

When benchmarking using bun, all three approaches run at the same speed.

Here bun vs node on the first benchmark:

>  node  test.js
time:  218
>  bun run test.js
time:  53

The cost of promises in Node seems extraordinarily high.

[lemire]

You can basically recover the performance by calling readFileSync within an async function...

const fs = require('fs');
const readFileSync = fs.readFileSync;
async function f(name, options) {
  return readFileSync(name, options);
}

const start = Date.now();
let count = 0;
for (let i = 0; i < 10000; i++) {
        f("./text.txt", { encoding: 'utf-8' })
                .then((data) => {
                        count++
                        if (count === 10000) {
                                console.log('time: ', Date.now() - start);
                        }
                })
                .catch((err) => {
                        throw 1
                })
}

[meyfa]

You can basically recover the performance by calling readFileSync within an async function

That cannot be optimal? Maybe in this synthetic example it is, but won't readFileSync just block the whole program until the entire file is read? The await in this case just gives the illusion of being async as a microtick is introduced after the entire file has been read synchronously...

[meyfa]

readFileSync has to return the file contents synchronously, so it must read the whole file before returning. And since JS isn't multi-threaded this will block execution. fs.promises.readFile can allow other stuff to run while it fetches the file asynchronously, fulfilling the returned promise when done. The presence of await or the fact that the readFileSync call happens in an async function does not affect its internal behavior.

While readFileSync may speed up the benchmark, I think it is unsuitable for applications that could be doing other work while waiting for I/O to complete.

[aduh95]

You can basically recover the performance by calling readFileSync within an async function

That's going to depend on the size of your file and if you're program has other things to do – like @meyfa said, using readFileSync might get you the content of the file slightly faster, but the tradeoff is that your program cannot do anything in the mean time.

[GeoffreyBooth]

JS isn’t multi-threaded

https://nodejs.org/api/worker_threads.html

[ronag]

I think we are getting a bit off topic here... I would recommend all mentions of readFileSync should be hidden as off topic.

[lemire]

I ran some benchmarks on a large server. See blog post.

[benjamingr]

@Linkgoron

[benjamingr]

Yeah last time we looked at it there were many opportunities to make the slower discrete versions (e.g readFile and not createReadStream) much much faster so this result isn’t too surprising.

[lemire]

I think we are getting a bit off topic here... I would recommend all mentions of readFileSync should be hidden as off topic.

In Bun, fs.readFileSync can be slower than fs.promises, and fs.readFile definitively appears to be able to be faster than fs.readFileSync. In Node.js, the opposite is true. See my numbers above. Granted, it is one test on one system, and one's results will vary.

Is that what one expects?

[lemire]

@aduh95

the tradeoff is that your program cannot do anything in the mean time.

In my test (see blog post link), readFileSync uses 300 ms of CPU time, fs.promises uses 7 seconds of CPU time (over several cores). And fs.promises return the file with significantly higher latency.

These are not small differences.