Wild results, cannot reproduce

I have tested your epoll and io_uring examples and I get 250k (req/sec) with your epoll example and only 220k with io_uring. I also get 250k with my own epoll implementation so that confirms we are both using efficient use of epoll.

I'm running on Linux 5.7 Clear Linux - do you have any hints on how I can reproduce your results?

When I strace your example I get a lot of these:

io_uring_enter(4, 59, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 59
io_uring_enter(4, 61, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 61
io_uring_enter(4, 59, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 59
io_uring_enter(4, 61, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 61
io_uring_enter(4, 59, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 59
io_uring_enter(4, 61, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 61
io_uring_enter(4, 59, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 59
io_uring_enter(4, 61, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 61
io_uring_enter(4, 59, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 59
io_uring_enter(4, 61, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 61

Isn't it supposed to poll without any syscalls to maximize the performance advantage?

I absolutely cannot reproduce your results. I've tested on a completely different machine, with a Fedora Rawhide system with Linux 5.8, mitigations for spectre default. This machine confirms my findings from above; io_uring (this example at least) is not faster than epoll.

I get consistently worse results with io_uring, 227k vs. 235k and the like. I have no idea where you would get 99% performance increase with io_uring like your reddit post claims, and not even 45%. I only get worse results, with or without spectre mitigations. I really cannot reproduce the findings you claim.

[frevib]

Hi Alex,

The 99% increase was with an early version of io_uring and buggy version of liburing. Besides, the performance test tool used had some bugs as well: #2 (comment)

So these early results were quite off.

If you use this latest echo server and have support for IORING_FEAT_FAST_POLL, you should in most cases get a peformance increase: https://twitter.com/hielkedv/status/1234135064323280897?s=21

This was at the time using Ubuntu + Linux 5.6 + Jens’ IORING_FEAT_FAST_POLL branch. Current state of io_uring and liburing could have changed the performance. However, using IORING_FEAT_FAST_POLL saves a syscall so theoretically it should be faster.

The test I ran was reporting support of IORING_FEAT_FAST_POLL and yes, I agree that in theory it should be faster since there are a lot fewer syscalls. But in practice, it's not. It even goes the wrong way -> the more clients I add the more the advantage goes for epoll, which is the opposite one would expect since you have more syscalls the more clients you have (epoll_wait, (recv, send) * N).

But I don't care what the theory says, this looks similar to when everyone and their mother was telling me writev was so much faster because it was scatter/gather, but in my testing it was slower than just copying up to 4kb to an append-buffer and write:ing that copy off with the old send syscall.

@axboe Do you have any benchmarks of your own in this regard?

So these early results were quite off.

If you use this latest echo server and have support for IORING_FEAT_FAST_POLL, you should in most cases get a peformance increase

I have run the latest version many times on many different machines and kernels and it does not perform better than epoll.

[frevib]

This echo server also uses IORING_OP_PROVIDE_BUFFERS, which causes a performance drop: https://twitter.com/hielkedv/status/1255492941960949760?s=21

You could try running it without automatic buffer selection, using your own buffer management implementation or no buffer management at all.

I don't have the time to tinker with your code right now - I was hoping to simply confirm or deny your findings, using your provided code (as per anything scientific). So far I cannot confirm it, your findings, and it seems you hint of it not really being possible right now.

I would like to see an example that does in fact prove the efficiency of io_uring over epoll.

[frevib]

The benchmarks provided are without buffer selection: https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md

So what commit should I use? You didn't tag any release so I can only check out master.

[Mathnerd314]

My guess is you'd use this branch: https://github.com/frevib/io_uring-echo-server/tree/io-uring-feat-fast-poll, like he linked in the benchmark description.

[frevib]

This could very well the branch, it’s indeed without buffer selection but with fast poll. I haven’t had the time to look for the exact commit hash.

[frevib]

Maybe try this one: https://github.com/frevib/io_uring-echo-server/tree/b003989ecb6343b5815999527310251601531acc

This commit is right before buffer selection was implemented.

[Qix-]

I'm with @alexhultman on this. I simply cannot reproduce anything close to the wild, game-changing claims others have made.

I allocated a 4 vcpu machine on a google cloud instance. Seeing as how the bechnmarks were run on a VM on with a MacOS host I would imagine this (should) not affect the benchmark so much that epoll is just as fast.

Note: I did test this locally on a Windows host running VMWare player with an Ubuntu guest, but I did not isolate the CPUs so I didn't consider them here. However, I got the exact same results - epoll and uring are neck and neck, all things considered.

I upgraded to mainline 5.8.12-050812-generic, set isolcpus=0,1 and told systemd to affine to 2 and 3. Rebuilt GRUB, rebooted, and ran the benchmarks using taskset -c 0|1 <cmd> <args...>.

After a few runs, I got very mixed results, but they were all within the same range. Sometimes uring achieved more throughput, sometimes epoll. Never with a spread with more than about 2k requests a second, however.

I tried both the io-uring-feat-fast-poll branch and the direct commit in #8 (comment). Both segfault for me upon boot.

Tests were run with a variety of parameters to the echo server benchmark tool. All of the spreads were roughly the same - +/- 2k req/sec with epoll and uring alternating between the winners over a 60 second period.

I'm beginning to think this is another case of "too good to be true". Just my take.

---

Further, the linked tweet above is quoting the same benchmark wiki here claiming 68% increase. Sorry, but that's misleading and simply wrong.

[frevib]

To understand these performance gains that the echo server is claiming, maybe some extra context is needed.

• When io_uring just got out in Kernel 5.1, I used a version of liburing and the testing tool that contained several bugs. This caused a massively overstated performance gain, and was fixed here: Code cleanup and fix #1 #2 (comment). After this fix, io_uring was only a few percentages faster than epoll.

• Then came IORING_FEAT_FAST_POLL, which made polling unnecessary and therefore could avoid syscalls to poll for completed IO. The maximum performance increase I got is indeed 68%, using 500 connections with 512 bytes message size. If you use 1 connection and 128 bytes the performance increase is minimal. Using > 500 connections and message sizes > 1000 bytes also decreases the performance. As Any io_uring performance tests after all of the work that has been done? netty/netty#10622 (comment) also states, it seems io_uring prefers lots of simultaneous connections. The trade off here is that you have to manage the buffers that you give to the sockets yourself.

• Then came IORING_OP_PROVIDE_BUFFERS, which could automatically manage buffers for you. This however meant that you have to reassign a "used" buffer each time a buffer has been used. This means the iterations in the event loop doubled, and caused a performance drop. However, in many connections size/message size configs io_uring is still a bit faster.

Of course echo server is not a real use case, so have a look at some of the "real" implementations like Netty or NodeJS.