guest/net: New implementation of network setup with SLAAC and own DHC… #111

sbrivio-rh · 2024-11-25T16:31:28Z

…P client

The existing implementation has a couple of issues:

it doesn't support IPv6 or SLAAC
it relies on either dhclient(8) or dhcpcd(8), which need a significant amount of time to configure the network as they are rather generic DHCP clients
on top of this, dhcpcd, by default, unless --noarp is given, will spend five seconds ARP-probing the address it just received before configuring it

Replace the IPv4 part with a minimalistic, 73-line DHCP client that just does what we need, using option 80 (Rapid Commit) to speed up the whole exchange.

Add IPv6 support (including IPv4-only, and IPv6-only modes) relying on the kernel to perform SLAAC. Safely avoid DAD (we're the only node on the link) by disabling router solicitations, starting SLAAC, and re-enabling them once addresses are configured.

Instead of merely triggering the network setup and proceeding, wait until everything is configured, so that connectivity is guaranteed to be ready before any further process runs in the guest, say:

$ ./target/debug/muvm -- ping -c1 2a01:4f8:222:904::2
PING 2a01:4f8:222:904::2 (2a01:4f8:222:904::2) 56 data bytes
64 bytes from 2a01:4f8:222:904::2: icmp_seq=1 ttl=255 time=0.256 ms

--- 2a01:4f8:222:904::2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.256/0.256/0.256/0.000 ms

The whole procedure now takes approximately 1.5 to 2 ms (for both IPv4 and IPv6), with the DHCP exchange and configuration taking somewhere around 300-500 µs out of that, instead of hundreds of milliseconds to seconds.

Matching support in passt for option 80 (RFC 4039) and for the DHCP "broadcast" flag (RFC 2131) needs this series:

https://archives.passt.top/passt-dev/[email protected]/

[I'll update this commit message once we have an upstream release with it]

sbrivio-rh · 2024-11-25T16:32:53Z

Supersedes #64. Test passt static builds, along with RPMs and Debian packages (x86 only, sorry) at https://passt.top/builds/latest/x86_64/

crates/muvm/Cargo.toml

crates/muvm/src/guest/net.rs

sbrivio-rh · 2024-11-25T19:09:12Z

Hmm, I can't test this anymore after rebasing to latest upstream. I'm getting one of these two errors ("Failed to create the microVM" about 30% of the times, Failed to execute muvm-server as child process about 70%). These are attempts in a relatively tight loop:

$ ./target/debug/muvm -- /bin/false
Error: Failed to execute `muvm-server` as child process

Caused by:
    No such file or directory (os error 2)
$ ./target/debug/muvm -- /bin/false
Error: Failed to create the microVM

Caused by:
    Invalid argument (os error 22)
$ ./target/debug/muvm -- /bin/false
Error: Failed to create the microVM

Caused by:
    Invalid argument (os error 22)
$ ./target/debug/muvm -- /bin/false
Error: Failed to execute `muvm-server` as child process

Caused by:
    No such file or directory (os error 2)
$ ./target/debug/muvm -- /bin/false
Error: Failed to execute `muvm-server` as child process

Caused by:
    No such file or directory (os error 2)

With RUST_LOG=debug:

[2024-11-25T19:05:58Z ERROR krun] Building the microVM failed: GuestMemoryMmap(UnsortedMemoryRegions)

or:

[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::linux::passthrough] do_lookup: "muvm-server"
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::server] opcode: 3
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::worker] Fs: queue event: 1
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::server] opcode: 1
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::linux::passthrough] do_lookup: "muvm-server"
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::worker] Fs: queue event: 1
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::server] opcode: 1
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::linux::passthrough] do_lookup: "muvm-server"
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::worker] Fs: queue event: 1
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::server] opcode: 15
[2024-11-25T19:08:04Z DEBUG devices::virtio::fs::linux::passthrough] read: 56
Error: [2024-11-25T19:08:04Z DEBUG devices::virtio::fs::server] opcode: 25
Failed to execute `muvm-server` as child process

I reverted a few commits but I can't seem to get this to work anymore.

sbrivio-rh · 2024-11-25T19:54:47Z

Oh, okay, I didn't pull for a while. It works if I do:

284b520 (HEAD) Revert "Share /dev/shm as a separate mount with DAX"
44ef0b6 Revert "Protect muvm-server with a cookie"
cdf7027 Revert "Start a privileged muvm-server"
df6c549 Revert "Implement a host memory monitor"

...posting another version of that commit with two functions taken out of configure_network() now that I can test things. Those other issues I'm facing, I have no idea where to start debugging them...

sbrivio-rh · 2024-11-25T22:15:26Z

$ ./target/debug/muvm -- /bin/false
Error: Failed to execute `muvm-server` as child process

Caused by:
    No such file or directory (os error 2)

Fixed by #112

$ ./target/debug/muvm -- /bin/false
Error: Failed to create the microVM

Caused by:
    Invalid argument (os error 22)

Fixed by updating libkrun.

sbrivio-rh · 2024-11-27T13:28:09Z

This is now supported by passt 2024_11_27.c0fbc7e, matching Fedora updates passt-0^20241127.gc0fbc7e-1.fc40, passt-0^20241127.gc0fbc7e-1.fc41, passt-0^20241127.gc0fbc7e-1.fc42, as well as Debian's passt-0.0~git20241127.c0fbc7e-1.

slp · 2024-11-28T10:08:07Z

Thanks a lot @sbrivio-rh , I really like this approach. A couple questions:

What should we do with resolv.conf? Perhaps I'm missing it, but doesn't seem like it's doing nameserver resolution.
This introduces a dependency on neli, but it's not yet packaged into Fedora. Do you want to package it yourself?

sbrivio-rh · 2024-11-28T10:23:33Z

What should we do with resolv.conf? Perhaps I'm missing it, but doesn't seem like it's doing nameserver resolution.

Ah, right, I thought resolv.conf could simply be the one from the host, but with systemd-resolved it's common to have a loopback address as resolver, and that wouldn't work.

So, yes, I should also configure resolv.conf here from DHCP options (6 and 119) and NDP (RDNSS, option 25), passt already sends the right ones. Let me fix this.

This introduces a dependency on neli, but it's not yet packaged into Fedora. Do you want to package it yourself?

Oops, I didn't check. I'm not exactly the right person as I barely understand what a crate is (do I?) and I don't even use Fedora regularly, but yes, I can probably do that, it should be low effort.

I found this abandoned Copr by the way, https://copr.fedorainfracloud.org/coprs/zurdo/i3status-rs-update/package/rust-neli/, it looks pretty easy. Let me give it a try, but if you know somebody else who could be interested...

sbrivio-rh · 2024-11-28T12:31:10Z

This introduces a dependency on neli, but it's not yet packaged into Fedora. Do you want to package it yourself?

Naive question: if it's statically linked, does it really become a dependency? Or is it just a build dependency? Does that matter also if the crate is downloaded as needed...?

slp · 2024-11-28T12:36:52Z

This introduces a dependency on neli, but it's not yet packaged into Fedora. Do you want to package it yourself?

Naive question: if it's statically linked, does it really become a dependency? Or is it just a build dependency? Does that matter also if the crate is downloaded as needed...?

It's just a build dependency. In Fedora, every crate you depend on must be independently packaged, and builds are done offline. Luckily, rust2rpm helps a lot with this.

sbrivio-rh · 2024-11-28T12:47:31Z

It's just a build dependency. In Fedora, every crate you depend on must be independently packaged, and builds are done offline. Luckily, rust2rpm helps a lot with this.

Oh, oops, I just had a look at https://src.fedoraproject.org/user/slp/projects... let me package that. :)

crates/muvm/src/guest/net.rs

sbrivio-rh · 2024-11-28T18:38:33Z

So, yes, I should also configure resolv.conf here from DHCP options (6 and 119) and NDP (RDNSS, option 25), passt already sends the right ones. Let me fix this.

I just added support for nameservers over DHCP (option 6), omitting for the moment:

handling of the domain search list (option 119) also implemented by passt. It's at least 10-20 LoCs due to domain compression encoding from RFC 1035, and I doubt it's of any use for muvm... we can add it later for completeness anyway but I'd rather fix the current situation first
handling of NDP option 25 (RDNSS), implemented by passt as well, for DNS resolvers over SLAAC. That's also some extra bit of complexity. I think this one makes sense no matter what, for IPv6-only setups, but I'd rather avoid to make this patch explode at the moment

sbrivio-rh · 2024-11-28T18:39:49Z

Oops, I just noticed the cargo fmt -- --check warnings, I thought cargo clippy would be enough. Fixing those as well...

sbrivio-rh · 2024-11-28T18:49:00Z

Oops, I just noticed the cargo fmt -- --check warnings, I thought cargo clippy would be enough. Fixing those as well...

Gosh, the reformatted version looks horrible, with 100 columns that don't fit pretty much anywhere and things wildly misaligned. Is cargo fmt enforced?

crates/muvm/src/guest/net.rs

sbrivio-rh · 2024-11-28T21:40:26Z

It's just a build dependency. In Fedora, every crate you depend on must be independently packaged, and builds are done offline. Luckily, rust2rpm helps a lot with this.

Oh, oops, I just had a look at https://src.fedoraproject.org/user/slp/projects... let me package that. :)

Package reviews:
https://bugzilla.redhat.com/show_bug.cgi?id=2329411
https://bugzilla.redhat.com/show_bug.cgi?id=2329412

sbrivio-rh · 2025-02-03T19:46:48Z

Current status of package dependencies in Fedora:

I'm now the proud maintainer of the rust-neli-proc-macros package
rust-neli: package ready for review at https://bugzilla.redhat.com/show_bug.cgi?id=2329412, no reviewers for the moment
rust-const-str-proc-macro: ready for re-re-review at https://bugzilla.redhat.com/show_bug.cgi?id=2338679, @decathorpe currently assigned
rust-const-str: review at https://bugzilla.redhat.com/show_bug.cgi?id=2330150, but it needs to wait for rust-const-str-proc-macros

decathorpe · 2025-02-03T19:48:24Z

I'll be taking another look, as soon as I'm home from FOSDEM.

sbrivio-rh · 2025-02-18T18:44:40Z

rust-const-str-proc-macro packaged, current status of dependencies in Fedora:

I'm now the proud maintainer of the rust-neli-proc-macros and of the rust-const-str-proc-macro packages
rust-neli: package ready for review at https://bugzilla.redhat.com/show_bug.cgi?id=2329412, no reviewers for the moment
rust-const-str: now that rust-const-str-proc-macro is packaged, I can proceed with this as soon as the latter reaches mirrors, then I'll get back to the review request at https://bugzilla.redhat.com/show_bug.cgi?id=2330150

teohhanhui · 2025-03-16T10:41:03Z

Looks like both packages have finally been approved. 🥳

sbrivio-rh · 2025-03-18T19:08:21Z

Looks like both packages have finally been approved. 🥳

Right! Update:

I'm now the proud maintainer of the rust-const-str package as well, builds are available
I'm also the proud maintainer of the rust-neli package, but unfortunately tests fail on s390x, so builds are not available yet. From: https://kojipkgs.fedoraproject.org//work/tasks/7795/130447795/build.log

test rtnl::test::test_rta_deserialize ... FAILED

I'm currently looking into endianness issues.

sbrivio-rh · 2025-03-18T19:48:45Z

jbaublitz/neli#273 -- feel free to contribute if you have some ideas...

decathorpe · 2025-03-18T22:02:09Z

Sorry about that, usually build failures like that should be caught during package review. My bad :(

sbrivio-rh · 2025-03-18T23:03:59Z

Sorry about that, usually build failures like that should be caught during package review. My bad :(

It's not really a build failure, it's a test failure on big-endian only. How would you catch that during review?

decathorpe · 2025-03-18T23:48:50Z

Sorry about that, usually build failures like that should be caught during package review. My bad :(

It's not really a build failure, it's a test failure on big-endian only. How would you catch that during review?

Well, a test failure is a build, failure, as you've found out. :) Many people (and the fedora-create-review tool, by default) run a scratch build for all architectures when submitting a package for review, which would have caught this.

sbrivio-rh · 2025-03-19T13:02:24Z

So, after further baby-sitting from @decathorpe, I can now finally announce that we have builds (at least for Fedora Rawhide) of all the dependencies added here:

@teohhanhui @slp is that enough to merge this, or do we have to wait for packages for a specific Fedora version...?

teohhanhui · 2025-03-19T18:33:36Z

@sbrivio-rh I think this needs a rebase, so that the CI job can have a fresh run.

sbrivio-rh · 2025-03-19T18:41:48Z

Oops, I saw "No conflicts with base branch" and I thought everything was fine, but now that I'm rebasing I actually see conflicts...

sbrivio-rh · 2025-03-19T19:22:19Z

New pull, new warnings:

$ ./target/debug/muvm bash
Error setting up FEX in binfmt_misc: Failed to find `FEXInterpreter` in PATH
Failed to find or configure FEX, falling back to Box
Error setting up Box in binfmt_misc: Failed to find `box64` in PATH
No emulators were configured, x86 emulation may not work
Using default interface naming scheme 'v255'.

anyway, those are just warnings I'll ignore for now. Things seem to work.

sbrivio-rh · 2025-03-19T19:29:24Z

Reverting 5f9f9d9 fixes those, by the way.

nrabulinski · 2025-03-20T10:38:57Z

From my understanding this is the expected behavior, those warnings will go away once I get around implementing #159. Trying to configure a binfmt_misc handler is a good default for now since the bulk of muvm usage comes from folks running Asahi kernel wanting to run 4K page apps.

sbrivio-rh · 2025-03-20T10:52:02Z

those warnings will go away once I get around implementing #159

It looks like that, yes, I wasn't aware of it, thanks for the pointer and thanks in advance for the implementation.

sbrivio-rh · 2025-03-28T15:38:33Z

All the dependencies are now available in "stable" repositories for Fedora 40, 41, 42 and Rawhide. Is there anything else blocking this?

teohhanhui · 2025-04-09T16:21:30Z

/ping @slp

…P client The existing implementation has a couple of issues: - it doesn't support IPv6 or SLAAC - it relies on either dhclient(8) or dhcpcd(8), which need a significant amount of time to configure the network as they are rather generic DHCP clients - on top of this, dhcpcd, by default, unless --noarp is given, will spend five seconds ARP-probing the address it just received before configuring it Replace the IPv4 part with a minimalistic, 90-line DHCP client that just does what we need, using option 80 (Rapid Commit) to speed up the whole exchange. Add IPv6 support (including IPv4-only, and IPv6-only modes) relying on the kernel to perform SLAAC. Safely avoid DAD (we're the only node on the link) by disabling router solicitations, starting SLAAC, and re-enabling them once addresses are configured. Instead of merely triggering the network setup and proceeding, wait until everything is configured, so that connectivity is guaranteed to be ready before any further process runs in the guest, say: $ ./target/debug/muvm -- ping -c1 2a01:4f8:222:904::2 PING 2a01:4f8:222:904::2 (2a01:4f8:222:904::2) 56 data bytes 64 bytes from 2a01:4f8:222:904::2: icmp_seq=1 ttl=255 time=0.256 ms --- 2a01:4f8:222:904::2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.256/0.256/0.256/0.000 ms The whole procedure now takes approximately 1.5 to 2 ms (for both IPv4 and IPv6), with the DHCP exchange and configuration taking somewhere around 300-500 µs out of that, instead of hundreds of milliseconds to seconds. Configure nameservers received via DHCP option 6 as well: passt already takes care care of translating DNS traffic directed to loopback addresses read from resolv.conf, so we can just write those to resolv.conf in the guest. At least for the moment being, for simplicity, omit handling of option 119 (domain search list), as I doubt it's going to be of much use for muvm. I'm not adding handling of the NDP RDNSS option (25, RFC 8106) either, for the moment, as it involves a second netlink socket subscribing to the RTNLGRP_ND_USEROPT group and listening to events while we receive the first router advertisement. The equivalent userspace tool would be rdnssd(8), which is not called before this change anyway. I would rather add it at a later time instead of making this patch explode. Matching support in passt for option 80 (RFC 4039) and for the DHCP "broadcast" flag (RFC 2131) needs at least passt 2024_11_27.c0fbc7e: https://archives.passt.top/passt-user/20241127142126.3c53066e@elisabeth/ Signed-off-by: Stefano Brivio <[email protected]> Co-authored-by: Teoh Han Hui <[email protected]>

sbrivio-rh requested review from WhatAmISupposedToPutHere, alyssarosenzweig, hoshinolina, slp and teohhanhui as code owners November 25, 2024 16:31

teohhanhui reviewed Nov 25, 2024

View reviewed changes

crates/muvm/Cargo.toml Outdated Show resolved Hide resolved

teohhanhui reviewed Nov 25, 2024

View reviewed changes

crates/muvm/src/guest/net.rs Show resolved Hide resolved

sbrivio-rh force-pushed the main branch 2 times, most recently from ca7ea9c to e1eb9df Compare November 25, 2024 18:42

sbrivio-rh force-pushed the main branch from e1eb9df to 2be0214 Compare November 25, 2024 19:56

sbrivio-rh force-pushed the main branch from 2be0214 to bd1f9e9 Compare November 27, 2024 13:23

slp mentioned this pull request Nov 28, 2024

fall back to 1.1.1.1 if DHCP client doesn't update resolv.conf #110

Open

teohhanhui reviewed Nov 28, 2024

View reviewed changes

crates/muvm/src/guest/net.rs Outdated Show resolved Hide resolved

sbrivio-rh force-pushed the main branch from bd1f9e9 to 68094c0 Compare November 28, 2024 18:33

sbrivio-rh force-pushed the main branch from 68094c0 to ae3208a Compare November 28, 2024 18:47

teohhanhui reviewed Nov 28, 2024

View reviewed changes

crates/muvm/src/guest/net.rs Outdated Show resolved Hide resolved

sbrivio-rh mentioned this pull request Jan 22, 2025

config-libkrunfw_x86_64: Chop chop containers/libkrunfw#85

Merged

mkurz mentioned this pull request Jan 27, 2025

Do not keep resolv.conf asahi-alarm/RootFS#3

Merged

sbrivio-rh mentioned this pull request Mar 19, 2025

test::test_rta_deserialize fails on s390x (on big-endian in general, I suppose) jbaublitz/neli#273

Closed

sbrivio-rh force-pushed the main branch 2 times, most recently from caaf8aa to 2f1e54c Compare March 19, 2025 19:37

mkurz mentioned this pull request Mar 21, 2025

Steam client has no internet asahi-alarm/asahi-alarm#4

Open

teohhanhui force-pushed the main branch from 2f1e54c to d80410b Compare April 9, 2025 16:32

slp merged commit ee486ed into AsahiLinux:main Apr 9, 2025
2 checks passed

slp mentioned this pull request Apr 9, 2025

Autoconfigure network without DHCP #64

Closed

WhatAmISupposedToPutHere mentioned this pull request Apr 10, 2025

Skip using resolvconf (more dhcp/dns fixes) #53

Closed

sbrivio-rh mentioned this pull request Nov 25, 2025

krun: switch to passt-based networking containers/crun#1913

Draft

Uh oh!

guest/net: New implementation of network setup with SLAAC and own DHC… #111

guest/net: New implementation of network setup with SLAAC and own DHC… #111

Uh oh!

Conversation

sbrivio-rh commented Nov 25, 2024

Uh oh!

sbrivio-rh commented Nov 25, 2024

Uh oh!

Uh oh!

Uh oh!

sbrivio-rh commented Nov 25, 2024

Uh oh!

sbrivio-rh commented Nov 25, 2024

Uh oh!

sbrivio-rh commented Nov 25, 2024

Uh oh!

sbrivio-rh commented Nov 27, 2024

Uh oh!

slp commented Nov 28, 2024

Uh oh!

sbrivio-rh commented Nov 28, 2024

Uh oh!

sbrivio-rh commented Nov 28, 2024

Uh oh!

slp commented Nov 28, 2024

Uh oh!

sbrivio-rh commented Nov 28, 2024

Uh oh!

Uh oh!

sbrivio-rh commented Nov 28, 2024

Uh oh!

sbrivio-rh commented Nov 28, 2024

Uh oh!

sbrivio-rh commented Nov 28, 2024

Uh oh!

Uh oh!

sbrivio-rh commented Nov 28, 2024

Uh oh!

sbrivio-rh commented Feb 3, 2025

Uh oh!

decathorpe commented Feb 3, 2025

Uh oh!

sbrivio-rh commented Feb 18, 2025

Uh oh!

teohhanhui commented Mar 16, 2025

Uh oh!

sbrivio-rh commented Mar 18, 2025

Uh oh!

sbrivio-rh commented Mar 18, 2025

Uh oh!

decathorpe commented Mar 18, 2025

Uh oh!

sbrivio-rh commented Mar 18, 2025

Uh oh!

decathorpe commented Mar 18, 2025

Uh oh!

sbrivio-rh commented Mar 19, 2025

Uh oh!

teohhanhui commented Mar 19, 2025

Uh oh!

sbrivio-rh commented Mar 19, 2025

Uh oh!

sbrivio-rh commented Mar 19, 2025

Uh oh!

sbrivio-rh commented Mar 19, 2025

Uh oh!

nrabulinski commented Mar 20, 2025

Uh oh!

sbrivio-rh commented Mar 20, 2025

Uh oh!

sbrivio-rh commented Mar 28, 2025

Uh oh!

teohhanhui commented Apr 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects