Support systemd in containers with podman-style `--systemd` flag by sazzy4o · Pull Request #2785 · containerd/nerdctl

sazzy4o · 2024-02-01T02:49:05Z

Adds support for systemd to nerdctl with a --systemd based on the flag used in podman

Usage:

$ sudo nerdctl run --systemd=always --rm -it registry.hub.docker.com/sazzy4o/build
:systemd
systemd v246.15-1.fc33 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Fedora 33 (Container Image)!

Set hostname to <9eff88d04af6>.
Queued start job for default target Graphical Interface.
[  OK  ] Created slice Slice /system/getty.
[  OK  ] Created slice Slice /system/modprobe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on User Database Manager Socket.
         Starting Rebuild Dynamic Linker Cache...
         Starting Journal Service...
         Starting Create System Users...
[  OK  ] Finished Rebuild Dynamic Linker Cache.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Finished Create System Users.
[  OK  ] Finished Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Rebuild Journal Catalog...
         Starting Network Name Resolution...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Finished Update UTMP about System Boot/Shutdown.
[  OK  ] Finished Rebuild Journal Catalog.
         Starting Update is Completed...
[  OK  ] Finished Update is Completed.
[  OK  ] Reached target System Initialization.
[  OK  ] Started dnf makecache --timer.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting MariaDB 10.4 database server...
         Starting The PHP FastCGI Process Manager...
         Starting Home Area Manager...
         Starting User Login Management...
         Starting Permit User Sessions...
         Starting D-Bus System Message Bus...
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
         Starting The nginx HTTP and reverse proxy server...
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Home Area Manager.
[  OK  ] Started User Login Management.
[  OK  ] Started The PHP FastCGI Process Manager.
[  OK  ] Started The nginx HTTP and reverse proxy server.
[  OK  ] Started MariaDB 10.4 database server.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

Fedora 33 (Container Image)
Kernel 6.5.0-14-generic on an x86_64 (console)

9eff88d04af6 login:

sazzy4o · 2024-02-01T02:50:34Z

 	cmd.Flags().StringSlice("cap-drop", []string{}, "Drop Linux capabilities")
 	cmd.RegisterFlagCompletionFunc("cap-drop", capShellComplete)
 	cmd.Flags().Bool("privileged", false, "Give extended privileges to this container")
+	cmd.Flags().String("systemd", "false", "Enable systemd integration (default: false)")


This differs from the podman flag, which has a default value of true

sazzy4o · 2024-02-01T02:51:24Z

 		return nil, nil, err
 	}
 	cOpts = append(cOpts, restartOpts...)
-	cOpts = append(cOpts, withStop(options.StopSignal, options.StopTimeout, ensuredImage))


Moved. See: https://github.com/containerd/nerdctl/pull/2785/files#diff-2ab6676c895e1ac2caa11b1f39c2b22d51b4c8c8219119492ab124d3de0ff2cdR409

AkihiroSuda · 2024-02-01T02:53:53Z

+	systemdPaths := []string{
+		"/usr/sbin/init",
+		"/sbin/init",
+		"/usr/local/sbin/init",


Is there any image that has /usr/local/sbin/init?

I set these to be the same as podman:

true enables systemd mode only when the command executed inside the container is systemd, /usr/sbin/init, /sbin/init or /usr/local/sbin/init.

(But, most images I have seen use /sbin/init)

https://docs.podman.io/en/latest/markdown/podman-run.1.html#systemd-true-false-always

AkihiroSuda · 2024-02-01T02:54:09Z

+
+	systemdPaths := []string{
+		"/usr/sbin/init",
+		"/sbin/init",


Wondering if /sbin should have higher priority over /usr/sbin

Yeah, /sbin/init is more common. I'll move higher

sazzy4o · 2024-02-01T02:54:39Z

+
+			// See: https://github.com/containers/podman/issues/15878
+			if !privilegedWithoutHostDevices {
+				return nil, nil, errors.New("If --privileged is used with systemd `--security-opt privileged-without-host-devices` must also be used")


In the podman implementation all /dev/tty* devices are unmounted to prevent causing host to crash

I could not find an easy to achieve this, so instead I return an error (also prevents causing host to crash). If that functionality is required, maybe it can be added in a future PR

AkihiroSuda · 2024-02-01T02:54:44Z

 	"path"
 	"path/filepath"
 	"runtime"
+	"slices"


This doesn't work with Go 1.20.
I guess we can drop the support for Go 1.20 though.

Switch to using or statement to maintain compatibility with 1.20

AkihiroSuda · 2024-02-01T02:55:18Z

 - :whale: `--cap-add=<CAP>`: Add Linux capabilities
 - :whale: `--cap-drop=<CAP>`: Drop Linux capabilities
 - :whale: `--privileged`: Give extended privileges to this container
+- :nerd_face: `--systemd=(true|false|always)`: Enable systemd compatibility (default: false).


How does always differ from true?

You may also want to update https://github.com/containerd/nerdctl?tab=readme-ov-file#features-present-in-nerdctl-but-not-present-in-docker

Added some more to docs about the options and added a note to nerdctl specific features in README:
https://github.com/containerd/nerdctl/pull/2785/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R206

sazzy4o · 2024-02-01T02:57:05Z

+				{Type: "tmpfs", Source: "tmpfs", Destination: "/var/lib/journal"},
+			}),
+		)
+		stopSignal = "SIGRTMIN+3"


SIGTERM causes restart in systemd (This functionality is the same as podman)

See:
https://www.freedesktop.org/software/systemd/man/latest/systemd.html#Signals

AkihiroSuda · 2024-02-01T02:57:37Z

Thanks,

please add tests
please squash commits

sazzy4o · 2024-02-01T02:59:05Z

+	} else if len(ensured.ImageConfig.Cmd) > 0 {
+		entrypointPath = ensured.ImageConfig.Cmd[0]
+	}
+


There might be an easier way to determine the entrypoint executable path, if there is I am open to updating

sazzy4o · 2024-02-04T03:25:20Z

 done
 echo "signal quit"`).AssertOK()
 	base.Cmd("stop", testContainerName).AssertOK()
+	base.Cmd("inspect", "--format", "{{json .Config.Labels}}", testContainerName).AssertOutContains("SIGQUIT")


Saw that this test was not checking container labels, so added assert

sazzy4o · 2024-02-04T03:26:23Z

 	DockerAuthImage             = mirrorOf("cesanta/docker_auth:1.7")
 	FluentdImage                = mirrorOf("fluent/fluentd:v1.14-1")
 	KuboImage                   = mirrorOf("ipfs/kubo:v0.16.0")
+	SystemdImage                = "ghcr.io/containerd/stargz-snapshotter:0.15.1-kind"


Those this image since it has a working systemd and is controlled by containerd

(with --systemd flag) Signed-off-by: Spencer von der Ohe <s.vonderohe40@gmail.com>

sazzy4o · 2024-02-04T15:54:10Z

@AkihiroSuda I have added some tests and squashed the commits. Could you please have another look?

AkihiroSuda

Thanks

jfernandez · 2024-11-10T18:58:32Z

+		}
+
+		opts = append(opts,
+			oci.WithoutMounts("/sys/fs/cgroup"),


@sazzy4o I found your change while looking into supporting containers with systemd using k8s + containerd. I did the tmpfs mounts for /run, /tmp/, etc., but I was mounting the host's /sys/fs/cgroup as ready-only, which didn't work.

Here, you are removing the mount, which caught my attention. Is this so that systemd creates /sys/fs/cgroup when it initializes?

@jfernandez Yes, this allow systemd to run inside the container and create /sys/fs/cgroup

This was based on the podman --systemd flag

sazzy4o commented Feb 1, 2024

View reviewed changes

AkihiroSuda reviewed Feb 1, 2024

View reviewed changes

sazzy4o commented Feb 1, 2024

View reviewed changes

AkihiroSuda reviewed Feb 1, 2024

View reviewed changes

Comment thread cmd/nerdctl/container_run.go Outdated

sazzy4o commented Feb 1, 2024

View reviewed changes

AkihiroSuda added this to the v2.0.0 milestone Feb 1, 2024

AkihiroSuda added the enhancement New feature or request label Feb 1, 2024

sazzy4o commented Feb 1, 2024

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

sazzy4o commented Feb 4, 2024

View reviewed changes

sazzy4o force-pushed the main branch 5 times, most recently from 5cd85bf to ed090fb Compare February 4, 2024 06:18

Add support to run systemd in containers

741d06e

(with --systemd flag) Signed-off-by: Spencer von der Ohe <s.vonderohe40@gmail.com>

sazzy4o force-pushed the main branch from ed090fb to 741d06e Compare February 4, 2024 06:30

sazzy4o requested a review from AkihiroSuda February 4, 2024 15:54

AkihiroSuda approved these changes Feb 5, 2024

View reviewed changes

AkihiroSuda merged commit 9b76bcc into containerd:main Feb 5, 2024

jfernandez reviewed Nov 10, 2024

View reviewed changes

Conversation

sazzy4o commented Feb 1, 2024

Uh oh!

sazzy4o Feb 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sazzy4o Feb 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sazzy4o Feb 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Feb 1, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sazzy4o commented Feb 4, 2024

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

jfernandez Nov 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sazzy4o Feb 1, 2024 •

edited

Loading

sazzy4o Feb 1, 2024 •

edited

Loading

sazzy4o Feb 1, 2024 •

edited

Loading

jfernandez Nov 10, 2024 •

edited

Loading