A small Linux container runtime in Go, built to learn what happens underneath
Docker: namespaces, pivot_root, cgroups v2, veth networking and OCI images.
Status: all seven milestones of PLAN.md are done (process isolation, the container filesystem, resource limits, networking, lifecycle management, security and images).
Needs Linux on x86-64 (WSL2 works), root, iptables, and Go 1.25 or newer.
make build builds a static binary (CGO_ENABLED=0), which --init needs.
make build
sudo bin/microbox pull alpine:3.20 # or: sudo scripts/get-alpine.sh, a minirootfs as "alpine"
sudo bin/microbox run --rm alpine:3.20 /bin/sh/ # hostname
4f1c9a0b2e7d
/ # ps -o pid,comm
PID COMMAND
1 sh
7 ps
/ # ip -4 addr show eth0 | grep inet
inet 10.88.0.2/16 brd 10.88.255.255 scope global eth0
/ # wget -qO- http://example.com | grep -o '<title>.*</title>'
<title>Example Domain</title>
IMAGE is a pulled image, a root filesystem under /var/lib/microbox/images
(like alpine from get-alpine.sh), or a path to one that starts with / or
./. Without a COMMAND, the image's own runs. Options:
| Flag | Effect |
|---|---|
-d |
run in the background and print the container ID |
--rm |
remove the container when it exits; without it, it stays until microbox rm, as in docker |
--init |
PID 1 is microbox's init, which passes signals on to the command and reaps zombies |
-u, --user USER |
run as USER, UID, USER:GROUP or UID:GID from the image's /etc/passwd (default: the image's user) |
--hostname NAME |
hostname (default: the container ID) |
--memory 256m |
memory.max, with memory.swap.max=0 so the limit can't be dodged by swapping |
--cpus 0.5 |
cpu.max = 50000 100000: 50 ms of CPU time per 100 ms |
--pids 64 |
pids.max: forks beyond it fail |
--net bridge |
the default: own address on the mb0 bridge, NAT to the outside |
--net none |
own network namespace with only lo |
--net host |
the host's network, no isolation |
-p 8080:80 |
publish container TCP port 80 on host port 8080 (repeatable) |
$ sudo bin/microbox run --rm --memory 32m alpine sh -c 'x=$(head -c 100m /dev/zero | tr "\0" a)'
microbox: container reached its memory limit; the OOM killer killed 1 process(es)
$ echo $?
137
Exit codes are the command's own, or 125 for a runtime error, 126 when the command can't be executed, 127 when it isn't found, and 128+N when the container is killed by signal N.
$ sudo bin/microbox pull nginx:alpine
e2de96513ba9 pulling 3.7 MB
d9aae54b5831 pulling 1.8 MB
...
e72112c14215 pulling 19.6 MB
pulled nginx:alpine (3dd08163706a)
$ sudo bin/microbox images
IMAGE ID LAYERS SIZE
nginx:alpine 3dd08163706a 8 25.1 MB
alpine (directory)
$ sudo bin/microbox run -d -p 8080:80 nginx:alpine
a9d4404c5d09
$ curl -s localhost:8080 | grep -o '<title>.*</title>'
<title>Welcome to nginx!</title>
$ sudo bin/microbox exec a9d4 ps -o pid,user,args
PID USER COMMAND
1 root nginx: master process nginx -g daemon off;
30 nginx nginx: worker process
...
$ sudo bin/microbox pull nginx:alpine
nginx:alpine is up to date
Names work as in docker: alpine is alpine:latest on Docker Hub,
bitnami/redis:7 is a user's repository there, and a first part with a dot
or a port is a registry, as in ghcr.io/owner/app:v1. A registry on
localhost or 127.* is spoken to over plain HTTP. rmi IMAGE forgets an
image, and gc then deletes the layers that no image and no existing
container uses.
How pull works (internal/image):
HEAD /v2/<repo>/manifests/<tag>with the four manifest media types (OCI and Docker, index and single-platform) inAccept. The registry answers 401 with aWWW-Authenticate: Bearer realm=...,service=...,scope=...challenge, so microbox first gets an anonymous token from the realm, which is how Docker Hub works even for public images. If the digest the HEAD reports is the one recorded at the last pull, the image is up to date. HEAD requests don't count against Docker Hub's limit on anonymous pulls.GETthe manifest. An index lists one manifest per platform, and microbox takes linux on its own architecture, fetched by its digest.GET /v2/<repo>/blobs/<digest>for the config, which has the command, environment, working directory and user, and for each layer. Registries redirect blob downloads to a CDN, and Go's HTTP client drops theAuthorizationheader when the host changes.- Every download is checked against its sha256 digest and size before anything reads it. Layers go to a temporary file first, so a layer whose digest doesn't match is never unpacked.
- Each layer is unpacked into
/var/lib/microbox/oci/layers/<digest>/, once, however many images share it. All paths go through anos.Rooton that directory (Go 1.25), which refuses any path that resolves outside it, including through a symlink the layer itself created. An entry like../../etc/passwd, or a write through a symlink to/, fails the pull instead of reaching the host. A layer is unpacked into a temporary directory and renamed into place, so a layer directory is always complete. - A tar can only add files, so a layer marks deletions with special
entries, which become what overlayfs reads:
.wh.NAME(NAME was deleted) becomes a character device 0:0 called NAME..wh..wh..opq(hide everything below this directory) becomes thetrusted.overlay.opaque=yxattr on the directory.
- The record goes to
/var/lib/microbox/oci/images/<name>.json.
run then passes the layer directories, top first, as overlayfs lowerdirs
(lowerdir=top:...:bottom). The kernel copies mount options into one page,
which limits an image to about 40 layers. The config becomes the container's
process, as in docker:
- The command is the entrypoint plus the command given on the command line, or else the image's command.
- The environment is the image's, plus
HOSTNAME, andHOMEfrom the user's/etc/passwdentry. - init creates the working directory if needed and changes to it.
- It looks the user up in the image's
/etc/passwdand/etc/group, sets the groups, GID and UID, and execs. A user other than root ends up with no capabilities, as in docker.execruns its command the same way.
Not supported: zstd layers, private registries (no credentials), pulling by digest, and xattrs from layers other than the opaque marker, such as file capabilities.
$ sudo bin/microbox run -d -p 8080:80 alpine sh -c 'while :; do
printf "HTTP/1.0 200 OK\r\nContent-Length: 6\r\n\r\nhello\n" | nc -l -p 80 >/dev/null; echo served; done'
cd6132790b2e
$ sudo bin/microbox ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
cd6132790b2e alpine "sh -c while :; do p…" 0s ago Up 0s 8080->80/tcp
$ curl localhost:8080
hello
$ sudo bin/microbox logs cd6132790b2e
served
$ sudo bin/microbox exec cd61 ps -o pid,comm
PID COMMAND
1 sh
10 nc
12 ps
$ time sudo bin/microbox stop cd61
cd61
real 0m10.135s
$ sudo bin/microbox ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
cd6132790b2e alpine "sh -c while :; do p…" 10s ago Exited (137) 0s ago 8080->80/tcp
$ sudo bin/microbox rm cd61
cd61
A container is named by its ID or any unique prefix of it. A published port
answers on the host's addresses, including localhost; on WSL, Windows
reaches it at localhost:8080 too.
The stop took the full 10 seconds: sh as PID 1 ignores SIGTERM (see
below), so stop killed it after the timeout, hence the 137. With --init
the same stop takes milliseconds and the exit code is 143.
| Command | What it does |
|---|---|
ps [-a] |
running containers, or all of them |
stop [-t SECONDS] CONTAINER... |
SIGTERM to PID 1, then SIGKILL to every process in the container after the timeout (default 10) |
rm [-f] CONTAINER... |
remove stopped containers; -f kills running ones first |
exec CONTAINER COMMAND... |
run another command inside a running container, as its user and in its working directory, exiting with its code |
logs [-f] CONTAINER |
the output of a -d container; -f follows it until the container exits |
gc |
clean up after microbox processes that were killed (see below) |
One binary, two processes.
microbox run (host side, internal/container)
- Checks for root, resolves the image and makes a 12-hex-digit ID. Creates
/var/lib/microbox/containers/<id>/withupper,workandmerged, and the container's record,state.json. With-dit now hands the container to a supervisor process of its own (see the next section) and that process does the rest. - Creates the cgroup
/sys/fs/cgroup/microbox/<id>and writes the limits. The parentmicroboxgroup never holds processes, because cgroup v2 only hands controllers down from groups that have none. - Prepares the network (
internal/network):- Makes sure the shared pieces exist: bridge
mb0with 10.88.0.1/16,ip_forward, a MASQUERADE rule for traffic leaving 10.88.0.0/16, and FORWARD rules that let bridge traffic through. - Reserves the first free address by creating
/var/lib/microbox/network/ips/<ip>withO_EXCL, which is atomic. - For each
-p, adds DNAT rules: PREROUTING for traffic from outside, OUTPUT for the host's own connections. It also starts a proxy on127.0.0.1, because loopback traffic can't be DNATed to another interface (Docker's docker-proxy does the same).
- Makes sure the shared pieces exist: bridge
- Opens a pipe and starts
/proc/self/exe initwithCLONE_NEWUTS | CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWIPC | CLONE_NEWCGROUP | CLONE_NEWNET.- Go is multithreaded, and a multithreaded process can't move itself into a new mount namespace, so microbox starts a fresh copy of itself inside the new namespaces instead.
- The start uses clone3 with
CLONE_INTO_CGROUP(Go'sUseCgroupFD), so the child is born inside its cgroup, and its cgroup namespace starts at that group.
- While init waits on the pipe, attaches the network:
- Creates a veth pair (a virtual cable): one end
mb<id>plugged intomb0, the other created inside the container aseth0. - Through a netlink socket opened inside the container's namespace, it
brings
loup and giveseth0its address and a default route via 10.88.0.1.
- Creates a veth pair (a virtual cable): one end
- Writes the config as JSON into the pipe (fd 3 in the child). It includes
/etc/hostswith the container's address, and the host'sresolv.confwithout loopback nameservers, which the container can't reach. Then it waits for init to exec the command, and only after that records the container as running, with the host PID of its init. Until the exec, PID 1 is still microbox's init, and a signal fromstopwould hit init instead of the command. The wait uses a second pipe: init holds its write end with close-on-exec set, so the exec closes it and the read returns. - Waits, reports any OOM kills from
memory.events, then removes the port rules, the proxy, the veth, the address and the cgroup, records the exit code, and exits with it. With--rmit also removes the container's directory.
microbox init (container side, internal/nsinit), PID 1 in the new namespace
- Reads the config and sets the hostname.
- Marks
/as recursively private, so its mounts don't show up on the host. - Mounts an overlay on
merged: the image's layers are the read-only lower layers anduppertakes the container's writes, so the image never changes. The mount exists only in the container's mount namespace, so the kernel removes it when the container exits. pivot_root(".", ".")intomerged, then detaches the old root with all the host mounts under it.- Mounts
/proc(only this PID namespace), a read-only/sys, a read-only/sys/fs/cgroupshowing only the container's own group, and a tmpfs/devwith only null, zero, full, random, urandom and tty, plus its own devpts,/dev/shmand/dev/mqueue. - Hides the parts of
/procand/systhat belong to the host kernel, and makes others read-only (see Security). - Writes
/etc/hostname,/etc/hostsand/etc/resolv.conf. This happens after the pivot, so a symlink in the image can't point the write at a host file. - Drops capabilities, sets
no_new_privsand installs the seccomp filter. - Changes to the working directory and switches to the image's user (see Images).
- Execs the command, which takes over PID 1. With
--initit execs microbox's reaper instead, which stays PID 1 and starts the command as its child (see below).
Details worth knowing:
Pdeathsig: SIGKILLkills the container if its supervisor dies. It fires when the forking thread exits, so the supervisor holdsruntime.LockOSThread().- Ctrl-C: the terminal sends SIGINT to the whole foreground process group, so the container already gets it. microbox only catches it to stay alive. SIGTERM and SIGHUP are forwarded.
- A namespace's PID 1 ignores every signal it has no handler for, even SIGKILL
from inside. So
microbox run alpine sleep 30can't be stopped with Ctrl-C, the same asdocker runwithout--init. - Deleting one end of a veth deletes both. When the container's namespace
dies, the kernel removes
eth0and with itmb<id>, but asynchronously, so microbox deletesmb<id>itself.
The record. state.json holds the container's config and its status:
created, then running, then exited with an exit code. It is written to a
temporary file and renamed into place, so ps never reads half of one. The
process that writes it while the container runs is its supervisor: the
run command itself in the foreground, or the shim with -d. The record
keeps the PIDs of the supervisor and of the container's init together with
their start times from /proc/<pid>/stat, because PIDs get reused. A
container whose supervisor is gone without recording an exit shows up as
dead.
run -d starts microbox shim <id> with Setsid. The shim is a copy of
microbox in a session of its own and with no terminal, the job
containerd-shim does for Docker. Closing the terminal sends SIGHUP to its
session, which the shim isn't in. The shim starts the container exactly
like a foreground run does, with the container's output going straight
into containers/<id>/log. It reports "started" or the error back over a
pipe, and run -d exits as soon as it hears back. The shim stays behind to
wait for the container, hold the published-port proxies, and clean up.
stop sends SIGTERM to the container's PID 1 and waits until the
supervisor has recorded the exit. After the timeout it writes 1 to
cgroup.kill, which SIGKILLs every process in the cgroup at once, including
ones exec started. When stop returns, the cgroup, veth, address and port
rules are gone.
rm renames the container's directory to .removing-<id> and then
deletes it, so it disappears from ps at once, and gc can finish a
removal that was interrupted.
exec runs microbox exec-init inside the container, and that process
execs the command. Joining the namespaces is split between the two:
execjoins the PID namespace itself.setns(CLONE_NEWPID)never moves the caller, only the children it forks afterwards. It locks its goroutine to one OS thread first, because setns only changes the calling thread.exec-initjoins the others (IPC, UTS, network, cgroup, mount) after it has started. This is the order that works:- If
execjoined the mount namespace before forking, the kernel would load microbox's binary with the container's/. A plaingo buildis dynamically linked (thenetpackage pulls in cgo), so the kernel looks for/lib64/ld-linux-x86-64.so.2in the image. Alpine doesn't have it, so the exec fails with ENOENT. exec-initis cloned straight into the container's cgroup (CLONE_INTO_CGROUP, as inrun). The kernel only allows that from outside the container's cgroup namespace, because cgroup2 is mounted withnsdelegate: a process inside a cgroup namespace may only move processes between groups inside it.- Joining a mount namespace needs a thread that doesn't share its root
and working directory with other threads. Go's threads do share them,
which is why runc's nsexec does this in C before the Go runtime starts.
unshare(CLONE_FS)on the locked thread gives it its own copy, and thensetnsworks from Go.
- If
--init keeps microbox as PID 1. It starts the command as a child,
passes on every signal it receives, and reaps every child that exits,
including orphans the kernel reparents to it. When the command exits, it
exits with the same code, and the kernel kills whatever is left in the
namespace. tini (docker run --init) does the same.
gc cleans up after a supervisor that was killed. Its container died
with it through Pdeathsig, but the cgroup, the address, any port rules and
the record stay behind. gc removes dead containers and those resources,
plus any cgroup or network resource whose container directory no longer
exists. It finds port rules by the microbox:<id> comment they carry, and
addresses by the container ID in the reservation file, so it needs nothing
from the dead process. The same cleanup function runs when a container
exits normally.
Root in the container is still root on the host: there is no user namespace (see PLAN.md). The rest of Docker's default protections are there, so that root can't use the host's kernel against it:
$ sudo bin/microbox run --rm alpine sh -c '
grep -E "^(CapEff|NoNewPrivs|Seccomp):" /proc/self/status
mount -t tmpfs none /mnt; reboot -f; insmod /etc/hostname; unshare -U true
echo x > /proc/sys/kernel/domainname; wc -c < /proc/kcore'
CapEff: 00000000a80425fb
NoNewPrivs: 1
Seccomp: 2
mount: permission denied (are you root?)
reboot: : Operation not permitted
insmod: can't insert '/etc/hostname': Operation not permitted
unshare: unshare(0x10000000): Operation not permitted
sh: can't create /proc/sys/kernel/domainname: Read-only file system
0
On the host the same insmod gets as far as Invalid module format, and
unshare -U works.
- Capabilities. Docker's default 14 of root's 41 (
a80425fbabove): enough tochown, bind ports below 1024,pingandmknod, but notCAP_SYS_ADMIN(mounts, most namespace work),CAP_SYS_MODULE,CAP_SYS_BOOT,CAP_SYS_PTRACE,CAP_NET_ADMINorCAP_SYS_TIME. init removes the rest from the bounding set and empties the inheritable and ambient sets. For root, execve then computes the new capabilities as the bounding set plus the inheritable set, which leaves the 14. - no_new_privs. execve can't grant privileges any more, for instance through a setuid binary in the image.
- seccomp. A BPF program the kernel runs on every system call
(
internal/nsinit/seccomp_amd64.go). It denies, with EPERM, what Docker's profile leaves out: mounting,unshareandsetns, kernel modules, reboot and kexec, setting the clock, swap, the kernel keyring,bpf,perf_event_open,userfaultfd, io_uring,open_by_handle_at(the "shocker" escape) and a few obsolete calls.clonewith a namespace flag fails too.clone3gets ENOSYS, because BPF can't read its flags from memory, and the C library falls back toclone. Calls from 32-bit or x32 code kill the process, since their numbers differ from the table the filter checks. Many of these already fail without the capabilities; the ones that don't are the reason for the filter:unshare -U(a user namespace needs no capability), the keyring and io_uring. - Masked paths. Parts of
/procand/sysaren't namespaced and show or change the host:/proc/kcoreis the host's memory,/proc/sysrq-triggeracts on the host kernel. init binds/dev/nullover files like those and mounts an empty read-only tmpfs over directories, and makes/proc/sys,/proc/sysrq-trigger,/proc/bus,/proc/fsand/proc/irqread-only. Same lists as Docker. - A sealed copy of microbox. init, exec-init and the reaper run from a
copy of the binary in a sealed memfd, not from the file on the host. A
process's
/proc/<pid>/exeleads to the file it runs, and if a process in the container could open that link for a microbox process in the container, it could overwrite microbox on the host. That was CVE-2019-5736 in runc, which fixed it the same way. --init's PID 1. Capabilities,no_new_privsand seccomp filters belong to a thread. init applies them on one locked thread and execs from it, so the command gets them. The reaper runs for the container's whole life, so instead of running it inside init, whose other Go threads would keep every capability, init execs a fresh copy that starts from the restricted thread. That exec happens afterpivot_root, so the binary has to be static. The reaper also makes itself non-dumpable, which closes its/proc/1/exeand/proc/1/memto processes withoutCAP_SYS_PTRACE.
cmd/microbox/ CLI entry point: the commands, plus the hidden init, exec-init, reap and shim
internal/container/ host side: run, the shim and start/wait (run.go); the record (state.go);
ps, stop, rm, logs, gc (manage.go); exec (exec.go); the sealed copy
of the binary (binary.go); pull, images, rmi (images.go)
internal/image/ references (ref.go), the registry client and pull (registry.go),
unpacking layers (layer.go), the image store (store.go); imagetest/
is a registry in memory for the tests
internal/cgroups/ cgroup v2 group per container: limits, OOM count, kill, removal
internal/network/ bridge and NAT, IP allocation, veth + addressing, published ports, resolv.conf
internal/nsinit/ container side: hostname, exec, exec-init (init.go); overlay, pivot_root,
/dev, masked paths, /etc (rootfs.go); the --init reaper (reap.go);
capabilities and no_new_privs (security.go); the seccomp filter
(seccomp_amd64.go); working directory and user (user.go)
internal/spec/ config and exit codes shared by both sides
e2e/ tests that run the real binary
scripts/get-alpine.sh
make test runs go vet, the gofmt check, and then the tests as root. The e2e
tests build the binary and run it against the alpine image. They check:
- the hostname, and that the command is PID 1
pssees only the container- all six namespace inodes differ from the host's
- the exact list of container mounts (
/sysand the masks read-only), and an unchanged host mount table /devholds exactly the default devices and links/etc/hostname,/etc/hostsand/etc/resolv.conf- writes and deletes stay in the container: the image and the next container don't see them, parallel containers don't see each other's, and the container directory is gone afterwards
- exit codes, and SIGKILL from inside vs from the host
/proc/self/cgroupis0::/, and the limits (ormax) are readable inside- a 100 MB shell variable under
--memory 32mis OOM-killed with exit 137 --pids 8peaks at 8 processes and counts the refused forks- a one-second busy loop under
--cpus 0.2is throttled to well under 400 ms of CPU - bad flag values exit 2, and no cgroup is left behind
- the container has only
eth0andlo, an address in 10.88.0.0/16 matching/etc/hosts, and a default route via the bridge - two containers ping each other and talk over TCP
- the internet is reachable through NAT, DNS included (skipped when the host is offline)
-pserves through both127.0.0.1and the host's bridge address, and its rules and listener are gone after exit- a port that is already taken fails cleanly, with nothing left behind
--net nonehas only a workinglo, and--net hostshares the host's namespace- no veth or IP reservation is left behind
run -dprints an ID,pslists the container andlogshas its output; it keeps running after the terminal that started it is gone, and its supervisor has its own session and no terminalstopkills asleepas PID 1 after the timeout (exit 137), while with--initthe SIGTERM gets through at once (exit 143)- without
--initan orphan stays a zombie under PID 1; with it, it is reaped execgets the container's six namespaces and cgroup, and passes on exit codes (3, 126, 127); exec into a stopped container fails with 125- after
stopandrm: no cgroup, veth, address, port rule, mount or directory rmrefuses a running container,rm -fkills it, and a secondrmfails cleanly- without
--rma container stays as exited untilrm logs -ffollows until the container exits- killing the supervisor leaves a dead container, and
gcremoves it and its leftovers - a
-dcontainer that fails to start leaves nothing behind - the command, an exec'd process and every thread of
--init's PID 1 have Docker's capabilities,no_new_privsand a seccomp filter mount,reboot,insmod,unshare -U,unshare -manddate -sfail, and the container keeps runningchown,mknodandpingstill work/proc/kcoreis empty,/proc/sysand/proc/sysrq-triggerare read-only,/sys/firmwareis empty- the container can't read
/proc/1/exeof--init's PID 1, which runs the sealed memfd copy rather than the host's binary - a three-layer image (alpine, then a layer adding a file, then one deleting a file and making a directory opaque) pulled from a registry in the test runs its own command with its environment, working directory and user, and shows only what the top layers left
- a command replaces the image's,
--userthe image's user, and an unknown user fails with 125 - a second pull makes no GET requests,
rmiandgcremove the image and its layers, and a container keeps its layers afterrmiuntil it is removed busybox:1.36pulled from Docker Hub runs (skipped when offline)
Unit tests cover size parsing, the limit files (including cpu.max rounding),
the device-number encoding against the host's own /dev nodes, port and mode
parsing, the resolv.conf filter, address allocation, parsing tagged
iptables rules, the capability mask, and the seccomp program, run through a
small BPF interpreter on sample system calls. For images:
- reference parsing and the
WWW-Authenticateparser - a pull from the in-memory registry: token auth, index, digests, whiteout devices, the opaque xattr, hard links, setuid bits and owners, and the overlay those layers make
- a layer with a changed byte fails its digest check and leaves nothing
../paths, writes through a symlink and hard links out of the layer fail the pull without touching anything outside- unused-layer removal
Without root or the image, the tests skip.
The code uses Linux-only syscalls. Open the folder through VS Code's WSL
extension, or set GOOS=linux and GOARCH=amd64 for gopls.