Skip to content

About

Static in-sandbox server for 'hf sandbox' — command exec, file transfer and port forwarding on Hugging Face Jobs

Resources

Stars

14 stars

Watchers

1 watching

Forks

Latest commit

 

History

42 Commits

Folders and files

Repository files navigation

sandbox-server

The in-sandbox agent powering hf sandbox / huggingface_hub.Sandbox — isolated cloud machines built on Hugging Face Jobs.

A single static binary (~671KB, x86_64 musl, zero runtime dependencies) that runs in any Docker image with /bin/sh — no Python, pip, or framework required. The Sandbox client injects it at job startup and talks to it through the Jobs proxy (https://<job_id>--<port>.hf.jobs).

Two modes (one binary)

The same binary serves both:

  • Dedicated mode — one job is one sandbox. Operations hit /v1/exec, /v1/processes, /v1/files/* directly. Full VM isolation; used for GPU / untrusted workloads.
  • Host mode — one job hosts many lightweight sandboxes (huggingface_hub.SandboxPool). A sandbox is a dedicated uid + a private 0700 home + a per-sandbox Landlock LSM ruleset, created server-side in ~1ms. Operations are scoped under /v1/sandboxes/{id}/* and run as the sandbox uid, confined to its home. This packs dozens of isolated CPU sandboxes into one VM with sub-second per-sandbox cold start.

Host mode needs root + CAP_SETUID/SETGID/KILL (the Docker default on HF Jobs) and refuses to start if Landlock cannot deliver the documented guarantees (see SBX_MIN_LANDLOCK_ABI); pass --allow-unconfined to accept uid-only isolation instead. See src/landlock.rs for the confinement model (FS → own home + RO system dirs and selected /dev nodes; no TCP bind; ABI-6 abstract-socket scoping).

What it does

  • Command execution with live output streaming (NDJSON over chunked HTTP/1.1, flushed per event — the HTTP layer is hand-rolled because mainstream minimal frameworks buffer chunked responses until completion).
  • Background processes: start detached, list, and terminate (process-group kill) via a small /v1/processes registry with server-assigned opaque ids.
  • File API: raw-body read/write (no base64), offset/length params for parallel ranged transfers, list/stat/delete/mkdir.
  • Keepalive pings every 15s on all streams so proxies never kill idle connections.
  • Idle watchdog: exits when no request arrives and no process runs for SBX_IDLE_TIMEOUT seconds — abandoned sandboxes stop billing.

HTTP API

GET  /health   no auth → {"status","protocol"}   with a valid token → adds version,
                       uptime_ms, sandboxes, mode, auth, landlock{abi,features}, network
POST /v1/exec        {cmd, shell?, env?, cwd?, timeout?, stdin?, background?, tag?}
                     foreground → NDJSON stream: start / stdout / stderr / ping / exit
                     background → {"id", "pid", "tag"}
POST /v1/processes   {cmd, shell?, env?, cwd?, tag?}   → {"id", "pid", "cmd", "tag"}  (background)
GET  /v1/processes                        → [{"id","pid","cmd","tag","running","exit_code",...}]
DELETE /v1/processes/{id}                 → {"id","killed"}  (terminate + forget; idempotent.
                                             `id` is the opaque id, never the OS pid)
GET  /v1/files/read?path=&offset=&length= → raw bytes
PUT  /v1/files/write?path=&mode=&offset=&truncate_to=  → raw body to file (parents created;
                                             truncate_to sets the final size for ranged writes)
GET  /v1/files/list?path=&limit=&after=   → {"entries","truncated","next"}  (paginated)
GET  /v1/files/stat?path=
DELETE /v1/files/delete?path=&recursive=
POST /v1/files/mkdir?path=

# host mode (many sandboxes per job)
POST   /v1/sandboxes        {count?, env?, max_procs?, max_mem_mb?, network?}
                                                 → {"sandboxes":[{id,token,uid,home,confinement,network}]}
GET    /v1/sandboxes                                                 → live sandbox list
DELETE /v1/sandboxes                                                 → delete all
DELETE /v1/sandboxes/{id}                                            → delete one (frees the uid)
GET    /v1/sandboxes/{id}/token                                      → recover its capability token
# every dedicated route above also exists scoped to a sandbox, e.g.:
POST   /v1/sandboxes/{id}/exec        ...   GET /v1/sandboxes/{id}/processes
GET    /v1/sandboxes/{id}/files/read  ...   PUT /v1/sandboxes/{id}/files/write

cmd is either a string (run via /bin/sh -c) or an argv array. Pass shell (bool) to make that choice explicit instead of inferring it from the type: shell=true requires a string, shell=false requires an argv array. In host mode, file paths are rooted at the sandbox's private home (a leading / is taken relative to it) and created files are chowned to the sandbox uid. The privileged file API resolves every component relative to an open home descriptor with O_NOFOLLOW, so it never follows a symlink out of the home, and it assigns ownership through the resulting descriptor rather than by path.

Configuration (env vars)

var default meaning
SBX_PORT 8000 listen port (the client uses 49983 to keep common dev ports free)
SBX_TOKEN required all endpoints except /health require this value in the X-Sandbox-Token header (constant-time compare); removed from the env before any child process spawns. The server refuses to start without it, unless launched with --allow-no-auth (local development only — it is an argv flag, not an env var, so a Job's user-supplied env can never set it)
SBX_IDLE_TIMEOUT unset seconds of inactivity (no authed request, no running process) before clean exit
SBX_CAPACITY 64 host mode: max concurrent sandboxes. Refuses to start if unparseable (it used to fall back to unlimited)
SBX_MAX_CONNECTIONS 512 max concurrent connections; past it the server answers 503 without spawning a worker
SBX_NETWORK unset dedicated mode: network policy for every command, none or {"allow": [...]} (see Network policies). The server refuses to start if it cannot parse or enforce it. Host mode takes network per sandbox instead
SBX_MIN_LANDLOCK_ABI 6 host mode: minimum Landlock ABI to start with. 4 adds TCP-bind denial, 6 adds abstract-socket scoping — both are part of the documented model, so the default requires them. Lower it to accept a reduced set (/health reports what is in force)

Network policies

A sandbox can run without network access, or with access to an allowlist only. The policy applies to the commands the server runs, never to the server itself, so the server keeps serving the API and mounted volumes keep working (the platform mounts them outside the container).

"none"                                          no outbound access at all
{"allow": ["10.0.0.0/8",                        an address range
           "203.0.113.10:443",                  one address, one port
           "[2001:db8::1]:8000", "2001:db8::/32",
           "huggingface.co", "vllm.internal:8000"]}   hostnames, with an optional port

Dedicated mode reads it from SBX_NETWORK. In host mode, pass network to POST /v1/sandboxes, so isolated and connected sandboxes can share a host. The create response and /health echo the policy back: an older server ignores the field and echoes nothing, so a client can tell.

How it works (src/network.rs). Both layers are inherited by every descendant and cannot be removed, even by root in the sandbox:

  • A seccomp filter allows AF_UNIX and AF_NETLINK sockets, and TCP over IPv4/IPv6. It refuses UDP (so DNS too), ICMP, raw and packet sockets, any other family, TCP Fast Open, io_uring, ptrace, process_vm_writev and pidfd_getfd.
  • A Landlock ruleset (ABI 4) refuses every TCP connect(), loopback included. Binding and listening still work, so the port proxy keeps working.
  • With an allowlist, the server runs the command through a connect supervisor: the same binary (sbx-server --net-supervise), started as the command's parent with the sandbox's uid, limits and Landlock domain. The filter sends the sandbox's IP socket(), connect() and listen() calls to it. The supervisor creates each IP socket and installs it as the result of socket(), keeping a handle on it. On connect(), it checks the destination and connects that handle, which is the sandbox's own socket, to the address it checked. epoll registrations, duplicated descriptors and socket options keep working, and changing the address in memory after the check has no effect.
  • The supervisor is in the command's process group, so a timeout or a kill ends it with the command. It runs one thread per pending socket call, which counts against the sandbox's process limit (max_procs in host mode), not the server's. It handles at most 64 pending calls at a time (socket(), connect() and listen(), unix sockets included); past that they fail with EAGAIN. A blocking connect() stops waiting as soon as the caller is gone. If the supervisor cannot start the command (a missing program, for example), it tells the server, and the request fails with the same error as without a policy.
  • Allowlisted hostnames are resolved once by the server and pinned in /etc/hosts. The sandbox resolves them without DNS, to exactly the addresses that were allowed.

Limits:

  • With an allowlist, background processes that outlive the command that started them lose their socket access when it exits: connect() and listen() then fail with ENOSYS for every socket, unix sockets included, and so does creating a TCP socket. Start long-running services as background processes of their own (/v1/processes).
  • With an allowlist, the supervisor holds a handle on a socket until it connects, listens or is closed, and notices a close within about 3 seconds. Until then, a socket the sandbox closed keeps its local port: code that picks a free port by binding port 0, closing the socket and binding the port again can fail with EADDRINUSE. A connection closed before its non-blocking connect() completed can also be shut down late.
  • With an allowlist, a process that runs as another user than the command, or that makes itself non-dumpable, cannot connect: the supervisor cannot read its connect() arguments without CAP_SYS_PTRACE. This affects programs that drop privileges (apt-get downloads as _apt: pass -o APT::Sandbox::User=root), su, gosu, and agents such as ssh-agent.
  • An allowlist holds at most 1024 rules, counting every address its hostnames resolve to.
  • Tools that talk to themselves over local TCP (Ray, NCCL, Jupyter kernels) need 127.0.0.1 in the allowlist. In dedicated mode that also lets the sandbox reach the server's port, which still requires its token.
  • Debuggers (gdb, strace) do not work in a restricted sandbox: ptrace is refused.
  • A writable volume is still a way to move data out. Mount volumes read-only for strict isolation.
  • In host mode, a sandbox with network can pass a connected socket to a restricted one over a unix socket, if both cooperate. Pooled sandboxes share one trust boundary (see below).

Security model

The dedicated routes (/v1/exec, /v1/files/*, /v1/processes, /v1/proxy) act with the server's own privileges — root, unconfined, in the host's environment — so they exist only in dedicated mode, where the job is the sandbox. Host mode serves only /v1/sandboxes*, whose exec handlers drop to the sandbox uid and apply Landlock. File and proxy handlers remain privileged and use descriptor-relative paths. The two surfaces are mutually exclusive; the wrong one for the current mode answers 404.

Two layers when running on HF Jobs:

  1. The Jobs proxy requires an HF token with read access to the job's namespace.
  2. SBX_TOKEN is delivered via encrypted job secrets; the client derives it as HMAC-SHA256(user_hf_token, nonce) with the nonce stored in job labels — so reconnection is stateless and the HF token itself never enters the sandbox.

In dedicated mode the job is the sandbox, so SBX_TOKEN is already scoped to it.

In host mode there are two kinds of credential:

  • SBX_TOKEN is the host management token. It creates, lists and deletes sandboxes, and recovers their tokens. It is held by whoever runs the pool.
  • Each sandbox gets its own random 256-bit capability token, returned by POST /v1/sandboxes and recoverable with GET /v1/sandboxes/{id}/token. It authorizes that sandbox's routes and nothing else — not a sibling, not the pool. This is the credential to hand to whoever operates a single sandbox, including into a browser or WebSocket client via the port proxy.

Per-sandbox routes, including proxies, require that sandbox's capability token. The host management token is never accepted there. The former SBX_COMPAT_HOST_TOKEN setting has been removed; setting it cannot restore the fallback. This is a breaking authentication change (protocol 3): upgrade clients to use scoped tokens before deploying this server. Dedicated sandboxes and host management/token recovery retain their existing credentials.

Known limitations

Host mode packs mutually-visible tenants behind one privileged control plane. The following are known gaps rather than design intent, and are being worked through — treat host mode as a boundary between workloads inside one trust boundary, and use dedicated mode (one job per sandbox, a real VM) for mutually distrusting code.

  • No aggregate resource quotas. Per-process rlimits do not partition CPU shares, total memory, disk space, inodes, or network capacity between sandboxes. Directory pagination limits the response but still materializes all entries on the server. API file writes have no disk quota; dedicated-mode writes to special files can block.
  • Shared channels remain: outbound TCP, loopback access to the control server, UDP, kernel IPC, and readable process-list metadata. GPU isolation is untested in host mode.
  • The host token remains a management capability. It can recover every sandbox token; keep it private even though it cannot be used directly on scoped routes.
  • A hijacked proxy connection is authenticated and routed only once, then bytes are spliced until EOF. A second HTTP request written on that connection reaches the first backend without new routing, depending on the upstream proxy's behaviour.
  • A setsid descendant outlives a per-process kill or timeout (it leaves the signalled process group). Deleting the sandbox does terminate it — the uid sweep catches what a group kill misses.

Build

rustup target add x86_64-unknown-linux-musl
cargo build --release --target x86_64-unknown-linux-musl
# → target/x86_64-unknown-linux-musl/release/sbx-server (static-pie, stripped)

The binary is distributed via a Hugging Face bucket and downloaded at job startup by a /bin/sh bootstrap (wget → curl → bucket-mount fallback chain), which verifies the download against a digest pinned in the client before making it executable.

Testing

Three layers, split across two workflows so the check that gates a merge stays short.

cargo clippy --locked --all-targets -- -D warnings   # required
cargo test --locked                                  # required

# the target that actually ships, not the default gnu one
cargo build --locked --release --target x86_64-unknown-linux-musl
cargo test  --locked --target x86_64-unknown-linux-musl

scripts/*-regression.sh are live suites against a running server: they assert the isolation properties end to end (route surfaces, token scope, the Landlock denials, symlink confused deputies, process supervision, resource bounds). They need root — they create per-sandbox uids — and a Landlock-capable kernel, so run them in a container:

docker run --rm -v "$PWD:/src" -w /src ubuntu:24.04 sh -c \
  'apt-get update -qq && apt-get install -y -qq curl python3 python3-pip procps
   rm -f /usr/lib/python3*/EXTERNALLY-MANAGED   # so `pip install --user` works, as on Jobs
   for s in scripts/*-regression.sh; do sh "$s" || exit 1; done'

Each suite prints skip for an assertion its kernel's ABI cannot support, so read the reported ABI before trusting a green run: below 4 there is no TCP-bind denial and below 6 no abstract-socket scoping, and a suite that skipped them proves less than one that did not.

fuzz/ holds cargo fuzz targets for the two pure functions that see untrusted bytes first — the HTTP request-head parser and the host-mode path resolver. Both assert the invariant, not just the absence of a crash.

cargo +nightly fuzz run request-head fuzz/corpus/request-head fuzz/seeds/request-head \
  -- -max_total_time=60

.github/workflows/ci.yml runs the required checks; assurance.yml runs the musl build, the root suites and the fuzz targets on pushes to main, nightly, and on demand; audit.yml runs cargo audit weekly and on any lockfile change.

Releasing

.github/workflows/publish.yml runs on a v* tag (or a workflow_dispatch naming an explicit ref) and publishes the built binary under its own sha256, plus a manifest, plus the sbx-server alias. Digest-pinned clients keep using their pinned binary; older clients that fetch the mutable alias consume the new binary on their next launch. Updating a pinned client is a two-repo operation:

Before the first release, configure the GitHub production environment with required reviewers and restrict the HF Trusted Publisher to that environment. Naming an environment in the workflow does not configure an approval gate.

  1. Bump version in Cargo.toml, and PROTOCOL in src/main.rs if the wire contract changed in a way a client can be wrong about.
  2. Tag and push; the workflow prints the digest in its run summary.
  3. Update SANDBOX_SERVER_SHA256 (and SANDBOX_SERVER_VERSION) in huggingface_hub's _sandbox.py, and its expected protocol if it moved. Until that ships, every job keeps fetching and verifying the previously pinned digest.

Pool hosts keep running the binary they downloaded at boot for up to 24h, so a client and a server from different releases will meet in production. That is what the protocol field in /health is for: the client refuses the host instead of discovering the difference on some later route.

Status

Working prototype. See the huggingface_hub draft PR for the client, CLI, design notes and benchmarks (cold start ~6s, exec ~110ms p50, 340+ MiB/s parallel file transfer).

About

Static in-sandbox server for 'hf sandbox' — command exec, file transfer and port forwarding on Hugging Face Jobs

Resources

Stars

14 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages