Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
fix(sandbox): validate a pool host before sending it a credential
Pool hosts were found by filtering Jobs on labels, and adopted on that
basis alone. Labels are set by whoever creates the Job, and the nonce the
host token is derived from is itself a public label -- so any namespace
member who can create a Job could publish one carrying another user's pool
labels and that host's nonce, and the victim's client would adopt it and
send it the token derived for the *real* host. The Jobs proxy strips
`Authorization` but forwards `X-Sandbox-Token`, so the impostor receives
exactly the capability the real host accepts.

Nothing in the old path bound a Job to its creator, image, flavor, command
or URL. `_discover_hosts` even claimed in its docstring to match on
image/flavor; it did not.

Adoption now requires everything the *backend* asserts about a Job to line
up, not just the labels its creator chose:

- `initiator` -- the load-bearing check, and the one field `run_job` gives no
  way to set, so it cannot be forged client-side;
- owner namespace, image (normalized for registry prefixes and case), flavor,
  and that the command is the sandbox bootstrap;
- exactly one exposed URL, over HTTPS, whose hostname is derived from *this*
  job's id -- so a Job cannot name where the credentials should go.

New `adopt_hosts` argument on `SandboxPool` and `SandboxPool.connect`:
`"own"` (default) adopts only this principal's hosts, `"namespace"` restores
cross-user sharing for namespaces whose members trust each other, `"never"`
disables adoption. A refusal names the opt-in, so someone legitimately
sharing hosts is not left guessing.

`SandboxPool.connect`'s cold path gets the same treatment: it rebuilds the
pool's entire configuration from a discovered host's spec, so an unvalidated
match would let a Job in the namespace choose the image the pool's next
hosts boot. `_connect_host` addresses a job the caller named explicitly, so
ownership there is the caller's assertion, but the URL and bootstrap checks
still apply.

Separately, the HF bearer is no longer frozen when the transport is built.
A host lives up to 24h, and an OIDC/OAuth credential can expire inside that
window, after which every request failed even though the handle was fine. An
`httpx.Auth` now reads it per request. The `X-Sandbox-Token` is deliberately
*not* re-derived: the server holds the value derived from the bearer that
created the job, so re-deriving from a rotated bearer would produce a token
it has never seen. The two credentials answer different gates and only the
bearer needs refreshing. (A host created under one HF token therefore stays
reachable only by holders of that token -- inherent to deriving the
credential, and now stated in the docs.)

The discovery tests were part of the problem: their Job fixtures set only
labels, so they asserted that a label match is sufficient. They now describe
a Job the way the Jobs API really does, and a new suite covers the hostile
cases -- an impostor's initiator, a mismatched image/flavor/command/namespace,
a URL belonging to another job, a plain-HTTP URL, extra exposed ports, and a
missing initiator (refused rather than assumed).

Validation: 55 tests pass, up from 44. Writing them caught two real bugs in
this change -- `urlparse` lowercases hostnames, which broke the URL check on
a mixed-case job id, and a fixture whose host had no capacity.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
  • Loading branch information
Wauplin and claude committed Sep 14, 2026
commit 1add227bb4c28f9bb63ef335576f76d2462a16b5
6 changes: 3 additions & 3 deletions docs/source/en/concepts/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ One nonce is minted per **Job**, so the derived token is a *job* credential. In
| **pool host token** | `HMAC(hf_token, nonce)` from the host job's label | pool management: create, list and delete sandboxes on that host, and recover their tokens |
| **pooled sandbox token** | random 256 bits, minted by the host server per sandbox | that one sandbox — not a sibling, not the pool |

The client uses the narrow one automatically: `pool.create()` receives it in the create response, `Sandbox.connect("<host>.<id>")` recovers it with the host token, and [`proxy_headers`] hands out the sandbox's token rather than the host's — those headers usually end up in a browser or WebSocket client, so they should confer access to one sandbox and nothing more.
The client uses the narrow one automatically: `pool.create()` receives it in the create response, `Sandbox.connect("<host>.<id>")` recovers it with the host token, and [`proxy_headers`] hands out the sandbox's token rather than the host's (resolving the HF bearer at the moment you read it, so a long-lived handle does not hand out a stale one) — those headers usually end up in a browser or WebSocket client, so they should confer access to one sandbox and nothing more.

What this means for a leak: a **pooled sandbox token** compromises that sandbox. A **host token** compromises the host — every sandbox on it, current and future, plus its management routes — so treat it as the pool's admin credential. Members of your namespace hold a different HF token and cannot derive yours, but see [Known limitations](#known-limitations) for how a token can be *delivered* to the wrong place.

Expand Down Expand Up @@ -225,7 +225,7 @@ This section is deliberately exhaustive rather than reassuring: if you are decid
| --- | --- | --- |
| Anonymous internet user | Cannot reach the sandbox: the Jobs proxy requires an HF token with namespace read access. | Same. |
| Namespace member with read access | Reaches the proxy, but not the API: they cannot derive your sandbox token. Can see the job exists, its labels, and `/health`. | Same, plus they can read the pool's labels and nonce. |
| Namespace member who can create Jobs | — | Can publish a Job carrying your pool's labels and nonce. Your client may adopt it as a host and send it the token derived for your real host (see "Host discovery" below). |
| Namespace member who can create Jobs | — | Can publish a Job carrying your pool's labels and nonce, but it is not adopted: `adopt_hosts` defaults to hosts this principal started. Relevant again if you opt into `adopt_hosts="namespace"`. |
| Code running in the sandbox | Runs as root in the VM alongside `sbx-server`; owns the VM. | Confined by uid + Landlock as described above, but shares the kernel, VM and control plane with its neighbours. |
| Your own client | Holds the HF token and the sandbox token. | Same; in a pool the sandbox token covers the whole host. |

Expand All @@ -236,7 +236,7 @@ This section is deliberately exhaustive rather than reassuring: if you are decid
- the **port proxy** connects to `$SBX_PROXY_DIR/<port>.sock` without rejecting symlinks or checking the socket's owner, so a sandbox can point it at another sandbox's socket.

Both require a legitimate caller to invoke the endpoint (the sandbox has no token of its own), so they are confused-deputy problems rather than direct escapes — but they do break confidentiality and integrity between pooled sandboxes.
- **Host discovery trusts Job labels.** Hosts are found by filtering Jobs on labels, which any Job creator in the namespace can set, and the nonce that derives the token is a public label. Nothing binds a Job to its creator, image, or pool. In a namespace whose members do not all trust each other, prefer `Sandbox.create`, or use a namespace you control for pools.
- **Host discovery starts from Job labels**, which any Job creator in the namespace can set — so a label match is a claim, not proof. By default (`adopt_hosts="own"`) the client only adopts hosts *this principal started*, per the Jobs API's `initiator`, and additionally checks the image, flavor, command and exposed URL. Setting `adopt_hosts="namespace"` restores cross-user host sharing and, with it, the ability of any Job creator in that namespace to publish a host your client will send a token to — only use it in a namespace whose members you trust.
- **Landlock can degrade silently.** If Landlock is unavailable, or its ruleset cannot be built, the server currently falls back to uid-only isolation and creates the sandbox anyway — without telling the client. Under uid-only isolation, `/tmp`, `/dev/shm`, TCP bind and cross-home filesystem access are *not* denied. The server also accepts Landlock ABI 1, while the ✅ list above needs ABI 4 (TCP bind) and ABI 6 (abstract sockets); production kernels provide ABI 6, but a lower one would silently drop those two guarantees.
- **Residual shared channels**, none of which Landlock or uid isolation closes: unrestricted outbound TCP; loopback access to the control server; **UDP bind is allowed** (Landlock has no UDP coverage); a sibling's `/proc/<pid>/cmdline` and `status` are readable (`environ` is not — that is the part that would leak credentials, and it is denied); `/proc` and `/sys` are readable and `/dev` is broadly readable and writable; kernel IPC and all machine resources are shared.
- **No CPU, disk, FD or total-memory quotas.** Only per-process `RLIMIT_NPROC` and `RLIMIT_AS` are set; cgroup delegation is not available on Jobs. One sandbox can starve its neighbours. The `max_procs`/`max_mem_mb` values are caller-supplied and not clamped server-side.
Expand Down
Loading
Loading