Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
fix(sandbox): bind the pool cache to the context that wrote it
The local pool cache is not passive data. `SandboxPool.connect()`
rebuilds a pool from `$HF_HOME/sandbox/pools/<pool-id>.json` with no
HTTP at all, `_seed_hosts_from_cache` turns each entry into a
`_SandboxServer` from the cached `base_url` and `nonce`, and the first
thing that happens to it is an authenticated `POST /v1/sandboxes`
carrying the HF bearer and the token derived for that nonce. Two
problems followed from that.

The file was keyed by pool id alone, and its `namespace` field was read
as configuration rather than compared against what the caller asked
for -- `namespace=cache.namespace if namespace is None else namespace`.
So a pool cached for org-A and then reached with
`connect(pool, namespace="org-B")` routed the workload, and its env, to
org-A's hosts. That needs no attacker: one credential and two orgs is
an ordinary state of affairs, and nothing in the flow ever contradicts
the file. Nor was the cache bound to the endpoint or the credential, so
two users sharing a machine shared each other's entries.

And the URL was taken on trust. A cache file (or anything that can
write `$HF_HOME`) could name `http://127.0.0.1:1234` and receive both
credentials on the first create -- with `follow_redirects=True` on the
client, a legitimate-looking HTTPS URL could bounce them elsewhere too.
`verified=False` only means "drop the host if it fails"; by then the
credentials have been sent.

What changed:

- The cache lives at `pools/<context>/<pool-id>.json`, where `<context>`
  is a digest of endpoint, credential fingerprint and namespace. The
  same digest is stored in the payload and re-checked on read, so an
  entry moved or copied into a matching directory is still a miss. The
  credential appears only as a `sha256` fingerprint; the token is never
  written. Fingerprinting is local on purpose: resolving the credential
  to a user id would cost a `whoami` and defeat the point of the cache.
- `connect()` therefore honours `namespace=` instead of inheriting the
  cached one, and a mismatch on any of the three is a plain cache miss
  (debug log, cold path, never an exception).
- A cached `base_url` is only used if it is the HTTPS jobs-proxy URL of
  the job that same entry names, reusing the reasoning already applied
  to a discovered host's URL -- both now go through
  `_server_host_rejection`. An entry naming anything else is dropped
  from the file.
- Every field is type- and range-checked on read. `dataclass` enforces
  nothing, so a well-shaped file with `"capacity": "50"` used to parse
  happily and blow up much later on `host.capacity - host.live`, as an
  uncaught `TypeError` inside `create()`. It is now the documented miss.
- `follow_redirects=False` on the credentialed client. There is no
  redirect in this protocol; the proxy serves the in-job server.
- Freshness carve-out: an entry written by this principal within 15
  minutes is still credited with no round-trip (the `pool create` ->
  `create --pool` case the cache exists for). Past that, `inspect_job`
  has to confirm the job is running, labelled for this pool, carrying
  that nonce, adoptable, and on that URL before the transport is built.

Local hygiene in the same pass, all of it in the same "the file is not
authenticated, so don't hand it authority" spirit:

- cache dirs `0700` (tightened even if they predate this), and the
  atomic write goes through `tempfile.mkstemp` -- `0600` regardless of
  umask, unguessable name, removed on any failure -- instead of a
  predictable `<name>.<pid>.tmp` opened at `0666 & ~umask`.
- `files.download()` writes to an `O_EXCL|O_NOFOLLOW`, `0600` temp file
  next to the destination and `os.replace`s it into place. Before, it
  wrote in place: an interrupted download left a truncated file at the
  final name, and `open(path, "wb")` followed a symlink planted there.
  `rename` does not follow symlinks, so such a destination is now
  replaced rather than written through.
- `files.upload()` opens the file once and both `fstat`s and streams
  that descriptor, instead of `stat`ing the path and reopening it.

Validation (`PYTHONPATH=src python -m pytest tests/test_sandbox.py -q`):
83 passed, including new cases for cross-principal / cross-namespace /
cross-endpoint misses, moved and renamed files, mistyped fields, URL
admission, the freshness carve-out, redirect handling and the two
transfer paths. Two of them assert against a recording server that a
cache naming `http://127.0.0.1:<port>`, and a cross-namespace
`connect()`, result in zero requests reaching it. `ruff check` and
`ruff format --check` are clean on the three files.

Latency, modelling a 100 ms round-trip and timing
`connect(pool) + create()` (median of 5, cross-process CLI shape):
fresh cache hit 141 ms before / 141 ms after, cold path 560 ms before /
557 ms after, and an entry past the TTL 249 ms (one added round-trip).
The common case is unchanged; only stale entries pay.

Behaviour changes: existing cache files are all misses (new key, new
layout, `version` 2), so the first run after upgrading takes the cold
path once. A pool cached under an explicit `namespace=` is only found
again when the same `namespace=` is passed. A rotated credential reads
as a different principal and gets a miss. Anyone relying on a partial
file being left behind by a failed download no longer gets one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
  • Loading branch information
Wauplin and claude committed Sep 14, 2026
commit 3956db24c3d599e9326068e9a9817eb821204bfc
20 changes: 12 additions & 8 deletions docs/source/en/concepts/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,30 +190,34 @@ A pool is deliberately not a local config file. A pool is its set of running hos

Having no authoritative local state is great for correctness but costs latency. A cold `hf sandbox create --pool <id>` (a fresh CLI process) would otherwise have to rediscover everything over the network before it can create a sandbox: `list_jobs` to scan the namespace → `inspect_job` each host to rebuild its URL and nonce → `GET /v1/sandboxes` to see how full each is → finally `POST` to create. Several round-trips of pure overhead, on every call.

A best-effort cache at `$HF_HOME/sandbox/pools/<pool-id>.json` removes that. After any create/warm, a process records the pool config plus, per host, its proxy URL, auth nonce, and last-seen free slots. The next process rebuilds the host transport straight from the file (no HTTP) and goes directly to the `POST`.
A best-effort cache at `$HF_HOME/sandbox/pools/<context>/<pool-id>.json` removes that. After any create/warm, a process records the pool config plus, per host, its proxy URL, auth nonce, and last-seen free slots. The next process rebuilds the host transport straight from the file (no HTTP) and goes directly to the `POST`.

`<context>` is a digest of the endpoint, the credential and the namespace the entry was written for — the credential as a fingerprint; your token itself is never written to the cache. Both the directory name and the file carry it, and a read that doesn't match on all three is a plain cache miss, so another user of the machine, another endpoint, or a `connect(pool, namespace="other-org")` gets the cold path instead of hosts that were cached for something else. Files are created `0600`, directories `0700`.

```mermaid
flowchart TD
start["hf sandbox create --pool ID"] --> rc{"cache hit?<br/>$HF_HOME/sandbox/pools/ID.json"}
rc -- "yes (warm)" --> seed["rebuild host transport<br/>from cached URL + nonce<br/>(no HTTP)"]
start["hf sandbox create --pool ID"] --> rc{"cache hit?<br/>pools/&lt;context&gt;/ID.json<br/>(endpoint + credential + namespace)"}
rc -- "yes, written &lt;15 min ago" --> seed["rebuild host transport<br/>from cached URL + nonce<br/>(no HTTP)"]
rc -- "yes, older" --> check["inspect_job: still a running host<br/>of this pool, on that URL?"]
check -- "yes" --> seed
check -- "no" --> slow
seed --> post["POST /v1/sandboxes"]
post -- "ok" --> done["sandbox ready<br/>(~1 round-trip)"]
post -- "host gone / full" --> slow
rc -- "no / corrupt / stale" --> slow["fallback: list_jobs +<br/>inspect_job + GET (the cold path)"]
rc -- "no / corrupt / another context" --> slow["fallback: list_jobs +<br/>inspect_job + GET (the cold path)"]
slow --> post2["POST /v1/sandboxes"] --> done2["sandbox ready<br/>+ refresh cache"]
```

What the cache is and isn't trusted for:

- **Not authoritative on capacity.** The in-job server stays authoritative, so a stale `live` count only ever costs a wasted request, never a mis-packed host.
- **Not trusted for where a host lives.** A cached URL is only used if it is the HTTPS jobs-proxy URL of the job that same entry names, so a cache file cannot choose the destination your HF bearer and sandbox token are sent to; an entry naming anything else is dropped from the file. Past 15 minutes an entry is no longer credited on its own either: `inspect_job` has to confirm the job is still running, still labelled for this pool, and still on that URL before the transport is built.
- **Not trusted for its shape.** Types and ranges are checked on read, so a hand-edited, truncated or stringly-typed file is a cache miss rather than an error part-way through `create()`.
- **Self-healing.** A cached host that is gone is dropped on the first failed request and pruned from the file; the create transparently falls back to label discovery.
- **Concurrency-safe.** Writes merge under a file lock (keyed by `job_id`) and commit atomically, so parallel `create` processes don't clobber each other and readers never see a half-written file.
- **Disposable.** Delete it and you get a cache miss: the cold path runs and everything still works. It is never shared across machines.

It *is*, however, trusted for one thing that matters: **the URL a host is reached at.** The fast path rebuilds the transport from the cached `base_url` and nonce and then makes an authenticated request to it, so the first thing that happens to a cached entry is that your HF bearer and sandbox token are sent to whatever URL it names. The file is not authenticated, so it is exactly as trustworthy as `$HF_HOME`:

- Treat `$HF_HOME` as sensitive. Do not share it between users, restore it from an untrusted source, or place it on a world-writable path.
- The cache is currently keyed by pool id alone, so it is not bound to the endpoint, the credential, or the namespace it was written for. Passing an explicit `namespace=` to `SandboxPool.connect()` that differs from the one the cache was written for will still use the cached hosts. Delete `$HF_HOME/sandbox/pools/<pool-id>.json` if you need to be sure a `connect()` re-discovers from labels.
Two things to keep in mind. Treat `$HF_HOME` as sensitive: an entry is still a reusable host URL plus the public nonce its token derives from, and while a file written under a different credential is ignored, one written under *yours* is credited for the 15-minute window above. And because the namespace is part of the key, a pool cached under an explicit `namespace=` is only found again when the same `namespace=` is passed — `hf sandbox create --pool <id>` wants the same `--namespace` the pool was created with, or it takes the cold path and looks in your own namespace.

## Known limitations

Expand Down
Loading
Loading