Skip to content

[sandbox audit] Bind the sandbox pool cache to the endpoint, credential and namespace that wrote it - #4838

Merged
Wauplin merged 1 commit into
mainfrom
security/bind-pool-cache
Sep 14, 2026
Merged

Wauplin merged 1 commit into
mainfrom
security/bind-pool-cache

Conversation

@Wauplin

@Wauplin Wauplin commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

[sandbox audit] — PR 4 of 9 in this repo's stack; merge in order.
Previous: #4834 · Next: #4839
Review only the commits this PR adds on top of its base; bases collapse to main as the stack lands.

What was wrong

The local pool cache is not passive data. SandboxPool.connect() rebuilds a whole pool from $HF_HOME/sandbox/pools/<pool-id>.json with no HTTP at all, _seed_hosts_from_cache turns each entry into a _SandboxServer from the cached base_url and nonce, and the first thing that happens to that transport is an authenticated POST /v1/sandboxes carrying the HF bearer and the token derived for that nonce. Two consequences:

1. The cache was not bound to the context that wrote it. The key was the pool id alone, and the file's namespace was read as configuration rather than compared with what the caller asked for:

namespace=cache.namespace if namespace is None else namespace,

So a pool cached for org-A and then reached with connect(pool, namespace="org-B") ran the workload — and its env — on org-A's hosts, while everything the caller could see said org-B. That needs no attacker: one credential and two orgs is an ordinary state of affairs. The key also ignored the endpoint and the credential, so two users sharing a machine (or one user with two accounts) shared each other's entries.

2. The URL was taken on trust. A cache file — or anything able to write $HF_HOME — could name http://127.0.0.1:1234 and receive both credentials on the first create, before any API call could contradict it. follow_redirects=True on that client made a well-formed HTTPS URL enough too, since httpx re-sends Authorization on a same-scheme redirect. verified=False only means "drop the host if it fails"; by then the credentials are gone.

Alongside that, some local hygiene on the same files: the cache temp file was <name>.<pid>.tmp at 0666 & ~umask in a default-mode directory; files.download() wrote in place, so an interrupted download left a truncated file at the final name and open(path, "wb") followed a symlink planted at the destination; files.upload() stated the path and then reopened it. And a well-shaped file with "capacity": "50" parsed fine — dataclass enforces nothing — then failed much later on host.capacity - host.live as an uncaught TypeError in the middle of create().

What changed

Cache binding:

  • The cache lives at pools/<context>/<pool-id>.json, where <context> is a digest of endpoint + credential fingerprint + namespace. The same digest is stored in the payload and re-checked on read, so an entry moved or copied into a matching directory is still a miss. The credential appears only as a sha256 fingerprint — the token is never written to disk. Fingerprinting is deliberately local: resolving the credential to a user id would cost a whoami on every read and defeat the point of the cache.
  • connect() now honours namespace= instead of inheriting the cached one. Any mismatch on the three is a plain cache miss: debug log, cold path, never an exception.
  • A cached base_url is used only if it is the HTTPS jobs-proxy URL of the job that same entry names. This reuses the reasoning already applied to a discovered host's URL — both go through the new _server_host_rejection(url, job_id=...) helper — so a cache file can only name a job the client could have identified by itself. Entries naming anything else are dropped from the file.
  • Types and ranges are checked on read (counts are counts, bool is not a count, nonce looks like a nonce, job id can't contain a path separator), so the TypeError failure mode becomes the documented miss.
  • follow_redirects=False on the credentialed client. There is no redirect in this protocol; the Jobs proxy serves the in-job server directly.
  • Freshness carve-out, so the latency win survives: an entry written by this principal within 15 minutes is still credited with no round-trip — the hf sandbox pool create → hf sandbox create --pool sequence the cache exists for. Past that, inspect_job must confirm the job is running, labelled for this pool, carrying that nonce, adoptable under the handle's adopt_hosts policy, and exposing exactly the cached URL before the transport is built. A definitive "no" prunes the entry; a transient failure just skips it.

Local hardening:

  • Cache directories 0700 (tightened even if they predate this), and the atomic write goes through tempfile.mkstemp: 0600 regardless of umask, unguessable name, removed on any failure.
  • files.download() writes to an O_EXCL|O_NOFOLLOW, 0600 temp file next to the destination and os.replaces it into place (both the small and the ranged path). rename does not follow symlinks, so a symlinked destination is replaced rather than written through, an interrupted download leaves nothing behind, and the temp name is unpredictable.
  • files.upload() opens the file once and both fstats and streams that same descriptor — no stat-then-reopen.

Docs: the cache section of concepts/sandbox.md describes the new key, the freshness rule and the URL admission, and its "keyed by pool id alone / an explicit namespace= still uses the cached hosts" caveat is gone.

Validation

$ ruff check src/huggingface_hub/_sandbox.py src/huggingface_hub/_sandbox_cache.py
All checks passed!
$ ruff format --check src/huggingface_hub/_sandbox.py src/huggingface_hub/_sandbox_cache.py
2 files already formatted
$ PYTHONPATH=src python -m pytest tests/test_sandbox.py -q
83 passed in 21.19s

New cases: cross-credential / cross-namespace / cross-endpoint misses; an entry copied into another context's directory, and a renamed one; mistyped and implausible fields (including that connect() on a mistyped file raises the ordinary "no running host" rather than TypeError); URL admission (loopback, plain HTTP, another job, another port, and a staging domain that must keep working); the freshness carve-out in all three directions (fresh → no inspect_job; aged → confirmed; aged-and-no-longer-backed → dropped and pruned, with a transient API failure only skipping); 0600/0700 modes and no temp file left behind; the redirect; download-over-a-symlink, failed download, and upload through a symlink.

Two of them are the reproducers, asserted against a recording server: a cache naming http://127.0.0.1:<port> and a cross-namespace connect() both result in zero requests reaching it.

Latency, modelling a 100 ms round-trip and timing connect(pool) + create() in the cross-process CLI shape (median of 5, same harness on both branches):

before after
cache hit, entry written seconds ago 141 ms 141 ms
cache hit, entry past the 15-min TTL 139 ms 249 ms (one added round-trip)
cache miss (cold path: discovery) 560 ms 557 ms

The common case is unchanged; only entries the fast path can no longer be honest about pay.

Behaviour changes

  • Existing cache files become misses (new key, new layout, version 2), so the first create --pool after upgrading takes the cold path once. No migration: rewriting a legacy entry would mean trusting the very fields being bound.
  • A pool cached under an explicit namespace= is only found again when the same namespace= is passed — hf sandbox create --pool <id> wants the same --namespace the pool was created with, or it takes the cold path and looks in your own namespace.
  • A rotated or swapped credential reads as a different principal and gets a miss (the cold path, which always works).
  • A failed download no longer leaves a partial file at the destination.
  • upload() still follows a symlinked source: hf_hub_download() returns a symlink into the blobs directory, and refusing those would break a normal use. The TOCTOU fix here is the single descriptor; the source path is one the caller chose in their own filesystem.

Adjacent, not folded in

  • upload() of a file above the parallel threshold still materializes it with f.read(). That is the separate full-file-buffering issue, and it belongs with the transfer-memory work rather than here.
  • The cache has always bypassed adopt_hosts for hosts it recorded itself (a fresh entry is seeded without the policy check). Pre-existing, and arguably right — the entry was written by this principal — but worth a look when adopt_hosts is next revisited.

Review follow-up (2026-09-14)

Cache deletion now treats glob metacharacters in pool names literally. The cache privacy test allows filelock versions that remove their lockfile. A live Jobs proxy check confirmed the API-provided origin is accepted, a 302 is not followed, Authorization is stripped by Jobs, and X-Sandbox-Token is forwarded.

Rebased onto current main and restacked; pushed to restart CI. Merging and releasing remain manual.


Note

High Risk
Changes authentication and credential routing for pool fast-path (cache URLs, redirects, namespace binding); mistakes could break connects or mis-route tokens, though invalid entries fall back to discovery.

Overview
Hardens sandbox pool cache usage so cached host URLs cannot steer HF bearer and sandbox tokens to arbitrary destinations, and so cache hits are scoped to the writer’s endpoint, credential fingerprint, and namespace.

Cache binding (v2 layout): Entries live under $HF_HOME/sandbox/pools/<context>/<pool-id>.json. SandboxPool.connect() no longer inherits namespace from the file; mismatches are cache misses. On read, schema checks turn bad files into misses instead of mid-create() TypeErrors. Directories/files are 0700/0600 with safer atomic writes.

Trust rules for cached hosts: A cached base_url is only used if it is the HTTPS jobs-proxy URL for that entry’s job_id (shared _server_host_rejection). Entries younger than 15 minutes can seed without HTTP; older ones require inspect_job (running, correct pool/nonce/adoption, same URL). The credentialed httpx client sets follow_redirects=False.

Local file I/O: files.download() writes via a private temp file and os.replace (symlink-safe, no partial files); files.upload() sizes and reads one open descriptor.

Docs in concepts/sandbox.md describe the new cache key, freshness, and URL admission. Existing v1 cache files become misses (cold path once after upgrade).

Reviewed by Cursor Bugbot for commit 3956db2. Bugbot is set up for automated code reviews on this repo. Configure here.

@bot-ci-comment

bot-ci-comment Bot commented Sep 8, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/huggingface_hub/_sandbox_cache.py Resolved
@Wauplin Wauplin changed the title Bind the sandbox pool cache to the endpoint, credential and namespace that wrote it [sandbox audit] Bind the sandbox pool cache to the endpoint, credential and namespace that wrote it Sep 9, 2026
@Wauplin
Wauplin added this pull request to stack #4903 September 14, 2026 10:11
@Wauplin
Wauplin force-pushed the security/host-adoption-admission-control branch from 04c3deb to 5a327ec Compare September 14, 2026 10:41
@Wauplin
Wauplin force-pushed the security/bind-pool-cache branch 2 times, most recently from 567de61 to c4c87ef Compare September 14, 2026 10:47

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c4c87ef. Configure here.

Comment thread src/huggingface_hub/_sandbox.py Resolved
Comment thread src/huggingface_hub/_sandbox.py Resolved
@Wauplin
Wauplin force-pushed the security/bind-pool-cache branch from c4c87ef to b407857 Compare September 14, 2026 11:00
@Wauplin
Wauplin marked this pull request as ready for review September 14, 2026 12:23
@Wauplin
Wauplin force-pushed the security/host-adoption-admission-control branch from 2cabd71 to 974a3f7 Compare September 14, 2026 12:25
@Wauplin
Wauplin force-pushed the security/bind-pool-cache branch from b407857 to 990f83c Compare September 14, 2026 12:25
@Wauplin
Wauplin removed this pull request from stack #4903 September 14, 2026 12:30
@Wauplin
Wauplin force-pushed the security/host-adoption-admission-control branch from 974a3f7 to 1add227 Compare September 14, 2026 12:31
Base automatically changed from security/host-adoption-admission-control to main September 14, 2026 12:31
The local pool cache is not passive data. `SandboxPool.connect()`
rebuilds a pool from `$HF_HOME/sandbox/pools/<pool-id>.json` with no
HTTP at all, `_seed_hosts_from_cache` turns each entry into a
`_SandboxServer` from the cached `base_url` and `nonce`, and the first
thing that happens to it is an authenticated `POST /v1/sandboxes`
carrying the HF bearer and the token derived for that nonce. Two
problems followed from that.

The file was keyed by pool id alone, and its `namespace` field was read
as configuration rather than compared against what the caller asked
for -- `namespace=cache.namespace if namespace is None else namespace`.
So a pool cached for org-A and then reached with
`connect(pool, namespace="org-B")` routed the workload, and its env, to
org-A's hosts. That needs no attacker: one credential and two orgs is
an ordinary state of affairs, and nothing in the flow ever contradicts
the file. Nor was the cache bound to the endpoint or the credential, so
two users sharing a machine shared each other's entries.

And the URL was taken on trust. A cache file (or anything that can
write `$HF_HOME`) could name `http://127.0.0.1:1234` and receive both
credentials on the first create -- with `follow_redirects=True` on the
client, a legitimate-looking HTTPS URL could bounce them elsewhere too.
`verified=False` only means "drop the host if it fails"; by then the
credentials have been sent.

What changed:

- The cache lives at `pools/<context>/<pool-id>.json`, where `<context>`
  is a digest of endpoint, credential fingerprint and namespace. The
  same digest is stored in the payload and re-checked on read, so an
  entry moved or copied into a matching directory is still a miss. The
  credential appears only as a `sha256` fingerprint; the token is never
  written. Fingerprinting is local on purpose: resolving the credential
  to a user id would cost a `whoami` and defeat the point of the cache.
- `connect()` therefore honours `namespace=` instead of inheriting the
  cached one, and a mismatch on any of the three is a plain cache miss
  (debug log, cold path, never an exception).
- A cached `base_url` is only used if it is the HTTPS jobs-proxy URL of
  the job that same entry names, reusing the reasoning already applied
  to a discovered host's URL -- both now go through
  `_server_host_rejection`. An entry naming anything else is dropped
  from the file.
- Every field is type- and range-checked on read. `dataclass` enforces
  nothing, so a well-shaped file with `"capacity": "50"` used to parse
  happily and blow up much later on `host.capacity - host.live`, as an
  uncaught `TypeError` inside `create()`. It is now the documented miss.
- `follow_redirects=False` on the credentialed client. There is no
  redirect in this protocol; the proxy serves the in-job server.
- Freshness carve-out: an entry written by this principal within 15
  minutes is still credited with no round-trip (the `pool create` ->
  `create --pool` case the cache exists for). Past that, `inspect_job`
  has to confirm the job is running, labelled for this pool, carrying
  that nonce, adoptable, and on that URL before the transport is built.

Local hygiene in the same pass, all of it in the same "the file is not
authenticated, so don't hand it authority" spirit:

- cache dirs `0700` (tightened even if they predate this), and the
  atomic write goes through `tempfile.mkstemp` -- `0600` regardless of
  umask, unguessable name, removed on any failure -- instead of a
  predictable `<name>.<pid>.tmp` opened at `0666 & ~umask`.
- `files.download()` writes to an `O_EXCL|O_NOFOLLOW`, `0600` temp file
  next to the destination and `os.replace`s it into place. Before, it
  wrote in place: an interrupted download left a truncated file at the
  final name, and `open(path, "wb")` followed a symlink planted there.
  `rename` does not follow symlinks, so such a destination is now
  replaced rather than written through.
- `files.upload()` opens the file once and both `fstat`s and streams
  that descriptor, instead of `stat`ing the path and reopening it.

Validation (`PYTHONPATH=src python -m pytest tests/test_sandbox.py -q`):
83 passed, including new cases for cross-principal / cross-namespace /
cross-endpoint misses, moved and renamed files, mistyped fields, URL
admission, the freshness carve-out, redirect handling and the two
transfer paths. Two of them assert against a recording server that a
cache naming `http://127.0.0.1:<port>`, and a cross-namespace
`connect()`, result in zero requests reaching it. `ruff check` and
`ruff format --check` are clean on the three files.

Latency, modelling a 100 ms round-trip and timing
`connect(pool) + create()` (median of 5, cross-process CLI shape):
fresh cache hit 141 ms before / 141 ms after, cold path 560 ms before /
557 ms after, and an entry past the TTL 249 ms (one added round-trip).
The common case is unchanged; only stale entries pay.

Behaviour changes: existing cache files are all misses (new key, new
layout, `version` 2), so the first run after upgrading takes the cold
path once. A pool cached under an explicit `namespace=` is only found
again when the same `namespace=` is passed. A rotated credential reads
as a different principal and gets a miss. Anyone relying on a partial
file being left behind by a failed download no longer gets one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Wauplin
Wauplin force-pushed the security/bind-pool-cache branch from 990f83c to 3956db2 Compare September 14, 2026 12:31
@Wauplin
Wauplin merged commit 40efc5e into main Sep 14, 2026
@Wauplin
Wauplin deleted the security/bind-pool-cache branch September 14, 2026 12:31
@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v1.32.0 release.

g0rdonL added a commit to g0rdonL/huggingface_hub that referenced this pull request Oct 8, 2026
* Post-release: bump version to 1.30.0.dev0 (#4763)

Co-authored-by: huggingface-hub-bot[bot] <huggingface-hub-bot[bot]@users.noreply.github.com>

* [CI] Enable enforcement in close-unscoped-community-prs workflow (#4766)

* Align `hf jobs scheduled ls` filtering with `hf jobs ls` (`--status`/`--label`/`--name`) (#4773)

* Align `hf jobs scheduled ls` filters with `hf jobs ls`

Requested by @Wauplin (Lucain Pouget) via Moon Bot.
Co-authored-by: Wauplin <Wauplin@users.noreply.github.com>
Slack thread: https://huggingface.slack.com/archives/C02V5EA0A95/p1787841077235539?thread_ts=1787841077.235539&cid=C02V5EA0A95

* Remove tests from PR

Requested by @Wauplin (Lucain Pouget) via Moon Bot.
Co-authored-by: Wauplin <Wauplin@users.noreply.github.com>
Slack thread: https://huggingface.slack.com/archives/C02V5EA0A95/p1787841077235539?thread_ts=1787841077.235539&cid=C02V5EA0A95

* Regenerate CLI reference docs

Requested by @Wauplin (Lucain Pouget) via Moon Bot.
Co-authored-by: Wauplin <Wauplin@users.noreply.github.com>
Slack thread: https://huggingface.slack.com/archives/C02V5EA0A95/p1787841077235539?thread_ts=1787841077.235539&cid=C02V5EA0A95

* Fix mypy: rename label loop variable to avoid shadowing

Requested by @Wauplin (Lucain Pouget) via Moon Bot.
Co-authored-by: Wauplin <Wauplin@users.noreply.github.com>
Slack thread: https://huggingface.slack.com/archives/C02V5EA0A95/p1787841077235539?thread_ts=1787841077.235539&cid=C02V5EA0A95

---------

Co-authored-by: moonbot[bot] <moonbot[bot]@users.noreply.github.com>
Co-authored-by: Wauplin <Wauplin@users.noreply.github.com>

* [Download] Follow redirects between Hub hosts when resolving files (#4739)

* [Download] Follow redirects between Hub hosts when resolving files

* [Download] Keep auth header when download location stays on a Hub host

* [Download] Drop redirect cap, keep the original unbounded loop

* [Download] Bound redirect following at httpx's default of 20

* [Download] Name the redirect bound

* less verbose please

---------

Co-authored-by: Lucain <lucain@huggingface.co>

* [Download] A `ResolvedRevision` only pins the repo it was resolved for (#4767)

* [Download] A `ResolvedRevision` only pins the repo it was resolved for

A commit hash means nothing outside of the repo it comes from, but a
`ResolvedRevision` used to be trusted by any repo: `resolve_revision`
returned it untouched, and the download helpers used its commit hash
whatever repo they were fetching from. A library resolving a revision for
one repo and reusing it for another (a base model and its adapter, a
pipeline component in its own repo, ...) would then request a commit that
does not exist there.

Instances now remember the repo (id + type) they were resolved against,
and only pin that one. Anywhere else, the revision initially requested is
used instead, and resolved again if needed.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Download] Scope the fix to resolve_revision: drop _commit_hash_for

Download helpers trust the passed `ResolvedRevision` again (`revision.resolved`
as before). The only behavior change is in `HfApi.resolve_revision`: a revision
resolved for another repo is resolved again for the requested one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [CI] Fix torch_1.11 case-arm label to match renamed matrix entry (#4776)

The torch~=1.11 matrix entry's test_name was renamed from
"Python 3.9, torch_1.11" to "Python 3.10, torch_1.11", but the
install-step case arm further down was not updated to match. The
esac has no `*)` default arm, so the case silently falls through,
torch is never installed on this lane, and its tests skip with
"Test requires torch" while the job still reports green.

* [Sandbox] Document experimental status (#4783)

* [Inference Providers] Do not require the conversational tag for chat-completion on hf-inference (#4784)

* Do not require the conversational tag for chat-completion on hf-inference

The Hub no longer requires the "conversational" tag to serve
text-generation / image-text-to-text models through the chat-completion
API for inference providers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Clarify comment: raw text-generation is hf-inference specific

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Remove comment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Update task-check test expectations

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* [CI] Disable auto close (#4800)

* [Release] Consolidate the HF CLI skill sync into the release workflow (#4765)

* [Release] Drop duplicate HF CLI skill sync job

Every release opened two PRs on huggingface/skills: one from this job and
one from sync-hf-cli-skill.yml, which the release tag push also triggers.
Only the standalone workflow bumps the plugin version and regenerates the
skills repo artifacts, so its PR is the one that passes validate.

* [Release] Keep the skill sync in the release workflow instead

Reverses the previous commit's direction: rather than deleting the job and
relying on the tag-triggered workflow, port the working implementation (uv,
plugin version bump, publish.sh, SKILL.md-scoped diff) into the release job
and delete sync-hf-cli-skill.yml. One sync, inside the workflow that already
reports to the release Slack thread and the final status check.

The PR body now points at the release tag rather than github.sha, which on a
workflow_dispatch run is main's HEAD, not the commit that was synced.

* Post-release: bump version to 1.31.0.dev0 (#4806)

* Bump the actions group across 1 directory with 5 updates (#4798)

Bumps the actions group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [huggingface/doc-builder/.github/workflows/build_main_documentation.yml](https://github.com/huggingface/doc-builder) | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | `c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27` |
| [huggingface/doc-builder/.github/workflows/build_pr_documentation.yml](https://github.com/huggingface/doc-builder) | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | `c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27` |
| [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `10.0.0` | `10.0.1` |
| [trufflesecurity/trufflehog](https://github.com/trufflesecurity/trufflehog) | `3.96.0` | `3.97.1` |
| [huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml](https://github.com/huggingface/doc-builder) | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | `c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27` |



Updates `huggingface/doc-builder/.github/workflows/build_main_documentation.yml` from af49b8dc3859c3d80c7220c8c0ba146431132e49 to c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27
- [Release notes](https://github.com/huggingface/doc-builder/releases)
- [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md)
- [Commits](https://github.com/huggingface/doc-builder/compare/af49b8dc3859c3d80c7220c8c0ba146431132e49...c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27)

Updates `huggingface/doc-builder/.github/workflows/build_pr_documentation.yml` from af49b8dc3859c3d80c7220c8c0ba146431132e49 to c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27
- [Release notes](https://github.com/huggingface/doc-builder/releases)
- [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md)
- [Commits](https://github.com/huggingface/doc-builder/compare/af49b8dc3859c3d80c7220c8c0ba146431132e49...c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27)

Updates `astral-sh/setup-uv` from 10.0.0 to 10.0.1
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](https://github.com/astral-sh/setup-uv/compare/ae62891fec2bb8e7d6c99fc78c9fec3a63790f8d...20cfd1bf945f4377ade1205e4dbc17946fc9a30d)

Updates `trufflesecurity/trufflehog` from 3.96.0 to 3.97.1
- [Release notes](https://github.com/trufflesecurity/trufflehog/releases)
- [Commits](https://github.com/trufflesecurity/trufflehog/compare/6f3c981e7b77f235fd2702dd74af25fc4b72bf11...20652fbbdefffcdaa493a5bf57ab2ac6b1db715b)

Updates `huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml` from af49b8dc3859c3d80c7220c8c0ba146431132e49 to c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27
- [Release notes](https://github.com/huggingface/doc-builder/releases)
- [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md)
- [Commits](https://github.com/huggingface/doc-builder/compare/af49b8dc3859c3d80c7220c8c0ba146431132e49...c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 10.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: huggingface/doc-builder/.github/workflows/build_main_documentation.yml
  dependency-version: c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27
  dependency-type: direct:production
  dependency-group: actions
- dependency-name: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml
  dependency-version: c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27
  dependency-type: direct:production
  dependency-group: actions
- dependency-name: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml
  dependency-version: c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27
  dependency-type: direct:production
  dependency-group: actions
- dependency-name: trufflesecurity/trufflehog
  dependency-version: 3.97.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [Docs] Clarify Jobs argument forwarding and align examples (#4809)

* [Docs] Clarify Jobs argument forwarding and align examples

* [Docs] Distinguish option ordering from argument forwarding

* [Docs] Focus Jobs guidance on the explicit argument separator

* [Docs] Explain where script help arguments are forwarded

* [httpx migration] Expose `httpx` as a `huggingface_hub` submodule (#4803)

* Expose httpx module

* add note in docs

* [hf_api] Make CommitInfo copy-able and pickle-able (#4822)

`CommitInfo.__new__` took `commit_url` as a required keyword-only argument, so
the reconstruction path used by `copy`/`pickle` (`cls.__new__(cls,
*self.__getnewargs__())`, i.e. positional string value only) raised a
TypeError. `commit_url` is now a regular parameter and a `__reduce__` is added
so all fields survive the round-trip, mirroring what `ResolvedRevision`
already does.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Download] Prevent cached file copies during local-dir dry runs (#4817)

* [Download] Prevent cached file copies during local-dir dry runs

* Apply suggestion from @Wauplin

* Apply suggestion from @Wauplin

---------

Co-authored-by: Lucain <lucainp@gmail.com>

* Fix: keep single-quoted values verbatim in load_dotenv (#4827)

* Keep single-quoted values verbatim in load_dotenv

* Apply style fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* [CLI] Fix 'hf buckets rm --recursive' deleting lexical siblings of the prefix (#4804)

* [CLI] Fix 'hf buckets rm --recursive' deleting lexical siblings of the prefix

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [CLI] Apply the same path-boundary filter to 'hf buckets ls'

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Sandbox] Support custom Job labels (#4824)

* [Sandbox] Support custom labels in Sandbox.create

* [CLI] Add labels to sandbox create

* [CLI] Don't let `hf update` self-upgrade a pip install on Windows (#4823)

* [CLI] Don't let `hf update` self-upgrade a pip install on Windows

Windows refuses to replace the `hf.exe` launcher of a running process,
so `pip install -U huggingface_hub` spawned by `hf update` fails with
WinError 32 after having uninstalled the current version, leaving a dead
launcher and no `huggingface_hub` module. Print the command for the user
to run instead, like pip does for its own self-upgrade.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [CLI] Use out.error + Exit(1) instead of CLIError

Avoids the "set HF_DEBUG=1 for full traceback" hint: this is an
instruction to the user, not an internal failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [HfFileSystem] Reject unsafe paths before local downloads (#4828)

* [HfFileSystem] Reject unsafe paths before local downloads

* Apply suggestion from @Wauplin

* Apply suggestion from @Wauplin

* [Download] Share retry handling for stream entry and body failures (#4826)

* [Download] Share retry handling for stream entry and body failures

* [Download] Remove added retry unit tests

* [Download] Tolerate missing HEAD Content-Length (#4805)

* [Download] Tolerate missing HEAD Content-Length

* [Download] Allow missing HEAD size when Xet is unavailable

---------

Co-authored-by: Celina Hanouti <hanouticelina@gmail.com>

* [Download] Write cache ref file atomically in `snapshot_download` (#4829)

`snapshot_download` rewrote `refs/<revision>` with a plain `open(path, "w")` on every
call, even when the commit hash was unchanged. `open(..., "w")` truncates the file
before rewriting it, so a concurrent reader can observe an empty revision and conclude
that a fully cached file is missing.

Use the existing `_cache_commit_hash_for_specific_revision` helper instead: it skips the
write when the ref is already up to date and otherwise writes to a temporary file and
`os.replace`s it into place. `hf_hub_download` and `HfApi.resolve_revision` already go
through it.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [CLI] Raise explicit error for shell-script extensions on Windows (#4846)

* [CLI] Raise explicit error for shell-script extensions on Windows

* [CLI] Surface install errors from extension auto-install

* [CLI] Drop Windows-specific extension install tests

* [Jobs] Add network groups to hf jobs run (#4833)

* [Jobs] Add network groups to hf jobs run

* Trim network groups guide sections

---------

Co-authored-by: Lucain <lucain@huggingface.co>

* Fix dotenv parser truncating unquoted values containing '#' (#4842)

* Fix dotenv parser truncating unquoted values containing '#'

* fix empty values swallowing inline comments and quoted values absoprtion bug

* bare keys with comments dropped bug fixed

* bare key alternative fix

* (LFS)Fix SliceFileObj.__iter__ yielding only the first 4MB chunk (#4844)

Co-authored-by: Lucain <lucain@huggingface.co>

* [CLI] Fix truncated command descriptions in the CLI reference (#4849)

* Post-release: bump version to 1.32.0.dev0 (#4856)

Co-authored-by: huggingface-hub-bot[bot] <huggingface-hub-bot[bot]@users.noreply.github.com>

* docs(jobs): scope network groups to the namespace and resource group (#4855)

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* [Community] Fix docstring for DiscussionComment.last_edited_by (#4864)

* [Community] Fix docstring for DiscussionComment.last_edited_by

* Update tests/test_cli_discussions.py

* Update tests/test_cli_discussions.py

---------

Co-authored-by: célina <hanouticelina@gmail.com>

* fix(sandbox): preserve terminal command results (#4851)

* [CLI] Read a UV script's `[tool.hf-jobs]` launch config (opus-generated) (#4598)

* [CLI] Read the launch config a UV script carries in `[tool.hf-jobs]`

Implements the `[tool.hf-jobs]` PEP 723 table discussed in #4558: a UV script
can carry the runtime it needs (image, flavor, python, env, secrets, timeout,
name, namespace, labels, volumes) and `hf jobs uv run` reads it client-side.

- CLI flags always win; env/secrets/labels/volumes merge entry by entry
- secrets list names only, values read from the caller's environment (error if
  unset, warning when sent)
- unknown keys and non-string values are errors, not silently dropped intents
- URL scripts are downloaded once at submit time and shipped like local scripts
- every run echoes the resolved config (secrets redacted); `--dry-run` prints it
  without submitting
- default Job name now hashes the resolved config, not just the command line

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [CLI] Address review: utf-8 script read, consistent auto-naming, API/CLI note

- read scripts as UTF-8 rather than with the locale encoding (review comment)
- `hf jobs run` and `hf jobs uv run` now hash the same resolved values into the
  default Job name (env values included, secret names only - never their values)
- document in `run_uv_job` / `create_scheduled_uv_job` and in the Jobs guide that
  the Python API ignores `[tool.hf-jobs]`, unlike `hf jobs uv run`

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [CLI] Simplify UV script loading: context-managed download, lazy environ

- `load_uv_script` is now a context manager built on `SoftTemporaryDirectory`:
  the downloaded script's lifetime is the `with` block, instead of a
  `TemporaryDirectory` kept alive by a `tmp_dir` field on `UvScript` plus a
  `source` field on `_UvJobConfig` whose only job was to keep *that* alive.
- `_resolve_uv_job_config` yields the config from inside the block, so the
  Job is always submitted while the download still exists.
- `parse_env_map` no longer copies `os.environ` + reads the token file when
  there are no env args and no env file to resolve (twice per `jobs run`).
- `_resolve_script_secrets` skips the same work when the script requests no
  secrets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* [CLI] Address review: request timeout on script download, dry-run secrets

- `_download_script` passes `constants.DEFAULT_REQUEST_TIMEOUT`, so an
  unreachable URL fails the CLI instead of hanging it.
- The secrets warning says "would be sent" under `--dry-run`, where nothing
  is submitted.
- A secret the script requests but the caller has not set is shown as
  `NAME=<not set>` in a dry run instead of erroring, so the full config of a
  not-yet-provisioned script stays visible. A real run still errors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [CLI] Avoid script metadata in the UV header module docstring

* Update docs/source/en/guides/cli.md

Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com>

* [CLI] Docs: scope Job auto-naming to the CLI, add TOML examples for [tool.hf-jobs] keys

Follow-up on review feedback:
- cli.md/jobs.md naming paragraphs now say the hash covers the resolved
  runtime settings (CLI only; the Python API still hashes the command).
- show timeout/env/labels/volumes syntax in the header example and spell
  out that env/labels values must be TOML strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com>
Co-authored-by: célina <hanouticelina@gmail.com>

* Add `--container-registry-{username,password}` (#4854)

* Add `--container-registry-{username,password}`

Custom containers might live in private registries e.g., a Hugging Face
Space or a private AWS ECR, and a `username` and `password` might be
required. Given that the underlying API already supports those *only
with custom containers*, then this commit adds both
`--container-registry-username` and `--container-registry-password` in
such cases. More information about the API spec at
https://api.endpoints.huggingface.cloud/#post-/v2/endpoint/-namespace-

Signed-off-by: Alvaro Bartolome <36760800+alvarobartt@users.noreply.github.com>

* Fix `mypy`

Signed-off-by: Alvaro Bartolome <36760800+alvarobartt@users.noreply.github.com>

---------

Signed-off-by: Alvaro Bartolome <36760800+alvarobartt@users.noreply.github.com>

* fix(serialization): save_torch_state_dict cleanup regex silently matches/deletes unrelated files (#4857)

* fix(serialization): escape+anchor cleanup regex in save_torch_state_dict

The pre-save cleanup step in save_torch_state_dict() built its
"existing files" regex via filename_pattern.format(suffix=...) and
matched with re.match(). Two bugs compound:

1. Literal characters in the pattern (e.g. the "." in the default
   "model{suffix}.safetensors") were left unescaped, so "." acted as
   a regex wildcard.
2. re.match() only anchors at the start, so any file merely starting
   with the shard filename also matched.

Together this means files like model.safetensors.backup,
model.safetensors.dvc (DVC sidecar), or modelXsafetensors were
silently deleted on every save (failures are only logged as
warnings, never raised).

Fix: split the pattern on the {suffix} placeholder, re.escape() both
literal halves, and match with fullmatch() instead of match(). Added
a regression test with near-miss fixture files.

AI-assisted: used an AI coding agent to help locate/verify/write this
fix; I read, tested, and understand the change and take responsibility
for it. All existing + new serialization tests pass locally
(pytest tests/test_serialization.py -k save_torch_state_dict, 14 passed);
ruff format/check clean.

* Update tests/test_serialization.py

* Update src/huggingface_hub/serialization/_torch.py

---------

Co-authored-by: célina <hanouticelina@gmail.com>

* Bump the actions group with 4 updates (#4848)

Bumps the actions group with 4 updates: [huggingface/doc-builder/.github/workflows/build_main_documentation.yml](https://github.com/huggingface/doc-builder), [huggingface/doc-builder/.github/workflows/build_pr_documentation.yml](https://github.com/huggingface/doc-builder), [trufflesecurity/trufflehog](https://github.com/trufflesecurity/trufflehog) and [huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml](https://github.com/huggingface/doc-builder).


Updates `huggingface/doc-builder/.github/workflows/build_main_documentation.yml` from c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27 to cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3
- [Release notes](https://github.com/huggingface/doc-builder/releases)
- [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md)
- [Commits](https://github.com/huggingface/doc-builder/compare/c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27...cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3)

Updates `huggingface/doc-builder/.github/workflows/build_pr_documentation.yml` from c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27 to cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3
- [Release notes](https://github.com/huggingface/doc-builder/releases)
- [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md)
- [Commits](https://github.com/huggingface/doc-builder/compare/c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27...cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3)

Updates `trufflesecurity/trufflehog` from 3.97.1 to 3.97.2
- [Release notes](https://github.com/trufflesecurity/trufflehog/releases)
- [Commits](https://github.com/trufflesecurity/trufflehog/compare/20652fbbdefffcdaa493a5bf57ab2ac6b1db715b...cc1fe982afc515d2991365ce8d4d0dd07170fcad)

Updates `huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml` from c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27 to cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3
- [Release notes](https://github.com/huggingface/doc-builder/releases)
- [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md)
- [Commits](https://github.com/huggingface/doc-builder/compare/c2d27f6f231df3d1b07f3a7ea54fe3d0f8bc3e27...cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3)

---
updated-dependencies:
- dependency-name: huggingface/doc-builder/.github/workflows/build_main_documentation.yml
  dependency-version: cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3
  dependency-type: direct:production
  dependency-group: actions
- dependency-name: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml
  dependency-version: cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3
  dependency-type: direct:production
  dependency-group: actions
- dependency-name: trufflesecurity/trufflehog
  dependency-version: 3.97.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml
  dependency-version: cf20b09f274383f0d91a7055c5e43f0a2ab3d1a3
  dependency-type: direct:production
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Key the doc build concurrency group on the pull request number (#4876)

* [Buckets] Fix file metadata size on redirects (#4874)

* [RepoCard] Fix AttributeError when model-index has source: null (#4870)

* [Spaces] Fix AttributeError in SpaceRuntime when hardware is null (#4873)

* [CI] Make CI less flaky and stop 6h hung jobs (#4830)

* [CI] Make CI less flaky and stop 6h hung jobs

Follow-up to an investigation of the recent CI flakiness. All the failures
themselves come from a hub-ci incident (plain-LFS uploads return 403
AccessDenied on s3:PutObject), which is being handled by the infra team. This
commit fixes how our CI *reacts* to such an incident.

- `_upload_large_folder`: cap consecutive errors instead of retrying forever.
  A permanent error made the worker loop spin until GitHub's 6h job timeout
  (136k retries, 60MB of logs, thousands of requests/min at the Hub).
- Add `timeout-minutes: 30` and a `concurrency` group to `python-tests.yml`,
  and stop triggering the whole suite on the `assigned` PR event.
- Drop the `lfs` matrix entry: `TestHfLargefiles` is `@pytest.mark.skip`ped, so
  the job selected 1 test and skipped it (green but empty).
- Speed up `TestValidateUploadLimits`: 100k `MagicMock`s -> plain objects
  (85s -> 1s).
- De-flake `test_commit_to_repo_in_background` (wall-clock assertion) and
  `test_list_private_models` (shared `__DUMMY_HUB_USER__` recency page).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Apply suggestion from @Wauplin

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Serialization] Harden torch checkpoint deserialization (#4877)

* fix(security): case-insensitive safetensors extension check

`_is_safetensors` was case-sensitive, so `model.SAFETENSORS` fell through to
`torch.load(weights_only=False)`. On Windows/macOS, where globbing is
case-insensitive, this made the pickle path reachable automatically.

Refs V1 of the torch serialization security analysis.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): thread `safe` through the single-file loading path

`load_torch_model`'s single-file branch never passed `safe` to
`load_state_dict_from_file`, so `load_torch_model(model, "model.bin", safe=True)`
still executed the pickle on every OS. `load_state_dict_from_file` now accepts a
keyword-only `safe` parameter and routes to the safetensors loader when it is True.

Refs V3 of the torch serialization security analysis.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): default `load_state_dict_from_file` to `safe=True`

Router selection was derived from the filename and defaulted to pickle, i.e. a
denylist by omission: anything not spelled exactly `.safetensors` reached
`torch.load`. `safe=True` is now the default and means "always use the
safetensors loader", whatever the file is called; the filename becomes a hint
used only in the (already unsafe) `safe=False` branch.

BREAKING CHANGE: `load_state_dict_from_file("model.bin")` now raises. Pass
`safe=False` to load pickle checkpoints.

Refs V2 of the torch serialization security analysis.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): default `weights_only=True` for pickle checkpoints

Since torch 2.6, `torch.load` defaults to `weights_only=True`. Passing an
explicit `weights_only=False` downgraded that protection and made
huggingface_hub strictly less safe than calling `torch.load` directly. The
pickle path now only uses the unrestricted unpickler when the caller explicitly
passes `weights_only=False`.

BREAKING CHANGE: pickle checkpoints holding non-tensor globals (optimizer
states, full models) now need `weights_only=False` to load.

Refs V4 of the torch serialization security analysis.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): reject pickle `filename_pattern` with `safe=True`

`filename_pattern` used to silently override `safe`, so
`load_torch_model(model, dir, safe=True, filename_pattern=PYTORCH_WEIGHTS_FILE_PATTERN)`
loaded pickle shards with no warning. The conflict is now an explicit
`ValueError`, and the docstring no longer advertises the old precedence.

Refs V5 of the torch serialization security analysis.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): pass `safe` down to sharded checkpoint loading

`_load_sharded_checkpoint` picked the loader per shard from a second,
independent string check. It now takes an explicit `safe` parameter and forwards
it to `load_state_dict_from_file`, so shards obey the same guarantee as the
single-file path. The index extension guard stays a strict, case-sensitive
allowlist, documented against the case-insensitive routing hint.

Refs V6 of the torch serialization security analysis.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): report loaded keys from the shards, not from the index

`strict` validation and the returned `_IncompatibleKeys` were computed from
`index["weight_map"]`, an attacker-controlled file that does not necessarily
match the shard contents. A caller gating on `missing_keys`/`unexpected_keys`
could be told whatever the repo author wanted, independently of what was
actually loaded. Keys are now collected from the shards while loading, and
divergence from the index is warned about.

Refs V7 of the torch serialization security analysis.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): make `strict=True` work for multi-shard checkpoints

A single shard never holds all the model's keys, so passing `strict` to the
per-shard `model.load_state_dict` raised `RuntimeError: Missing key(s)` on the
first shard of every legitimate multi-shard checkpoint. `strict=True` could
therefore never succeed, which pushed callers to turn it off — exactly the
opposite of what a careful caller should do when loading untrusted weights.

Shards are now loaded with `strict=False` and strictness is enforced after the
loop against the keys the shards actually contained (V7), keeping the existing
pre-validation against the index as a fail-fast check.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): load safetensors first in a directory when `safe=False`

The docstring promises that `safe=False` tries safetensors "if they are
available" and only then falls back to pickle files, but the single-file
lookup globbed `*.bin` outright under `safe=False`: a directory holding only
`model.safetensors` raised `Directory ... does not contain a valid checkpoint`,
and a directory holding a lone `pytorch_model.bin` was handed to the safetensors
loader (`safe` was not forwarded), so it could not be loaded either.

The glob now looks for safetensors first and only falls back to `.bin` when
nothing was found and `safe=False`, and forwards `safe` to the loader so the
selected file is actually parsed with the matching format.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): validate the checkpoint index before using it

`json.load` on the index file was followed directly by `index["weight_map"]`,
so a crafted index surfaced as a raw traceback from attacker-controlled input:
a missing `weight_map` raised `KeyError`, a non-dict raised `AttributeError`,
non-string values raised `TypeError` in `PureWindowsPath(...)`/`.endswith`, and
unhashable ones raised `TypeError` in `set(...)`. There was also no size cap, so
a huge `index.json` was parsed into memory before anything else happened.

The index is now rejected with an actionable `ValueError` unless it is a JSON
object whose `weight_map` is an object of strings, and files above a few MB are
refused before being read.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(security): forward `mmap` in the single-file branch of `load_torch_model`

`load_torch_model(model, "…/model.bin", mmap=True)` was a silent no-op: the
sharded and in-directory branches forwarded `mmap`, the single-file one did not.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* test(security): regression tests for unsafe checkpoint loading

Extend the hackerone#141 regression class (renamed `TestCheckpointLoadingSecurity`) with high-level tests:

- a pickle named `model.safetensors` / `.SAFETENSORS` / `.SafeTensors` is never deserialized as pickle, asserted
  with a payload that touches a marker file when unpickled, plus a control proving the payload is live;
- `weights_only` default refuses non-tensor globals;
- neither an explicit `.bin` path nor a directory picks up a pickle under `safe=True`;
- `strict=True` on a legitimate multi-shard checkpoint succeeds;
- reported keys come from the shard contents, not from the index declaring them;
- `safe=False` still loads a safetensors file;
- a malformed index raises a clear `ValueError`.

All of them fail against the previous implementation.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* [Serialization] Address review feedback on checkpoint loading hardening

- Fall back to `.bin` whenever the safetensors glob did not yield exactly one
  file, not only when it yielded zero. A directory holding several safetensors
  files next to a single `pytorch_model.bin` regressed under `safe=False`.
- Drop the pre-load `strict` validation against the index. It made the
  attacker-controlled index authoritative again, in the fail-closed direction:
  a stale or crafted `weight_map` entry failed `strict=True` on a checkpoint
  whose shards matched the model perfectly. The post-load check against the
  real loaded keys is the only one left.
- Only warn about `torch.load` when `weights_only=False`. Under the new
  `weights_only=True` default torch uses its restricted unpickler, so the
  warning was both untrue and emitted once per shard.
- Document the `safe=True` / `weights_only=True` defaults in the serialization
  guide, mention the new `ValueError` in the `Raises:` section, and note that
  `strict` is checked after loading (the model may be partially updated).
- Tests: cover the `.bin` fallback with several safetensors files, `mmap`
  forwarding in the single-file branch, and `strict=True` against a lying
  index. Drop an unused `mocker` fixture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Serialization] Use `> [!WARNING]` callout syntax in the loading docs

The `<Tip>` MDX components are gone from `docs/source/en/`; the blockquote
alert syntax is the only one used now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Serialization] Warn when `weights_only=True` cannot be honored (torch < 1.13)

The pickle warning was keyed off the requested `weights_only`, but that flag is
only forwarded to `torch.load` on torch >= 1.13. On older versions (CI still
covers `torch~=1.11`) the restricted unpickler never runs, so the new
`weights_only=True` default silently unpickled without restriction and the
warning was skipped — a fail-open on the very default this PR introduces.

Key the warning off what is actually passed to `torch.load` instead, so it
fires both when the caller opted out and when torch is too old to honor the
opt-in. Docstrings now say `weights_only` has no effect below 1.13 rather than
"only supported in PyTorch >= 1.13", which read as if the argument were merely
unavailable.

Reported by Cursor Bugbot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Code <noreply@anthropic.com>

* feat(jobs): expose the network group on JobInfo (#4862)

The Hub returns `network: {group, aliases}` on a job, but `JobInfo` dropped it, so neither `inspect_job` nor `hf jobs inspect` could tell which group a job joined. Adds a `JobNetwork` dataclass and a `network` field on `JobInfo`, `None` for jobs started without a group.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* [Http] Fix race in get_session() creating multiple clients (#4868)

* [CLI] Make --format human override agent auto-detection (#4878)

* [CLI] Make --format human override agent auto-detection

* Update _output.py

Co-authored-by: Lucain <lucain@huggingface.co>

---------

Co-authored-by: Lucain <lucain@huggingface.co>

* [Serialization] Sync `_get_dtype_size` with safetensors (#4902)

Adds uint16/uint32/uint64, float8_e4m3fnuz, float8_e5m2fnuz,
float8_e8m0fnu, float4_e2m1fn_x2 and complex64 to the dtype size table.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Safetensors] Recognize proposed U3 metadata (#4898)

* [Safetensors] Recognize proposed U3 metadata

* Delete tests/test_safetensors_u3.py

---------

Co-authored-by: Lucain <lucainp@gmail.com>

* [Download] Fix silently disabled tree cache on long Windows paths (#4896)

* [Download] Support Windows long paths for the tree cache file

`_tree_cache_path()` joined `<folder>/trees/<commit_hash>.json` with no
Windows extended-length handling, unlike every sibling path in
`_local_folder.py`, `_snapshot_download.py` and `file_download.py`. On a deep
`local_dir` or `cache_dir` (path over 255 characters with the default
LongPathsEnabled=0) the write failed with WinError 206, `write_tree_cache`
swallowed it, and the next read missed, so the tree cache was silently
disabled and every download re-fetched the tree listing.

Apply the extended-length prefix in the one function both the reader and the
writer go through, so they agree on the on-disk path and the in-memory cache
key. Drive paths become `\\?\C:\...`; UNC paths become
`\\?\UNC\server\share\...`, because prefixing a UNC path verbatim produces
`\\?\\\server\...`, which Windows rejects.

Add two Windows-only tests: a disk round-trip parametrized at 260 (only the
JSON file exceeds the limit) and 300 (the `trees/` folder does too), and a
pure-function check of the drive, UNC, already-prefixed and short-path
spellings.

* [Download] Share a single extended-length path helper

The `\\?\` prefixing was inlined 6 times across the download stack (`_local_folder.py`,
`file_download.py`, `_snapshot_download.py`), so the tree cache fix added a 7th copy.
Move it to `utils/_paths.as_extended_path()` and reuse it everywhere, UNC handling included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Lucain Pouget <lucainp@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: make the sandbox security contract match the implementation (#4831)

The sandbox docs promise a few isolation properties we do not currently
deliver, and a user reading them could reasonably pick a pool for a
workload that needs a dedicated VM. This corrects the claims and adds an
explicit "Known limitations" section so the trade-off is visible at the
point of the decision.

Corrections in `concepts/sandbox.md`:

- The token is per *job*, not per sandbox. In a pool, one nonce is minted
  per host and every sandbox on it shares that token, which also gates the
  host's management routes -- so a leak is host-wide, not per-sandbox. Added
  a "Token scope" table and dropped "a leaked sandbox token compromises
  that one sandbox only".
- "Only the HF token used to spin-up the Job can access the sandbox" was
  imprecise in both directions: any holder of that token can derive it
  (that is the point of stateless reconnect), and the token is derived for
  whichever job carries the matching nonce label.
- The uid+Landlock guarantee list describes what code *inside* a sandbox
  can do directly. It says nothing about the root control plane, which
  acts on some names the sandbox controls. Said so, next to the list.
- The file API's `..` guarantee is lexical. Symlinks are a separate
  question and currently not handled; said so instead of implying the
  path is confined.
- "The cache is safe precisely because it is never trusted as truth" --
  it is trusted for the URL credentials get sent to, which is the part
  that matters. Replaced with what it is and isn't trusted for, plus the
  practical advice ($HF_HOME is sensitive; the key is not bound to
  endpoint/principal/namespace).
- "No secret is ever stored on a host" -- true for disk and job metadata,
  not for the host server's memory, which holds a pooled sandbox's env for
  its lifetime. Stated precisely.
- Dedicated mode isolates the job from everything outside it; inside the
  VM the workload and sbx-server share root. Added, since "right choice
  for untrusted code" invites reusing one sandbox across trust boundaries.

New "Known limitations" section covering the control-plane confused
deputies, label-based host discovery, Landlock's fail-open path, the
residual shared channels (loopback, UDP, /proc, /dev, IPC), missing
quotas, and the lifecycle gaps (`kill()` addressing, the idle watchdog vs
foreground commands, per-process `max_hosts`, `close()` cancelling
discovered hosts, silent teardown failures, unpinned binary). Plus a
per-actor threat-model table for both modes.

Docstrings updated to match: `_derive_sandbox_token`, `SandboxFiles`,
`SandboxProcess.kill`, `SandboxPool`, `_discover_hosts`, `close`,
`idle_timeout` and `max_hosts`. `_discover_hosts` claimed to match hosts
on image/flavor; it matches on labels only.

This is documentation only -- no behaviour change. Each subsequent
hardening PR should delete the corresponding entry from Known
limitations.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(sandbox): use each pooled sandbox's own capability token (#4832)

The pool host's token authorized every sandbox on the host plus its
management routes, and it was the only credential the client had. So every
per-sandbox call -- and `proxy_headers`, which is handed to browsers and
WebSocket clients -- offered authority over the whole host.

sbx-server 0.6.0 mints a capability token per sandbox. This uses it:

- `_SandboxServer.request`/`stream` take an optional `sandbox_token` that
  overrides `X-Sandbox-Token` for one request; the client-level header keeps
  carrying the host credential for management calls.
- `Sandbox` holds its own token and sends it on every scoped call, including
  the `DELETE` that kills a pooled sandbox.
- `SandboxPool.create()` keeps the token from the create response.
- `Sandbox.connect("<host>.<id>")` recovers it via
  `GET /v1/sandboxes/{id}/token`, so reattaching stays stateless.
- `proxy_headers` hands out the sandbox's token, not the host's.

Hosts running an older server return no token and have no recovery route;
those sandboxes fall back to the host credential, so a pool booted before
the server upgrade keeps working. Requires the server change to be deployed
first.

Also makes the test fake enforce the same scoping as the real server --
host token on management routes, the sandbox's own token on scoped ones --
rather than accepting one value everywhere. That laxness is exactly how a
client/server protocol mismatch survives a green suite, and it caught two
of this change's own call sites while writing it.

Validation:
- 44 tests pass, including new ones asserting per-sandbox operations present
  the scoped token, `proxy_headers` does not leak the host one, and the
  older-server fallback still works.
- Driven end-to-end against a real sbx-server 0.6.0 in a container: A's
  token cannot exec, read, write, delete or proxy into B, cannot create,
  list or delete sandboxes, and cannot recover any token; the host token
  recovers A's token and matches.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(sandbox): validate a pool host before sending it a credential (#4834)

Pool hosts were found by filtering Jobs on labels, and adopted on that
basis alone. Labels are set by whoever creates the Job, and the nonce the
host token is derived from is itself a public label -- so any namespace
member who can create a Job could publish one carrying another user's pool
labels and that host's nonce, and the victim's client would adopt it and
send it the token derived for the *real* host. The Jobs proxy strips
`Authorization` but forwards `X-Sandbox-Token`, so the impostor receives
exactly the capability the real host accepts.

Nothing in the old path bound a Job to its creator, image, flavor, command
or URL. `_discover_hosts` even claimed in its docstring to match on
image/flavor; it did not.

Adoption now requires everything the *backend* asserts about a Job to line
up, not just the labels its creator chose:

- `initiator` -- the load-bearing check, and the one field `run_job` gives no
  way to set, so it cannot be forged client-side;
- owner namespace, image (normalized for registry prefixes and case), flavor,
  and that the command is the sandbox bootstrap;
- exactly one exposed URL, over HTTPS, whose hostname is derived from *this*
  job's id -- so a Job cannot name where the credentials should go.

New `adopt_hosts` argument on `SandboxPool` and `SandboxPool.connect`:
`"own"` (default) adopts only this principal's hosts, `"namespace"` restores
cross-user sharing for namespaces whose members trust each other, `"never"`
disables adoption. A refusal names the opt-in, so someone legitimately
sharing hosts is not left guessing.

`SandboxPool.connect`'s cold path gets the same treatment: it rebuilds the
pool's entire configuration from a discovered host's spec, so an unvalidated
match would let a Job in the namespace choose the image the pool's next
hosts boot. `_connect_host` addresses a job the caller named explicitly, so
ownership there is the caller's assertion, but the URL and bootstrap checks
still apply.

Separately, the HF bearer is no longer frozen when the transport is built.
A host lives up to 24h, and an OIDC/OAuth credential can expire inside that
window, after which every request failed even though the handle was fine. An
`httpx.Auth` now reads it per request. The `X-Sandbox-Token` is deliberately
*not* re-derived: the server holds the value derived from the bearer that
created the job, so re-deriving from a rotated bearer would produce a token
it has never seen. The two credentials answer different gates and only the
bearer needs refreshing. (A host created under one HF token therefore stays
reachable only by holders of that token -- inherent to deriving the
credential, and now stated in the docs.)

The discovery tests were part of the problem: their Job fixtures set only
labels, so they asserted that a label match is sufficient. They now describe
a Job the way the Jobs API really does, and a new suite covers the hostile
cases -- an impostor's initiator, a mismatched image/flavor/command/namespace,
a URL belonging to another job, a plain-HTTP URL, extra exposed ports, and a
missing initiator (refused rather than assumed).

Validation: 55 tests pass, up from 44. Writing them caught two real bugs in
this change -- `urlparse` lowercases hostnames, which broke the URL check on
a mixed-case job id, and a fixture whose host had no capacity.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(sandbox): bind the pool cache to the context that wrote it (#4838)

The local pool cache is not passive data. `SandboxPool.connect()`
rebuilds a pool from `$HF_HOME/sandbox/pools/<pool-id>.json` with no
HTTP at all, `_seed_hosts_from_cache` turns each entry into a
`_SandboxServer` from the cached `base_url` and `nonce`, and the first
thing that happens to it is an authenticated `POST /v1/sandboxes`
carrying the HF bearer and the token derived for that nonce. Two
problems followed from that.

The file was keyed by pool id alone, and its `namespace` field was read
as configuration rather than compared against what the caller asked
for -- `namespace=cache.namespace if namespace is None else namespace`.
So a pool cached for org-A and then reached with
`connect(pool, namespace="org-B")` routed the workload, and its env, to
org-A's hosts. That needs no attacker: one credential and two orgs is
an ordinary state of affairs, and nothing in the flow ever contradicts
the file. Nor was the cache bound to the endpoint or the credential, so
two users sharing a machine shared each other's entries.

And the URL was taken on trust. A cache file (or anything that can
write `$HF_HOME`) could name `http://127.0.0.1:1234` and receive both
credentials on the first create -- with `follow_redirects=True` on the
client, a legitimate-looking HTTPS URL could bounce them elsewhere too.
`verified=False` only means "drop the host if it fails"; by then the
credentials have been sent.

What changed:

- The cache lives at `pools/<context>/<pool-id>.json`, where `<context>`
  is a digest of endpoint, credential fingerprint and namespace. The
  same digest is stored in the payload and re-checked on read, so an
  entry moved or copied into a matching directory is still a miss. The
  credential appears only as a `sha256` fingerprint; the token is never
  written. Fingerprinting is local on purpose: resolving the credential
  to a user id would cost a `whoami` and defeat the point of the cache.
- `connect()` therefore honours `namespace=` instead of inheriting the
  cached one, and a mismatch on any of the three is a plain cache miss
  (debug log, cold path, never an exception).
- A cached `base_url` is only used if it is the HTTPS jobs-proxy URL of
  the job that same entry names, reusing the reasoning already applied
  to a discovered host's URL -- both now go through
  `_server_host_rejection`. An entry naming anything else is dropped
  from the file.
- Every field is type- and range-checked on read. `dataclass` enforces
  nothing, so a well-shaped file with `"capacity": "50"` used to parse
  happily and blow up much later on `host.capacity - host.live`, as an
  uncaught `TypeError` inside `create()`. It is now the documented miss.
- `follow_redirects=False` on the credentialed client. There is no
  redirect in this protocol; the proxy serves the in-job server.
- Freshness carve-out: an entry written by this principal within 15
  minutes is still credited with no round-trip (the `pool create` ->
  `create --pool` case the cache exists for). Past that, `inspect_job`
  has to confirm the job is running, labelled for this pool, carrying
  that nonce, adoptable, and on that URL before the transport is built.

Local hygiene in the same pass, all of it in the same "the file is not
authenticated, so don't hand it authority" spirit:

- cache dirs `0700` (tightened even if they predate this), and the
  atomic write goes through `tempfile.mkstemp` -- `0600` regardless of
  umask, unguessable name, removed on any failure -- instead of a
  predictable `<name>.<pid>.tmp` opened at `0666 & ~umask`.
- `files.download()` writes to an `O_EXCL|O_NOFOLLOW`, `0600` temp file
  next to the destination and `os.replace`s it into place. Before, it
  wrote in place: an interrupted download left a truncated file at the
  final name, and `open(path, "wb")` followed a symlink planted there.
  `rename` does not follow symlinks, so such a destination is now
  replaced rather than written through.
- `files.upload()` opens the file once and both `fstat`s and streams
  that descriptor, instead of `stat`ing the path and reopening it.

Validation (`PYTHONPATH=src python -m pytest tests/test_sandbox.py -q`):
83 passed, including new cases for cross-principal / cross-namespace /
cross-endpoint misses, moved and renamed files, mistyped fields, URL
admission, the freshness carve-out, redirect handling and the two
transfer paths. Two of them assert against a recording server that a
cache naming `http://127.0.0.1:<port>`, and a cross-namespace
`connect()`, result in zero requests reaching it. `ruff check` and
`ruff format --check` are clean on the three files.

Latency, modelling a 100 ms round-trip and timing
`connect(pool) + create()` (median of 5, cross-process CLI shape):
fresh cache hit 141 ms before / 141 ms after, cold path 560 ms before /
557 ms after, and an entry past the TTL 249 ms (one added round-trip).
The common case is unchanged; only stale entries pay.

Behaviour changes: existing cache files are all misses (new key, new
layout, `version` 2), so the first run after upgrading takes the cold
path once. A pool cached under an explicit `namespace=` is only found
again when the same `namespace=` is passed. A rotated credential reads
as a different principal and gets a miss. Anyone relying on a partial
file being left behind by a failed download no longer gets one.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(sandbox): bound what a transfer or a command's output costs the client (#4839)

Four unbounded allocations in the caller's process, each proportional to
something the sandbox controls.

**Output was accumulated even when a callback was consuming it.** `run()`
appended every chunk to `stdout_parts`/`stderr_parts` regardless, so a
streaming consumer still paid full memory for the command's entire output,
and a runaway command (a loop printing to stdout) was an unbounded
allocation. New `capture_output=False` hands output to the callbacks and
drops it; the default now raises above a ceiling instead of being
OOM-killed with no explanation, and the error names the flag to use.

An explicit flag rather than "stop capturing when a callback is present":
silently changing what `result.stdout` contains based on another argument is
a footgun, and code that passes a callback *and* reads `result.stdout` today
keeps working.

**A parallel read doubled peak memory.** `_read_ranges` collected every
chunk into a list and the caller then joined it, so a 2 GB file peaked at
about 4 GB. It is a generator now, yielding `(offset, bytes)` as ranges
land: `read()` places each chunk into one preallocated buffer, `download()`
seeks and writes it out, and both drop it immediately. `read()` also refuses
above a ceiling and points at `download()`, which streams to disk.

**A parallel upload read the whole file into memory.** `upload()` did
`f.read()` above the threshold -- 10 GB resident for a 10 GB file. Each
worker now `pread`s its own range off the shared descriptor (`pread`, not
`seek`+`read`, because the workers share the descriptor and a seeking read
would race with its siblings).

**And it left a stale tail.** A ranged write does not truncate, so
overwriting a larger file with a smaller one left the old bytes past the new
end. Each upload's final chunk now sends the intended total size, which the
server truncates to.

Also follows the server's new listing pagination, so `files.list()` still
returns one complete list however the directory is chunked on the wire --
and the fake server now serves `/files/list` at all, which it never did, so
that path had no coverage.

Validation: 88 tests pass, up from 83. New coverage for opting out of
capture, the runaway-output ceiling and its escape hatch, the read ceiling,
and cursor-following in `list()`.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(sandbox): decide host teardown per host, and report it honestly (#4840)

Three ways a pool's lifecycle could destroy or leak something without
saying so.

**`close()` cancelled hosts it had merely discovered.** Ownership was
decided once per *handle* (`_owns_hosts = not _connect_mode`), while
`_discover_hosts` appended hosts found via labels -- possibly started by
another process, or another member of the namespace, and possibly serving
their sandboxes -- into the same list. So a plain `with SandboxPool(...)`
block that happened to adopt a colleague's warm host terminated it, and
their sandboxes with it, on exit.

Ownership now belongs to the host: `_SandboxServer.owned` is set only by
`_boot_host`, and `close()` cancels only those. A discovered host has its
local HTTP client released and is otherwise left alone.

**A `create()` in flight could outlive `close()`.** `close()` snapshotted
the host list and set `_closed`, but a concurrent `create()` that had
already passed its `_closed` check could finish booting afterwards and
append to a list nobody would ever tear down. `create()` now registers
itself under the same lock as the closed check, and `close()` waits (with a
timeout, so a wedged create cannot hang process exit) for those to drain.

**A failed cancellation was reported as success.** `close()` logged a
warning, deleted the cache entry, and returned normally -- leaving a host
running and billing with no local record of it. It now raises, naming the
jobs and the command to cancel them, and *keeps* their cache entries so they
stay discoverable. Reached through `__exit__` with an exception already in
flight, the failure is logged instead, so it cannot mask the original error.

Also: `max_hosts` was compared against `len(self._hosts)`, an in-process
count, so two processes using one pool could each stay under the cap while
together exceeding it -- for a documented cost ceiling, with real money
attached. It now counts every host running for the pool, found via labels.
That narrows the window considerably but does not close it: two processes can
still list simultaneously and both decide there is room. A hard cap has to be
enforced where Jobs are created, and the docs now say best-effort rather than
implying a guarantee.

The test stubs were part of the problem: `_boot_host` is monkeypatched in
several tests, and a stub that does not mark the host as owned makes
`close()` correctly refuse to cancel it. They go through a `_booted_server`
helper now, which is the contract `_boot_host` guarantees.

Validation: 94 tests pass, up from 88. New coverage for a booted host being
cancelled, a discovered host being released but not cancelled, a failed
cancellation raising while keeping the cache entry, a teardown failure not
masking the original exception, `close()` waiting for an in-flight create and
tearing down what it booted, and `max_hosts` counting another process' hosts.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(sandbox): address background processes by their server-assigned id (#4836)

`SandboxProcess.kill()` sent the OS pid. The server matches on the opaque id
it assigned (`p-3`) and answered 200 regardless, so the call stopped
nothing, reported success, and left the process running -- which also kept
the sandbox non-idle, so idle eviction never fired and the job kept billing.

`SandboxProcess` now carries `id` as its identifier and `kill()` uses it.
`pid` stays, documented as observational: it is useful for correlating with
`ps` inside the sandbox, but the OS may reuse it. `kill()` returns whether
this call is what stopped the process, so a caller can tell "stopped it"
from "it had already exited" -- both are successes, but they are different
answers.

A process from a host running a server that predates opaque ids has
`id=None`; `kill()` raises with an explanation rather than sending a pid the
new server will reject.

The test fake is the reason this survived a green suite: it deleted by pid,
so it implemented a protocol nobody serves. It now deletes by opaque id only
and 400s on a pid, exactly like the real server -- and there is a test that
asserts the pid path *fails*, so the fake going lax again would be caught
rather than silently re-enabling the bug.

Also corrects the `processes()` docstring: the server keeps a bounded number
of finished processes, not all of them until the sandbox is deleted.

Requires the server change to be deployed first. Validation: 60 tests pass,
up from 55.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [sandbox audit] Pin the sandbox server binary by digest and verify it before running it (#4837)

* fix(sandbox): pin the server binary by digest and verify it before running it

Every sandbox job ran a bootstrap that fetched a *mutable* public object,
`chmod +x`ed it and `exec`ed it -- as root, as PID 1, holding the derived
sandbox token:

    d=/tmp/.sbx-server
    if command -v wget ...; then wget -q -O "$d" "$SBX_SERVER_URL"
    ...
    chmod +x "$d"
    exec "$d"

`$SBX_SERVER_URL` was `/buckets/huggingface/sbx-server/resolve/sbx-server`,
the alias the publish workflow overwrites on every release. Nothing in the
chain checked a version, a digest or a signature, so the client had no way
to know which bytes it had just made PID 1. A single write to that bucket
path reaches every sandbox of every user on their next cold start. Nothing
suggests that happened -- the point is that the design offered no guarantee
about the next fetch, even though the immutable artefact it needed was
already being published alongside the alias.

Related: `wait_ready` polled `/health` and read nothing out of it. A pool
host keeps the binary it downloaded at boot for up to 24h, so a client and a
server from different releases meet routinely, and the per-sandbox
capability token added recently is exactly the kind of change that fails
quietly -- a 403 on an unrelated route, several calls later.

What changed:

- `SANDBOX_SERVER_SHA256` and `SANDBOX_SERVER_VERSION` pin one exact server
  build. `SBX_SERVER_URL` now names the digest-addressed bucket object
  (`sbx-server-<sha256>`), not the alias: a fetch that names the content it
  wants cannot be answered with different content.
- The bootstrap verifies the download against that digest and refuses to run
  it on a mismatch. The check happens **before** `chmod +x` -- an unverified
  file that is already executable is one slip away from being executed.
- `sha256sum` is not guaranteed in an arbitrary image, so `openssl dgst
  -sha256` is the second try. With neither, the job **refuses** rather than
  running unverified code. `SBX_ALLOW_UNVERIFIED_SERVER=1` (passed through
  `env=`) is the only way past it, and it can only skip a *missing tool*,
  never a failed comparison.
- The bucket-mount fallback (for images with no wget or curl) reads the
  digest-named file and goes through the same check. It previously read the
  alias, so leaving it alone would have left the hole open on exactly the
  images least likely to notice.
- `wait_ready` reads `protocol` from `/health` and raises a `SandboxError`
  naming host recycling when the server is older than this client needs.
  Permissive in one direction only: a newer server that still reports a
  protocol this client knows is accepted; an older one, or one that reports
  none, is refused.

On failing closed: a survey of the plausible base images found nothing that
lacks both hash tools. `python:3.12` and `node:22` have sha256sum and
openssl; `python:3.12-slim` has both but neither wget nor curl;
`ubuntu:26.04`, `alpine:3.22` and `busybox` have sha256sum. So the default
is expected to cost nothing in practice, and the escape hatch is there for
the image we have not thought of.

Validation:

`PYTHONPATH=src pytest tests/test_sandbox.py -q` -> 64 passed (+9).
`ruff check` / `ruff format --check` clean; `ty check src` reports the same
4 pre-existing diagnostics as on the base.

The bootstrap was also run by hand -- the real musl binary, real images, a
local `python3 -m http.server`, never the real bucket -- across 14 shapes.
Every good case exec'd (`sbx-server 0.6.0 listening on 0.0.0.0:49912`);
every corrupted case exited 1 without exec'ing:

    1  python:3.12   wget, good download          exit=0  EXEC'D
    2  python:3.12   wget, CORRUPTED download     exit=1  did NOT exec
    3  python:3.12   curl only, good              exit=0  EXEC'D
    4  python:3.12   curl only, CORRUPTED         exit=1  did NOT exec
    5  python:3.12   no wget/curl, mount good     exit=0  EXEC'D
    6  python:3.12   no wget/curl, mount CORRUPT  exit=1  did NOT exec
    7  python:3.12   no sha256sum/openssl         exit=1  did NOT exec
    8  python:3.12   no hash tool + opt-out       exit=0  EXEC'D
    9  python:3.12   openssl only, good           exit=0  EXEC'D
    10 python:3.12   openssl only, CORRUPTED      exit=1  did NOT exec
    11 alpine:3.22   busybox, good                exit=0  EXEC'D
    12 alpine:3.22   busybox, CORRUPTED           exit=1  did NOT exec
    13 ubuntu:26.04  no dl tool, mount good       exit=0  EXEC'D
    14 ubuntu:26.04  no dl tool, mount CORRUPT    exit=1  did NOT exec

A mismatch is diagnosed rather than silent, which matters because
`_tail_job_logs` is what surfaces it in the `SandboxError`:

    sbx: sandbox server digest mismatch: got de7f7486...,
    sbx: expected 501290ea...
    sbx: refusing to run it. Upgrade huggingface_hub if this client is
    sbx: pinned to a digest that is no longer published.

Not validated: no live HF Jobs run. The digest constant is also a local
musl build of the companion server branch, so it has to be replaced…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant