Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
160 commits
Select commit Hold shift + click to select a range
5ec8e27
Post-release: bump version to 1.30.0.dev0 (#4763)
huggingface-hub-bot[bot] Aug 27, 2026
ae20a50
[CI] Enable enforcement in close-unscoped-community-prs workflow (#4766)
hanouticelina Aug 27, 2026
97655e8
Align `hf jobs scheduled ls` filtering with `hf jobs ls` (`--status`/…
moon-bot-app[bot] Aug 28, 2026
1de4817
[Download] Follow redirects between Hub hosts when resolving files (#…
hanouticelina Aug 28, 2026
4cc16bc
[Download] A `ResolvedRevision` only pins the repo it was resolved fo…
Wauplin Aug 28, 2026
94cb25d
[CI] Fix torch_1.11 case-arm label to match renamed matrix entry (#4776)
MohammedAlkindi Aug 31, 2026
6a2eb09
[Sandbox] Document experimental status (#4783)
Wauplin Sep 1, 2026
f0a5c6d
[Inference Providers] Do not require the conversational tag for chat-…
hanouticelina Sep 1, 2026
176d4f3
[CI] Disable auto close (#4800)
Wauplin Sep 2, 2026
b84090b
[Release] Consolidate the HF CLI skill sync into the release workflow…
hanouticelina Sep 3, 2026
c804cdd
Post-release: bump version to 1.31.0.dev0 (#4806)
huggingface-hub-bot[bot] Sep 3, 2026
15bb844
Bump the actions group across 1 directory with 5 updates (#4798)
dependabot[bot] Sep 4, 2026
deb9777
[Docs] Clarify Jobs argument forwarding and align examples (#4809)
davanstrien Sep 4, 2026
634ad9b
[httpx migration] Expose `httpx` as a `huggingface_hub` submodule (#4…
Wauplin Sep 7, 2026
f1bb85f
[hf_api] Make CommitInfo copy-able and pickle-able (#4822)
Wauplin Sep 7, 2026
21e687e
[Download] Prevent cached file copies during local-dir dry runs (#4817)
wakamex Sep 7, 2026
6c8c56b
Fix: keep single-quoted values verbatim in load_dotenv (#4827)
AnishPatel526 Sep 8, 2026
a9b9ac6
[CLI] Fix 'hf buckets rm --recursive' deleting lexical siblings of th…
Wauplin Sep 8, 2026
01fe4bc
[Sandbox] Support custom Job labels (#4824)
Wauplin Sep 8, 2026
204b717
[CLI] Don't let `hf update` self-upgrade a pip install on Windows (#4…
Wauplin Sep 8, 2026
aa3c2fd
[HfFileSystem] Reject unsafe paths before local downloads (#4828)
Wauplin Sep 8, 2026
9d7bbfe
[Download] Share retry handling for stream entry and body failures (#…
Wauplin Sep 8, 2026
8934ec0
[Download] Tolerate missing HEAD Content-Length (#4805)
Wauplin Sep 8, 2026
b9c14d9
[Download] Write cache ref file atomically in `snapshot_download` (#4…
Wauplin Sep 8, 2026
063b37b
[CLI] Raise explicit error for shell-script extensions on Windows (#4…
hanouticelina Sep 9, 2026
6cae778
[Jobs] Add network groups to hf jobs run (#4833)
co42 Sep 9, 2026
22fe960
Fix dotenv parser truncating unquoted values containing '#' (#4842)
BenYang12 Sep 9, 2026
c4f076c
(LFS)Fix SliceFileObj.__iter__ yielding only the first 4MB chunk (#4844)
BenYang12 Sep 9, 2026
f148666
[CLI] Fix truncated command descriptions in the CLI reference (#4849)
hanouticelina Sep 9, 2026
ff4292b
Post-release: bump version to 1.32.0.dev0 (#4856)
huggingface-hub-bot[bot] Sep 10, 2026
a2af2e0
docs(jobs): scope network groups to the namespace and resource group …
co42 Sep 10, 2026
0790f7b
[Community] Fix docstring for DiscussionComment.last_edited_by (#4864)
Saniyagupte Sep 10, 2026
bf85073
fix(sandbox): preserve terminal command results (#4851)
evalstate Sep 10, 2026
819cd77
[CLI] Read a UV script's `[tool.hf-jobs]` launch config (opus-generat…
Wauplin Sep 10, 2026
b853ec6
Add `--container-registry-{username,password}` (#4854)
alvarobartt Sep 10, 2026
e203a7f
fix(serialization): save_torch_state_dict cleanup regex silently matc…
g0rdonL Sep 10, 2026
3493edb
Bump the actions group with 4 updates (#4848)
dependabot[bot] Sep 10, 2026
4110cac
Key the doc build concurrency group on the pull request number (#4876)
albertvillanova Sep 11, 2026
edb2c83
[Buckets] Fix file metadata size on redirects (#4874)
davanstrien Sep 11, 2026
5694eda
[RepoCard] Fix AttributeError when model-index has source: null (#4870)
Saniyagupte Sep 11, 2026
b2678ec
[Spaces] Fix AttributeError in SpaceRuntime when hardware is null (#4…
Saniyagupte Sep 11, 2026
8a408cd
[CI] Make CI less flaky and stop 6h hung jobs (#4830)
Wauplin Sep 11, 2026
dd6919c
[Serialization] Harden torch checkpoint deserialization (#4877)
Wauplin Sep 11, 2026
9588035
feat(jobs): expose the network group on JobInfo (#4862)
co42 Sep 11, 2026
129bbb5
[Http] Fix race in get_session() creating multiple clients (#4868)
hanouticelina Sep 11, 2026
98e2db8
[CLI] Make --format human override agent auto-detection (#4878)
hanouticelina Sep 14, 2026
c5ff040
[Serialization] Sync `_get_dtype_size` with safetensors (#4902)
Wauplin Sep 14, 2026
f1e70a2
[Safetensors] Recognize proposed U3 metadata (#4898)
7dsolv Sep 14, 2026
fb21643
[Download] Fix silently disabled tree cache on long Windows paths (#4…
askalf Sep 14, 2026
490f06e
docs: make the sandbox security contract match the implementation (#4…
Wauplin Sep 14, 2026
1fb5b3b
feat(sandbox): use each pooled sandbox's own capability token (#4832)
Wauplin Sep 14, 2026
ef2662f
fix(sandbox): validate a pool host before sending it a credential (#4…
Wauplin Sep 14, 2026
40efc5e
fix(sandbox): bind the pool cache to the context that wrote it (#4838)
Wauplin Sep 14, 2026
0613ead
fix(sandbox): bound what a transfer or a command's output costs the c…
Wauplin Sep 14, 2026
c7e60d8
fix(sandbox): decide host teardown per host, and report it honestly (…
Wauplin Sep 14, 2026
ec61993
fix(sandbox): address background processes by their server-assigned i…
Wauplin Sep 14, 2026
0219ad5
[sandbox audit] Pin the sandbox server binary by digest and verify it…
Wauplin Sep 14, 2026
1173050
[sandbox audit] Keep secret values out of argv in the CLI and fix the…
Wauplin Sep 14, 2026
48cdc03
Fix doubled colon in snapshot download transfer bar description (#4890)
sandeepb2013 Sep 14, 2026
a3b62f6
[CLI] Add `hf --skills` as an alias for `hf skills preview` (#4906)
Wauplin Sep 14, 2026
bd9ece0
[CLI] Support file URIs in hf cache rm (#4847)
hanouticelina Sep 14, 2026
0a57df9
[Core] Restrict repo_id validation regex to ASCII word characters (#4…
bodapatisaikrishna Sep 15, 2026
af6b41e
[Core] Reject path traversal via embedded ".." segments in path_in_re…
bodapatisaikrishna Sep 15, 2026
2075fd6
[Cache] Support kernel repos in `hf cache` commands (#4905)
hanouticelina Sep 15, 2026
5a9cdda
[Core] Speed up package imports (#4914)
Wauplin Sep 15, 2026
6e3d21f
[Cache] Add cross-repo shared blob store (#4498)
hanouticelina Sep 17, 2026
f94ea55
[Buckets] Send mtime when copying files (#4920)
Wauplin Sep 17, 2026
d7b62c7
[CLI] Fix card data serialization (#4882)
davanstrien Sep 17, 2026
79dd41e
[Docs] Fix hf discussions info options that do not exist (#4886)
rishabhsai Sep 17, 2026
b064a14
[Download] Send X-HF-Download-Counter header on download calls (#4613)
Wauplin Sep 17, 2026
e9c55ce
Post-release: bump version to 1.33.0.dev0 (#4926)
huggingface-hub-bot[bot] Sep 17, 2026
618250b
[Download] Publish symlink support only after probing (#4923)
Wauplin Sep 17, 2026
77b6b2b
[Core] Skip invalid eval results entries instead of raising (#4924)
davanstrien Sep 17, 2026
34ae660
[Cache] Use posix paths for scan_cache_dir ref names (#4925)
gokay-ai Sep 18, 2026
0ba990a
[Download] Measure http_get progress from the initial file position (…
Wauplin Sep 18, 2026
cea607a
[Serialization] Fix device type when map_location is None (#4938)
Wauplin Sep 18, 2026
83ba61a
[CLI] Honor --format / --json / -q on jobs run commands (#4936)
Wauplin Sep 18, 2026
8c9546f
[Quality] Replace `re.match` with `re.fullmatch` / `re.search` and ba…
Wauplin Sep 18, 2026
22b5e87
[Docs] Sync provider task table with the PROVIDERS registry (#4930)
rishabhsai Sep 18, 2026
8efc0a9
[HfFileSystem] Make lpath optional in get_file (fsspec>=2026.9.0) (#4…
assafvayner Sep 21, 2026
4c80e7e
Fix `format_timesince` showing "60 seconds ago" / "60 minutes ago" / …
Sreekant13 Sep 21, 2026
55668e2
Fix Windows conftest erroring test subsets on lazy-loaded file_downlo…
dltsum Sep 21, 2026
7ce68b9
[Webhooks] Allow `bucket` as a watched item type (#4958)
davanstrien Sep 21, 2026
7025391
[MCP] Follow nextCursor when listing tools in MCPClient (#4960)
Kayvan-Zahiri Sep 22, 2026
80aef09
Align the `hf` package's python_requires with huggingface_hub (#4959)
sharziki Sep 22, 2026
94cf392
Remove dots from allowed characters in job labels. (#4962)
evalstate Sep 22, 2026
8d4fba6
[CLI] Honor `CLAUDE_CONFIG_DIR` in `hf skills add -g --claude` (#4957)
hanouticelina Sep 22, 2026
aa8742c
[CLI] Always install skills for Claude Code, deprecate --claude (#4967)
hanouticelina Sep 22, 2026
ce69fdb
[Buckets] Raise when a batch reports failed operations (#4954)
hanouticelina Sep 22, 2026
0f88f32
[Download] Extend tmp .incomplete path on Windows when too long (#4978)
Wauplin Sep 23, 2026
7532272
Fix HfFileSystem.get_file with a bare local filename (#4981)
davanstrien Sep 23, 2026
da5e244
Scope GITHUB_TOKEN permissions per job (#4982)
hf-security-analysis[bot] Sep 23, 2026
1e0c305
Send PrivateLink config under privateService in create_inference_endp…
co42 Sep 23, 2026
b904b94
Apply the repo_id length limit to the namespace as well (#4988)
anishmehta24 Sep 24, 2026
ac3ed2f
[Xet] Show Validating progress during shard finalization (#4478)
seanses Sep 24, 2026
fa26f34
[CI] Support major (pre-)releases in release workflow (#4991)
Wauplin Sep 24, 2026
c2e6e0a
Post-release: bump version to 1.34.0.dev0 (#4993)
huggingface-hub-bot[bot] Sep 24, 2026
9f51867
[Release] Prepare huggingface_hub 2.0 (#4994)
Wauplin Sep 24, 2026
89ae51f
unbump
Wauplin Sep 24, 2026
f5b624c
Post-release: bump version to 2.1.0.dev0 (#4999)
huggingface-hub-bot[bot] Sep 24, 2026
32456ab
[Buckets] Make include/exclude patterns case-sensitive on all platfor…
Wauplin Sep 25, 2026
c749dfa
Support secrets on Job-triggered webhooks (#5009)
moon-bot-app[bot] Sep 25, 2026
ee4f802
[HTTP] Fix repo_id in errors for query strings and download URLs (#5008)
Wauplin Sep 25, 2026
79874bc
[Buckets] Fix copy_files treating lexical siblings as an existing des…
AnishPatel526 Sep 25, 2026
ff0ccab
[CI] Bump doc-builder workflows pin to 63ec6f1 (#5010)
Wauplin Sep 25, 2026
34e8baa
[docs] xet caching and concurrency updates (#4883)
cbensimon Sep 25, 2026
62a1f2f
[HfFileSystem] Download Xet files with hf_xet in get_file (#4990)
hanouticelina Sep 25, 2026
e155ad3
[docs] Fix hf.cos typo in repository guide link (#5016)
pratikgx Sep 28, 2026
a076863
[datasets sql] Run the DuckDB CLI subprocess with UTF-8 encoding (#5020)
825pranav Sep 28, 2026
b15915b
[Auth] Store token names as UTF-8 instead of locale encoding (#5021)
Evolian-o Sep 28, 2026
41222c5
Fix scalar tags splitting into single-character list in card front ma…
zeenat28-ui Sep 28, 2026
14ee250
[CLI] Hint to use --global after local 'hf skills add/update' (#5024)
Wauplin Sep 28, 2026
f151320
[Sandbox] Require scoped credentials for pooled sandboxes (#4782)
Wauplin Sep 28, 2026
f64c26e
[Dataclasses] Accept bare typing.List/Dict/Set in @strict validation …
anishmehta24 Sep 29, 2026
7caeebb
Suggest `hf auth login` when `hf auth whoami` fails (#5033)
moon-bot-app[bot] Sep 29, 2026
5583746
[CLI] Align 'hf auth whoami' not-logged-in message with 'hf auth toke…
Wauplin Sep 29, 2026
862f23d
[Sandbox] Pin sbx-server 0.7.0 (protocol 3) (#5028)
Wauplin Sep 29, 2026
1b8b2fa
[CLI] Escape control characters in listings and table output (#5029)
Wauplin Sep 29, 2026
548f7e3
[Jobs] Allow rescheduling scheduled jobs (#5026)
Wauplin Sep 29, 2026
3fd18f2
[Jobs] Manage public and private exposed ports (#5027)
Wauplin Sep 29, 2026
231a2eb
[Jobs] Add rerun and retry attempts (#5025)
Wauplin Sep 29, 2026
d889712
[Tests] Reduce redundant Hub calls in repo settings and move tests (#…
Wauplin Sep 29, 2026
cc2f368
[Utils] Retry the first pagination page (#5035)
Wauplin Sep 29, 2026
1092497
[Spaces] Update docs for cpu-basic gating and dev mode disabling (#5037)
Wauplin Sep 29, 2026
013afa7
[CLI] Raise if a bare --secrets NAME is not set locally in hf jobs (#…
Wauplin Sep 30, 2026
83a714e
[CLI] Update hf upload help text after upload_large_folder removal (#…
davanstrien Sep 30, 2026
881a6c4
[Docs] Fix links in i18n issue template (#5051)
pratikgx Oct 1, 2026
60fdb08
[Docs] Fix duplicated words in docstring and comments (#5052)
Ad1th Oct 1, 2026
b5e3a12
[HfApi] Support resource_group_id in duplicate_repo (#5038)
Wauplin Oct 1, 2026
3bb08e2
[Endpoints] Migrate catalog to the new /api/v1 API (#4928)
Wauplin Oct 1, 2026
4c88ea6
[Repos] Warn when duplicated repo files are still being copied (#5041)
Wauplin Oct 1, 2026
604252e
[HfApi] Add visibility to create_bucket (#5053)
hanouticelina Oct 1, 2026
0b26b50
[Spaces] Add get_zero_gpu_quota and hf spaces zero-gpu-quota (#5040)
Wauplin Oct 1, 2026
ff61343
Post-release: bump version to 2.2.0.dev0 (#5054)
huggingface-hub-bot[bot] Oct 1, 2026
b3fa3a4
[CLI] Rename 'hf spaces zero-gpu-quota' to 'hf spaces zero-gpu quota'…
Wauplin Oct 1, 2026
7da19ff
[CI] Split tests into api / transfer / inference suites (#5042)
Wauplin Oct 1, 2026
7a54109
[Tests] Always clean up staging repos and buckets created by tests (#…
Wauplin Oct 1, 2026
1a76ad2
[CI] Reduce flaky tests: isolate jedi cache, sleep before read-after-…
Wauplin Oct 1, 2026
c31df86
[Bugfix] Fix wrong exception type and unbound variable guard (#5063)
vector15-05 Oct 2, 2026
37ce057
[CLI] Upload existing paths with glob characters as-is in hf upload (…
Ganesh2325 Oct 2, 2026
0ce4427
[Download] Warn when allow/ignore patterns match no files in snapshot…
Ad1th Oct 2, 2026
a625317
[Repocard] Respect configured endpoint when validating cards (#5069)
Wauplin Oct 2, 2026
d711944
[CLI] Read env/secrets files as UTF-8 and strip the BOM (#5066)
hanouticelina Oct 2, 2026
42f9085
[Setup] Install hf-xet on Windows ARM64 (#5074)
angt Oct 5, 2026
d699913
[Docs] Fix HTTP migration and async session references (#5079)
Yuilona Oct 5, 2026
bc9b691
Bump doc-builder workflow pin (#5087)
paulinebm Oct 5, 2026
3b05c63
[Jobs] Ignore 406 from metrics endpoint for finished jobs (#5091)
hanouticelina Oct 6, 2026
e817417
Bump doc-builder pin to 9fc8a41 in build_documentation.yaml (#5093)
paulinebm Oct 6, 2026
9071b17
[Jobs] Fix no-follow logs hanging while the job keeps printing (#5092)
hanouticelina Oct 6, 2026
68becf4
[CommitScheduler] Fix race that skips the first scheduled push (#5086)
hanouticelina Oct 6, 2026
ff4c989
[CLI] Print [] from hf jobs stats --json when no jobs are running (#5…
hanouticelina Oct 6, 2026
83ce8b0
[Download] Consider port when following /resolve redirects (#4942)
assafvayner Oct 6, 2026
a5277cf
[CLI] Snapshot by default in hf jobs stats, add -f/--follow (#5090)
hanouticelina Oct 6, 2026
14a48e7
[HfFileSystem] fix `ls()` on new empty bucket (#5097)
lhoestq Oct 7, 2026
38e0b3c
[HfFileSystem] Replace cached listings instead of appending to them (…
ChihebBENCHEIKH1 Oct 7, 2026
af74167
[Tests] Wait for bucket listings after writes (#5095)
hanouticelina Oct 7, 2026
cf992a9
[HfFileSystem] Make refresh=True clear cached not-found results (#5106)
hanouticelina Oct 7, 2026
bb3a875
[Docs] Fix from_pretrained integration example (#5105)
Ayushdevo Oct 7, 2026
8abbfe5
Fix race in `_get_version` that makes concurrent downloads skip hf_xe…
BramVanroy Oct 7, 2026
3cfa593
Guard upload_pr_documentation workflow to upstream repo only
g0rdonL Oct 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(sandbox): decide host teardown per host, and report it honestly (h…
…uggingface#4840)

Three ways a pool's lifecycle could destroy or leak something without
saying so.

**`close()` cancelled hosts it had merely discovered.** Ownership was
decided once per *handle* (`_owns_hosts = not _connect_mode`), while
`_discover_hosts` appended hosts found via labels -- possibly started by
another process, or another member of the namespace, and possibly serving
their sandboxes -- into the same list. So a plain `with SandboxPool(...)`
block that happened to adopt a colleague's warm host terminated it, and
their sandboxes with it, on exit.

Ownership now belongs to the host: `_SandboxServer.owned` is set only by
`_boot_host`, and `close()` cancels only those. A discovered host has its
local HTTP client released and is otherwise left alone.

**A `create()` in flight could outlive `close()`.** `close()` snapshotted
the host list and set `_closed`, but a concurrent `create()` that had
already passed its `_closed` check could finish booting afterwards and
append to a list nobody would ever tear down. `create()` now registers
itself under the same lock as the closed check, and `close()` waits (with a
timeout, so a wedged create cannot hang process exit) for those to drain.

**A failed cancellation was reported as success.** `close()` logged a
warning, deleted the cache entry, and returned normally -- leaving a host
running and billing with no local record of it. It now raises, naming the
jobs and the command to cancel them, and *keeps* their cache entries so they
stay discoverable. Reached through `__exit__` with an exception already in
flight, the failure is logged instead, so it cannot mask the original error.

Also: `max_hosts` was compared against `len(self._hosts)`, an in-process
count, so two processes using one pool could each stay under the cap while
together exceeding it -- for a documented cost ceiling, with real money
attached. It now counts every host running for the pool, found via labels.
That narrows the window considerably but does not close it: two processes can
still list simultaneously and both decide there is room. A hard cap has to be
enforced where Jobs are created, and the docs now say best-effort rather than
implying a guarantee.

The test stubs were part of the problem: `_boot_host` is monkeypatched in
several tests, and a stub that does not mark the host as owned makes
`close()` correctly refuse to cancel it. They go through a `_booted_server`
helper now, which is the contract `_boot_host` guarantees.

Validation: 94 tests pass, up from 88. New coverage for a booted host being
cancelled, a discovered host being released but not cancelled, a failed
cancellation raising while keeping the cache entry, a teardown failure not
masking the original exception, `close()` waiting for an in-flight create and
tearing down what it booted, and `max_hosts` counting another process' hosts.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
  • Loading branch information
Wauplin and claude authored Sep 14, 2026
commit c7e60d81a4959841ea9008191841d632510056b4
4 changes: 1 addition & 3 deletions docs/source/en/concepts/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,9 +251,7 @@ This section is deliberately exhaustive rather than reassuring: if you are decid
- **A detached descendant can outlive `kill()`.** `SandboxProcess.kill()` signals the command's process group; a descendant that calls `setsid()` leaves it. Deleting the sandbox (pool) or the job (dedicated) does terminate everything. Use `timeout=` if you need a hard bound.
- **`SandboxProcess.kill()` does not currently stop the process** — the client sends the OS pid where the server expects its own opaque process id, and the server answers `200` either way. Until this is fixed, stop background work by deleting the sandbox.
- **A long foreground command can trip the idle watchdog.** `idle_timeout` counts API requests, and a running foreground command is not counted as activity, so a command that runs longer than `idle_timeout` without other API traffic can have its sandbox shut down under it. Raise `idle_timeout` (or pass `None`) for long single commands.
- **`max_hosts` is a per-process cap.** Two processes using the same pool each count only their own hosts, so the global number of host Jobs can exceed it. Per-host `sandboxes_per_host` *is* enforced server-side.
- **`close()` on a pool may cancel hosts it did not create.** A pool handle that discovered a warm host started by another process (or another user in the same namespace) will cancel it on exit, killing that host's sandboxes. Use `SandboxPool.connect()` for handles that must not tear hosts down, and prefer distinct `name=` values for independent pools.
- **A failed teardown is reported as success.** `close()` logs a warning and returns normally if it cannot cancel a host job, and removes its cache entry — so a host can keep running and billing. Check `hf jobs ps --label hf-sandbox=1` if teardown matters to you.
- **`max_hosts` is best-effort, not a hard cap.** It is now checked against every host running for the pool (found via labels), not just the ones the current process tracks — but two processes can still count simultaneously and both decide there is room. A hard cap has to be enforced where Jobs are created, not by clients racing to count them. Per-host `sandboxes_per_host` *is* enforced server-side.
- **The server binary is not pinned or verified.** Each job downloads `sbx-server` from a mutable public bucket path and executes it as root without checking a digest or signature.
- **The server's HTTP front end is minimal.** No connection cap and no read deadlines, so a slow-request flood can exhaust its threads; a request with a malformed `Content-Length` is treated as having no body. Reaching it still requires passing the proxy gate.

Expand Down
180 changes: 145 additions & 35 deletions src/huggingface_hub/_sandbox.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,11 @@

DEFAULT_SANDBOXES_PER_HOST = 50

# How long `close()` waits for in-flight creates before giving up on them. A
# wedged create must not hang process exit, but returning immediately would let
# a host it is still booting escape teardown.
_CLOSE_DRAIN_TIMEOUT = 30.0

# Ceiling on the output `run()` will accumulate for its result. Without one, a
# runaway command (a loop printing to stdout, say) is an unbounded allocation in
# the caller's process -- and it happened even when a callback was already
Expand Down Expand Up @@ -554,6 +559,12 @@ def __init__(
# False only for hosts rebuilt from the (best-effort) pool cache: their job may
# be gone, so the first failed request drops them instead of failing the create.
self.verified = True
# Whether *this* process started this host job, and may therefore cancel
# it. Ownership belongs to the host, not to the pool handle: a handle used
# to cancel hosts it had merely discovered via labels, which may have been
# started by another process -- or another member of the namespace -- and
# may be serving their sandboxes.
self.owned = False
# httpx.Client is thread-safe, so a single client serves both sequential requests
# and the concurrent workers used for parallel file transfers / many sandboxes.
self._client = httpx.Client(
Expand Down Expand Up @@ -1372,7 +1383,14 @@ def __init__(
# Whether close()/`__exit__` cancels the host jobs. True for a pool we created; False
# for a connect()'d handle, which only releases its HTTP clients on exit — the shared
# hosts (possibly serving other clients) are left running, like Sandbox.connect().
# Kept as the default for hosts this handle boots (a connect()'d handle
# boots nothing during construction), but teardown is decided per host.
self._owns_hosts = not _connect_mode
# In-flight create/boot operations. `close()` waits for these to drain, so
# a host booted by a create() that was already past its closed check
# cannot land in a list nobody will ever tear down.
self._in_flight = 0
self._drained = threading.Condition(self._lock)
# Job ids of cached hosts we found dead this session; pruned from the cache on save.
self._dead_host_ids: set[str] = set()

Expand Down Expand Up @@ -1473,7 +1491,7 @@ def warm(self, num_hosts: int = 1) -> List[str]:
with self._lock:
shortfall = num_hosts - len(self._hosts)
if shortfall > 0:
booted = self._provision_hosts(shortfall)
booted = self._provision_hosts(shortfall, allow_partial=True)
with self._lock:
self._hosts.extend(booted)
with self._warmup_lock:
Expand Down Expand Up @@ -1510,8 +1528,25 @@ def create(
delivered to the host server at creation (never stored in the host job), so
it doesn't appear in any job's metadata.
"""
if self._closed:
raise SandboxError("This SandboxPool is closed.")
with self._lock:
if self._closed:
raise SandboxError("This SandboxPool is closed.")
# Registered under the lock together with the closed check, so
# `close()` cannot snapshot the host list between the two.
self._in_flight += 1
try:
return self._create_locked(env, idle_timeout, forward_hf_token)
finally:
with self._lock:
self._in_flight -= 1
self._drained.notify_all()

def _create_locked(
self,
env: dict[str, Any] | None,
idle_timeout: int | float | str | None,
forward_hf_token: bool,
) -> "Sandbox":
sandbox_env = dict(env or {})
if forward_hf_token:
sandbox_env["HF_TOKEN"] = _effective_token(self._api)
Expand Down Expand Up @@ -1611,36 +1646,72 @@ def close(self) -> None:
`with` block must not tear them down. Terminate a connected pool's hosts explicitly
with `hf sandbox pool delete <id>`.

> [!WARNING]
> Two rough edges here. Ownership is decided per *handle*, not per host: a pool we
> created also cancels hosts it merely discovered via labels, which may have been
> started by another process — or another member of the namespace — and may be
> serving their sandboxes. And a host job that cannot be cancelled is only logged as
> a warning while its cache entry is removed, so this can return successfully while
> a host keeps running and billing. Check `hf jobs ps --label hf-sandbox=1` if
> teardown matters to you.
Only hosts *this handle started* are cancelled. A host discovered via labels may
be serving another process' sandboxes, so it is released rather than terminated.

Raises [`SandboxError`] if a host job could not be cancelled, naming the jobs that
are still running — they keep billing, and their cache entries are kept so they stay
discoverable. When `close()` is reached through `__exit__` with an exception already
in flight, the failure is logged instead, so it cannot mask the original error.
"""
with self._lock:
if self._closed:
return
self._closed = True
# Wait for in-flight creates so a host booted by one that was already
# past its closed check lands in `self._hosts` before the snapshot,
# rather than being orphaned. Bounded, because a wedged create must
# not hang process exit.
drain_timeout = self._start_timeout + _CLOSE_DRAIN_TIMEOUT
if self._in_flight and not self._drained.wait_for(lambda: self._in_flight == 0, timeout=drain_timeout):
logger.warning(
f"Pool '{self.name}': {self._in_flight} operation(s) still in flight after "
f"{drain_timeout}s; any host they boot will keep running."
)
hosts = self._hosts
self._hosts = []
self._closed = True

survivors: List[_SandboxServer] = []
failures: List[str] = []
for host in hosts:
try:
if self._owns_hosts:
# Only what this handle started. A discovered host may be serving
# someone else's sandboxes, and cancelling it kills them.
if host.owned:
host.cancel_job()
except Exception as e:
# Keep it in the cache: that record is the user's only handle on a
# job that is still running and still billing.
survivors.append(host)
failures.append(f"{host.job_id} ({e})")
logger.warning(f"Failed to cancel sandbox host {host.job_id}: {e}")
finally:
host.close()
if self._owns_hosts:

if survivors:
# Rewrite rather than delete, so the survivors stay discoverable.
self._save_cache(hosts=survivors)
elif self._owns_hosts:
# The pool's hosts are gone; don't leave a stale entry.
delete_pool_cache(self.name, self._cache_context)
if failures:
raise SandboxError(
f"Pool '{self.name}' could not terminate host job(s): {'; '.join(failures)}. "
"They are still running and billing; cancel them with `hf jobs cancel <id>`."
)

def __enter__(self) -> "SandboxPool":
return self

def __exit__(self, *exc_info) -> None:
self.close()
try:
self.close()
except SandboxError:
# Never let a teardown failure mask the exception that is already
# propagating out of the `with` block.
if exc_info[0] is None:
raise
logger.warning(f"Pool '{self.name}' teardown failed while handling another error.", exc_info=True)

# ------------------------------------------------------------------ internals

Expand Down Expand Up @@ -1676,7 +1747,7 @@ def _ensure_warmed_up(self) -> None:
with self._lock:
shortfall = target - len(self._hosts)
if shortfall > 0:
booted = self._provision_hosts(shortfall)
booted = self._provision_hosts(shortfall, allow_partial=True)
with self._lock:
self._hosts.extend(booted)
self._warmed_up = True
Expand Down Expand Up @@ -1812,11 +1883,9 @@ def _discover_hosts(self) -> None:
instead of booting a new one. Hosts are found via job labels; each adopted
host's free capacity is read from the server, so packing stays accurate.

Matching is on labels only — `hf-sandbox-mode=pool` plus `hf-sandbox-pool=<name>` in
the target namespace. Job labels are set by whoever creates the job, so in a
namespace whose members do not all trust each other, any of them can publish a job
this will adopt. Adopted hosts are also not owned by this handle even though
`close()` currently cancels them; see `close()`.
Labels select candidates; creator, job specification and proxy URL admission
checks decide which candidates can receive credentials. Adopted hosts are released
by close(), not cancelled, because another handle may still be using them.
"""
known = {host.job_id for host in self._hosts}
matches = [
Expand Down Expand Up @@ -1851,16 +1920,49 @@ def _discover_hosts(self) -> None:
else:
self._hosts.append(server)

def _provision_hosts(self, num_new: int) -> List[_SandboxServer]:
"""Boot `num_new` host jobs in parallel, respecting `max_hosts`."""
def _live_host_count(self) -> int:
"""How many hosts are running for this pool, not just the ones we track.

`max_hosts` is documented as a cost ceiling, but it was compared against
`len(self._hosts)` -- an in-process count. Two processes using the same
pool each counted only their own hosts, so the number of billed host jobs
could quietly exceed the cap by a factor of however many processes were
involved.

Counting via labels narrows that considerably. It does not close it: two
processes can still list simultaneously and both decide there is room. A
hard cap has to be enforced where the jobs are created, not by the
clients racing to count them.
"""
with self._lock:
current = len(self._hosts)
if self.max_hosts is not None and current + num_new > self.max_hosts:
allowed = self.max_hosts - current
raise SandboxError(
f"Pool needs {num_new} more host(s) but max_hosts={self.max_hosts} "
f"allows only {max(0, allowed)} more. Raise max_hosts or kill some sandboxes."
)
tracked = {host.job_id for host in self._hosts}
try:
for job in self._api.list_jobs(
status="RUNNING",
labels={MODE_LABEL: MODE_POOL, POOL_LABEL: self.name},
namespace=self._namespace,
):
tracked.add(job.id)
except Exception as e:
# Fall back to the in-process count rather than refusing to boot.
logger.debug(f"Could not count live hosts for pool '{self.name}': {e}")
return len(tracked)

def _provision_hosts(self, num_new: int, *, allow_partial: bool = False) -> List[_SandboxServer]:
"""Boot `num_new` host jobs in parallel, respecting `max_hosts`."""
if self.max_hosts is not None:
current = self._live_host_count()
if allow_partial:
num_new = min(num_new, max(0, self.max_hosts - current))
if current + num_new > self.max_hosts:
allowed = self.max_hosts - current
raise SandboxError(
f"Pool needs {num_new} more host(s) but max_hosts={self.max_hosts} "
f"allows only {max(0, allowed)} more ({current} already running for this pool). "
"Raise max_hosts or kill some sandboxes."
)
if num_new == 0:
return []
# Boot in parallel, but collect every result before re-raising: if one boot fails,
# the others may have already started billable host jobs, so cancel those instead of
# leaking them (executor.map would surface the first error and drop the rest).
Expand Down Expand Up @@ -1946,6 +2048,8 @@ def _boot_host(self) -> _SandboxServer:
if server is not None:
server.close()
raise
# We started this job, so this handle is the one that should end it.
server.owned = self._owns_hosts
return server

def _create_one(self, host: "_SandboxServer", env: dict[str, Any], idle_secs: int | None) -> "Sandbox | None":
Expand Down Expand Up @@ -2096,10 +2200,16 @@ def _confirm_cached_host(self, ch: CachedHost) -> bool:
self._dead_host_ids.add(ch.job_id)
return False

def _save_cache(self) -> None:
"""Persist the pool config + current hosts (with their live counts) for next time."""
def _save_cache(self, hosts: "List[_SandboxServer] | None" = None) -> None:
"""Persist the pool config + hosts (with their live counts) for next time.

`hosts` overrides the tracked list, for the one case where they differ:
`close()` persisting the hosts it could *not* cancel, since that record is
the caller's only handle on a job that is still billing.
"""
with self._lock:
hosts = [
source = self._hosts if hosts is None else hosts
entries = [
CachedHost(
job_id=host.job_id,
owner=host.owner,
Expand All @@ -2109,7 +2219,7 @@ def _save_cache(self) -> None:
live=host.live,
updated_at=time.time(),
)
for host in self._hosts
for host in source
]
dead = set(self._dead_host_ids)
save_pool_cache(
Expand All @@ -2120,7 +2230,7 @@ def _save_cache(self) -> None:
sandboxes_per_host=self.sandboxes_per_host,
max_hosts=self.max_hosts,
idle_timeout=_duration_to_secs(self._idle_timeout) if self._idle_timeout is not None else None,
hosts=hosts,
hosts=entries,
dead_host_ids=dead,
)

Expand Down
Loading