Repository navigation
[sandbox audit] Bound what a transfer or a command's output costs the client #4839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
fix(sandbox): bound what a transfer or a command's output costs the c…
…lient Four unbounded allocations in the caller's process, each proportional to something the sandbox controls. **Output was accumulated even when a callback was consuming it.** `run()` appended every chunk to `stdout_parts`/`stderr_parts` regardless, so a streaming consumer still paid full memory for the command's entire output, and a runaway command (a loop printing to stdout) was an unbounded allocation. New `capture_output=False` hands output to the callbacks and drops it; the default now raises above a ceiling instead of being OOM-killed with no explanation, and the error names the flag to use. An explicit flag rather than "stop capturing when a callback is present": silently changing what `result.stdout` contains based on another argument is a footgun, and code that passes a callback *and* reads `result.stdout` today keeps working. **A parallel read doubled peak memory.** `_read_ranges` collected every chunk into a list and the caller then joined it, so a 2 GB file peaked at about 4 GB. It is a generator now, yielding `(offset, bytes)` as ranges land: `read()` places each chunk into one preallocated buffer, `download()` seeks and writes it out, and both drop it immediately. `read()` also refuses above a ceiling and points at `download()`, which streams to disk. **A parallel upload read the whole file into memory.** `upload()` did `f.read()` above the threshold -- 10 GB resident for a 10 GB file. Each worker now `pread`s its own range off the shared descriptor (`pread`, not `seek`+`read`, because the workers share the descriptor and a seeking read would race with its siblings). **And it left a stale tail.** A ranged write does not truncate, so overwriting a larger file with a smaller one left the old bytes past the new end. Each upload's final chunk now sends the intended total size, which the server truncates to. Also follows the server's new listing pagination, so `files.list()` still returns one complete list however the directory is chunked on the wire -- and the fake server now serves `/files/list` at all, which it never did, so that path had no coverage. Validation: 88 tests pass, up from 83. New coverage for opting out of capture, the runaway-output ceiling and its escape hatch, the read ceiling, and cursor-following in `list()`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Loading branch information
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Zero-size reads drop file content
Medium Severity
read()now always fetches withlengthequal tostat().size. Files that report size 0 but still have content, such as procfs or sysfs in dedicated mode, either come back empty or raiseSandboxErrorinstead of returning the bytes.download()of the same path still streams without a length, so the two APIs disagree.Additional Locations (1)
src/huggingface_hub/_sandbox.py#L352-L364Reviewed by Cursor Bugbot for commit eb4db0a. Configure here.