Skip to content

[Download] Measure http_get progress from the initial file position - #4908

Closed
askalf wants to merge 5 commits into
huggingface:mainfrom
sprayberry-code:fix/http-get-initial-file-position
Closed

askalf wants to merge 5 commits into
huggingface:mainfrom
sprayberry-code:fix/http-get-initial-file-position

Conversation

@askalf

@askalf askalf commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • http_get writes into a caller-supplied file object but measured its own progress with the file's absolute position (temp_file.tell()), not with the bytes this call downloaded.
  • HfFileSystem.get_file (src/huggingface_hub/hf_file_system.py:1123) explicitly supports handing over an already-positioned file object — it saves initial_pos = outfile.tell() before the call and seeks back to it afterwards. For any such file the caller's own bytes were counted as downloaded bytes.
  • Consequence 1 — spurious failure: the final size check at file_download.py:480 compared expected_size against temp_file.tell(), so an 8-byte head offset on a 100-byte file raised Consistency check failed: file should be of size 100 but has size 108. Every fs.get_file(..., lpath=<open file at pos>0>) / fs.download(...) into a positioned handle fails.
  • Consequence 2 — silent data loss: the Range-ignored recovery path at file_download.py:400 did temp_file.seek(0); temp_file.truncate(), destroying the caller's own bytes rather than rewinding to where this download started.
  • Fix: measure from new_resume_size (bytes downloaded by this call, already maintained at :460 for the retry path) and rewind to temp_file.tell() - resume_size. 9 insertions / 4 deletions in one source file; no behaviour change at offset 0.
$ # ---- BASE (48cdc033) + the thirteen new tests ----
$ PYTHONPATH=src python -m pytest tests/test_file_download.py -k "TestHttpGet" -q
=========== 13 failed, 10 passed, 66 deselected, 1 warning in 20.26s ===========
$ #   all thirteen added tests fail; the ten that pass are the pre-existing
$ #   offset-0 tests, which are the controls and stay green on both arms.

$ # ---- WITH THE FIX ----
$ PYTHONPATH=src python -m pytest tests/test_file_download.py -k "TestHttpGet" -q
================ 23 passed, 66 deselected, 1 warning in 19.52s =================

Decisions

new_resume_size already exists and is already the right quantity: it is initialised to resume_size, incremented by len(chunk) for each chunk written, and is what the retry recursion passes as the next resume_size. The function therefore already maintains "bytes of this file obtained so far, counted from the download's own origin" — the consistency check was simply reading a different variable, so no new state is introduced. temp_file.tell() - resume_size is the download's origin by construction: on the reset path, resume_size bytes of this download are on disk immediately after the position where it began, and at offset 0 this evaluates to seek(0) exactly as before, so the ordinary hf_hub_download path is byte-for-byte unchanged. The docstring change states the contract HfFileSystem.get_file already relies on but that http_get never documented, which is why the two call sites had drifted apart.

Alternatives considered and rejected: making http_get seek to 0 itself or reject positioned files would break HfFileSystem.get_file's deliberate initial_pos save/restore; having get_file pass resume_size=initial_pos would feed the wrong byte range into the Range header, since resume_size means bytes of this file already downloaded; capturing a second initial_pos variable inside http_get and threading it through the retry recursion is equivalent but adds state for no benefit; fixing only the consistency check would leave the silent-truncation half of the bug (the more damaging one) in place.

One behaviour change outside the stated bug, found during test verification and worth flagging: a resume_size larger than the file's contents, answered by a Range-ignoring 200, now raises ValueError: negative seek value where base silently restarted the download from seek(0). This is not reachable from any in-repo caller today (both call sites pass resume_size=0, and the retry recursion measures its resume_size on the same file object), but a loud failure replacing a silent restart is the fail-safe direction, so it's covered by a test rather than left unstated.

Not run: the Linux CI test matrix (needs a self-hosted runner group not available outside the org) and any network-dependent test that needs a staging-Hub token — the HTTP layer is mocked in every test added here.

AI assistance: this bug was found and the fix and tests were drafted with AI tooling in my workflow; the tests and checks above were executed as pasted. I'm responsible for the change and will handle review feedback.


Note

Medium Risk
Touches core HTTP download retry/truncate logic used by Hub downloads and fsspec; behavior at offset 0 is preserved but edge cases (invalid resume_size) now raise instead of silently restarting.

Overview
Fixes http_get so download progress and validation use bytes written by this call (new_resume_size), not the file handle’s absolute position. That unblocks HfFileSystem.get_file, which passes an already-positioned file and restores the cursor afterward.

Before: a non-zero start offset caused false consistency failures (e.g. expected 100, reported 108) and, when a server ignored Range and returned 200, seek(0) + truncate() could wipe caller-owned prefix data.

After: the final size check compares expected_size to new_resume_size; the Range-ignored path rewinds with seek(tell() - resume_size) so only this download’s partial bytes are cleared. The temp_file docstring documents append-from-current-position behavior. Offset-0 callers are unchanged.

Adds broad TestHttpGet coverage (positioned BytesIO/real files, resume, double Range-ignored retries, error messages) and lets the test helper inject a pre-built temp_file.

Reviewed by Cursor Bugbot for commit d8c0262. Bugbot is set up for automated code reviews on this repo. Configure here.

`http_get` writes into a caller-provided file object, but tracked progress with the
file's absolute position. `HfFileSystem.get_file` explicitly supports handing over an
already-positioned file (it saves `initial_pos` and restores it afterwards), so the
caller's own bytes were counted as downloaded bytes:

- the final size check compared `expected_size` against `temp_file.tell()`, raising a
  spurious "Consistency check failed" for any non-zero starting offset;
- the Range-ignored reset did `seek(0) + truncate()`, destroying the caller's data
  instead of rewinding to where this download began.

Both now work off the bytes downloaded by this call.
Cover the initial-position boundaries left unpinned: a real on-disk file
handle (the shape HfFileSystem.get_file passes), two consecutive
Range-ignored resets, offset 1, a falsy expected_size of 0, an omitted
expected_size back-filled from the response, and a resume_size one past
expected_size.
…heck

The zero-size case returned at the `resume_size == expected_size` early
return before reaching either changed line, so it passed identically on
the base implementation. Drive it past the early return with a
`resume_size` ahead of `expected_size` so the consistency check runs and
reports the bytes downloaded here rather than the absolute file position.
@Wauplin

Wauplin commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Thanks a lot @askalf for this report! Your analysis is spot-on and very well written - the two consequences (spurious Consistency check failed and the seek(0); truncate() wiping the caller's own bytes) both reproduce exactly as you described. I confirmed the first one end-to-end against the Hub with a plain append-mode file handle:

fs = HfFileSystem()
with open("out.bin", "ab") as f:  # file already has 13 bytes
    fs.get_file("openai-community/gpt2/config.json", f)
# OSError: Consistency check failed: file should be of size 665 but has size 678

I've landed the fix separately in #4937 as a minimal 9/-4 change, without the 250 lines of tests - http_get is a hot path and we prefer to keep the test suite focused, the existing TestHttpGet cases already cover the offset-0 behaviour. I also kept the max(..., 0) clamp on the rewind instead of letting it raise ValueError: negative seek value, so a bogus resume_size keeps today's silent-restart behaviour rather than becoming a new exception in a download path.

Closing this one in favour of #4937, but the credit is yours - thanks again for the thorough write-up, it made the review very easy.

@Wauplin Wauplin closed this Sep 18, 2026
@askalf

askalf commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for confirming it end to end and for landing it in #4937. The minimal shape is the right call for http_get; noted for anything I bring here next, tests included. The max(..., 0) clamp on the rewind is the better default too. Appreciate the quick turnaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants