Repository navigation
Conversation
`http_get` writes into a caller-provided file object, but tracked progress with the file's absolute position. `HfFileSystem.get_file` explicitly supports handing over an already-positioned file (it saves `initial_pos` and restores it afterwards), so the caller's own bytes were counted as downloaded bytes: - the final size check compared `expected_size` against `temp_file.tell()`, raising a spurious "Consistency check failed" for any non-zero starting offset; - the Range-ignored reset did `seek(0) + truncate()`, destroying the caller's data instead of rewinding to where this download began. Both now work off the bytes downloaded by this call.
Cover the initial-position boundaries left unpinned: a real on-disk file handle (the shape HfFileSystem.get_file passes), two consecutive Range-ignored resets, offset 1, a falsy expected_size of 0, an omitted expected_size back-filled from the response, and a resume_size one past expected_size.
…heck The zero-size case returned at the `resume_size == expected_size` early return before reaching either changed line, so it passed identically on the base implementation. Drive it past the early return with a `resume_size` ahead of `expected_size` so the consistency check runs and reports the bytes downloaded here rather than the absolute file position.
…a positioned file
|
Thanks a lot @askalf for this report! Your analysis is spot-on and very well written - the two consequences (spurious fs = HfFileSystem()
with open("out.bin", "ab") as f: # file already has 13 bytes
fs.get_file("openai-community/gpt2/config.json", f)
# OSError: Consistency check failed: file should be of size 665 but has size 678I've landed the fix separately in #4937 as a minimal 9/-4 change, without the 250 lines of tests - Closing this one in favour of #4937, but the credit is yours - thanks again for the thorough write-up, it made the review very easy. |
|
Thanks for confirming it end to end and for landing it in #4937. The minimal shape is the right call for |
Summary
http_getwrites into a caller-supplied file object but measured its own progress with the file's absolute position (temp_file.tell()), not with the bytes this call downloaded.HfFileSystem.get_file(src/huggingface_hub/hf_file_system.py:1123) explicitly supports handing over an already-positioned file object — it savesinitial_pos = outfile.tell()before the call and seeks back to it afterwards. For any such file the caller's own bytes were counted as downloaded bytes.file_download.py:480comparedexpected_sizeagainsttemp_file.tell(), so an 8-byte head offset on a 100-byte file raisedConsistency check failed: file should be of size 100 but has size 108. Everyfs.get_file(..., lpath=<open file at pos>0>)/fs.download(...)into a positioned handle fails.file_download.py:400didtemp_file.seek(0); temp_file.truncate(), destroying the caller's own bytes rather than rewinding to where this download started.new_resume_size(bytes downloaded by this call, already maintained at:460for the retry path) and rewind totemp_file.tell() - resume_size. 9 insertions / 4 deletions in one source file; no behaviour change at offset 0.Decisions
new_resume_sizealready exists and is already the right quantity: it is initialised toresume_size, incremented bylen(chunk)for each chunk written, and is what the retry recursion passes as the nextresume_size. The function therefore already maintains "bytes of this file obtained so far, counted from the download's own origin" — the consistency check was simply reading a different variable, so no new state is introduced.temp_file.tell() - resume_sizeis the download's origin by construction: on the reset path,resume_sizebytes of this download are on disk immediately after the position where it began, and at offset 0 this evaluates toseek(0)exactly as before, so the ordinaryhf_hub_downloadpath is byte-for-byte unchanged. The docstring change states the contractHfFileSystem.get_filealready relies on but thathttp_getnever documented, which is why the two call sites had drifted apart.Alternatives considered and rejected: making
http_getseek to 0 itself or reject positioned files would breakHfFileSystem.get_file's deliberateinitial_possave/restore; havingget_filepassresume_size=initial_poswould feed the wrong byte range into theRangeheader, sinceresume_sizemeans bytes of this file already downloaded; capturing a secondinitial_posvariable insidehttp_getand threading it through the retry recursion is equivalent but adds state for no benefit; fixing only the consistency check would leave the silent-truncation half of the bug (the more damaging one) in place.One behaviour change outside the stated bug, found during test verification and worth flagging: a
resume_sizelarger than the file's contents, answered by a Range-ignoring200, now raisesValueError: negative seek valuewhere base silently restarted the download fromseek(0). This is not reachable from any in-repo caller today (both call sites passresume_size=0, and the retry recursion measures itsresume_sizeon the same file object), but a loud failure replacing a silent restart is the fail-safe direction, so it's covered by a test rather than left unstated.Not run: the Linux CI test matrix (needs a self-hosted runner group not available outside the org) and any network-dependent test that needs a staging-Hub token — the HTTP layer is mocked in every test added here.
AI assistance: this bug was found and the fix and tests were drafted with AI tooling in my workflow; the tests and checks above were executed as pasted. I'm responsible for the change and will handle review feedback.
Note
Medium Risk
Touches core HTTP download retry/truncate logic used by Hub downloads and fsspec; behavior at offset 0 is preserved but edge cases (invalid resume_size) now raise instead of silently restarting.
Overview
Fixes
http_getso download progress and validation use bytes written by this call (new_resume_size), not the file handle’s absolute position. That unblocksHfFileSystem.get_file, which passes an already-positioned file and restores the cursor afterward.Before: a non-zero start offset caused false consistency failures (e.g. expected 100, reported 108) and, when a server ignored
Rangeand returned200,seek(0)+truncate()could wipe caller-owned prefix data.After: the final size check compares
expected_sizetonew_resume_size; the Range-ignored path rewinds withseek(tell() - resume_size)so only this download’s partial bytes are cleared. Thetemp_filedocstring documents append-from-current-position behavior. Offset-0 callers are unchanged.Adds broad
TestHttpGetcoverage (positionedBytesIO/real files, resume, double Range-ignored retries, error messages) and lets the test helper inject a pre-builttemp_file.Reviewed by Cursor Bugbot for commit d8c0262. Bugbot is set up for automated code reviews on this repo. Configure here.