Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
160 commits
Select commit Hold shift + click to select a range
5ec8e27
Post-release: bump version to 1.30.0.dev0 (#4763)
huggingface-hub-bot[bot] Aug 27, 2026
ae20a50
[CI] Enable enforcement in close-unscoped-community-prs workflow (#4766)
hanouticelina Aug 27, 2026
97655e8
Align `hf jobs scheduled ls` filtering with `hf jobs ls` (`--status`/…
moon-bot-app[bot] Aug 28, 2026
1de4817
[Download] Follow redirects between Hub hosts when resolving files (#…
hanouticelina Aug 28, 2026
4cc16bc
[Download] A `ResolvedRevision` only pins the repo it was resolved fo…
Wauplin Aug 28, 2026
94cb25d
[CI] Fix torch_1.11 case-arm label to match renamed matrix entry (#4776)
MohammedAlkindi Aug 31, 2026
6a2eb09
[Sandbox] Document experimental status (#4783)
Wauplin Sep 1, 2026
f0a5c6d
[Inference Providers] Do not require the conversational tag for chat-…
hanouticelina Sep 1, 2026
176d4f3
[CI] Disable auto close (#4800)
Wauplin Sep 2, 2026
b84090b
[Release] Consolidate the HF CLI skill sync into the release workflow…
hanouticelina Sep 3, 2026
c804cdd
Post-release: bump version to 1.31.0.dev0 (#4806)
huggingface-hub-bot[bot] Sep 3, 2026
15bb844
Bump the actions group across 1 directory with 5 updates (#4798)
dependabot[bot] Sep 4, 2026
deb9777
[Docs] Clarify Jobs argument forwarding and align examples (#4809)
davanstrien Sep 4, 2026
634ad9b
[httpx migration] Expose `httpx` as a `huggingface_hub` submodule (#4…
Wauplin Sep 7, 2026
f1bb85f
[hf_api] Make CommitInfo copy-able and pickle-able (#4822)
Wauplin Sep 7, 2026
21e687e
[Download] Prevent cached file copies during local-dir dry runs (#4817)
wakamex Sep 7, 2026
6c8c56b
Fix: keep single-quoted values verbatim in load_dotenv (#4827)
AnishPatel526 Sep 8, 2026
a9b9ac6
[CLI] Fix 'hf buckets rm --recursive' deleting lexical siblings of th…
Wauplin Sep 8, 2026
01fe4bc
[Sandbox] Support custom Job labels (#4824)
Wauplin Sep 8, 2026
204b717
[CLI] Don't let `hf update` self-upgrade a pip install on Windows (#4…
Wauplin Sep 8, 2026
aa3c2fd
[HfFileSystem] Reject unsafe paths before local downloads (#4828)
Wauplin Sep 8, 2026
9d7bbfe
[Download] Share retry handling for stream entry and body failures (#…
Wauplin Sep 8, 2026
8934ec0
[Download] Tolerate missing HEAD Content-Length (#4805)
Wauplin Sep 8, 2026
b9c14d9
[Download] Write cache ref file atomically in `snapshot_download` (#4…
Wauplin Sep 8, 2026
063b37b
[CLI] Raise explicit error for shell-script extensions on Windows (#4…
hanouticelina Sep 9, 2026
6cae778
[Jobs] Add network groups to hf jobs run (#4833)
co42 Sep 9, 2026
22fe960
Fix dotenv parser truncating unquoted values containing '#' (#4842)
BenYang12 Sep 9, 2026
c4f076c
(LFS)Fix SliceFileObj.__iter__ yielding only the first 4MB chunk (#4844)
BenYang12 Sep 9, 2026
f148666
[CLI] Fix truncated command descriptions in the CLI reference (#4849)
hanouticelina Sep 9, 2026
ff4292b
Post-release: bump version to 1.32.0.dev0 (#4856)
huggingface-hub-bot[bot] Sep 10, 2026
a2af2e0
docs(jobs): scope network groups to the namespace and resource group …
co42 Sep 10, 2026
0790f7b
[Community] Fix docstring for DiscussionComment.last_edited_by (#4864)
Saniyagupte Sep 10, 2026
bf85073
fix(sandbox): preserve terminal command results (#4851)
evalstate Sep 10, 2026
819cd77
[CLI] Read a UV script's `[tool.hf-jobs]` launch config (opus-generat…
Wauplin Sep 10, 2026
b853ec6
Add `--container-registry-{username,password}` (#4854)
alvarobartt Sep 10, 2026
e203a7f
fix(serialization): save_torch_state_dict cleanup regex silently matc…
g0rdonL Sep 10, 2026
3493edb
Bump the actions group with 4 updates (#4848)
dependabot[bot] Sep 10, 2026
4110cac
Key the doc build concurrency group on the pull request number (#4876)
albertvillanova Sep 11, 2026
edb2c83
[Buckets] Fix file metadata size on redirects (#4874)
davanstrien Sep 11, 2026
5694eda
[RepoCard] Fix AttributeError when model-index has source: null (#4870)
Saniyagupte Sep 11, 2026
b2678ec
[Spaces] Fix AttributeError in SpaceRuntime when hardware is null (#4…
Saniyagupte Sep 11, 2026
8a408cd
[CI] Make CI less flaky and stop 6h hung jobs (#4830)
Wauplin Sep 11, 2026
dd6919c
[Serialization] Harden torch checkpoint deserialization (#4877)
Wauplin Sep 11, 2026
9588035
feat(jobs): expose the network group on JobInfo (#4862)
co42 Sep 11, 2026
129bbb5
[Http] Fix race in get_session() creating multiple clients (#4868)
hanouticelina Sep 11, 2026
98e2db8
[CLI] Make --format human override agent auto-detection (#4878)
hanouticelina Sep 14, 2026
c5ff040
[Serialization] Sync `_get_dtype_size` with safetensors (#4902)
Wauplin Sep 14, 2026
f1e70a2
[Safetensors] Recognize proposed U3 metadata (#4898)
7dsolv Sep 14, 2026
fb21643
[Download] Fix silently disabled tree cache on long Windows paths (#4…
askalf Sep 14, 2026
490f06e
docs: make the sandbox security contract match the implementation (#4…
Wauplin Sep 14, 2026
1fb5b3b
feat(sandbox): use each pooled sandbox's own capability token (#4832)
Wauplin Sep 14, 2026
ef2662f
fix(sandbox): validate a pool host before sending it a credential (#4…
Wauplin Sep 14, 2026
40efc5e
fix(sandbox): bind the pool cache to the context that wrote it (#4838)
Wauplin Sep 14, 2026
0613ead
fix(sandbox): bound what a transfer or a command's output costs the c…
Wauplin Sep 14, 2026
c7e60d8
fix(sandbox): decide host teardown per host, and report it honestly (…
Wauplin Sep 14, 2026
ec61993
fix(sandbox): address background processes by their server-assigned i…
Wauplin Sep 14, 2026
0219ad5
[sandbox audit] Pin the sandbox server binary by digest and verify it…
Wauplin Sep 14, 2026
1173050
[sandbox audit] Keep secret values out of argv in the CLI and fix the…
Wauplin Sep 14, 2026
48cdc03
Fix doubled colon in snapshot download transfer bar description (#4890)
sandeepb2013 Sep 14, 2026
a3b62f6
[CLI] Add `hf --skills` as an alias for `hf skills preview` (#4906)
Wauplin Sep 14, 2026
bd9ece0
[CLI] Support file URIs in hf cache rm (#4847)
hanouticelina Sep 14, 2026
0a57df9
[Core] Restrict repo_id validation regex to ASCII word characters (#4…
bodapatisaikrishna Sep 15, 2026
af6b41e
[Core] Reject path traversal via embedded ".." segments in path_in_re…
bodapatisaikrishna Sep 15, 2026
2075fd6
[Cache] Support kernel repos in `hf cache` commands (#4905)
hanouticelina Sep 15, 2026
5a9cdda
[Core] Speed up package imports (#4914)
Wauplin Sep 15, 2026
6e3d21f
[Cache] Add cross-repo shared blob store (#4498)
hanouticelina Sep 17, 2026
f94ea55
[Buckets] Send mtime when copying files (#4920)
Wauplin Sep 17, 2026
d7b62c7
[CLI] Fix card data serialization (#4882)
davanstrien Sep 17, 2026
79dd41e
[Docs] Fix hf discussions info options that do not exist (#4886)
rishabhsai Sep 17, 2026
b064a14
[Download] Send X-HF-Download-Counter header on download calls (#4613)
Wauplin Sep 17, 2026
e9c55ce
Post-release: bump version to 1.33.0.dev0 (#4926)
huggingface-hub-bot[bot] Sep 17, 2026
618250b
[Download] Publish symlink support only after probing (#4923)
Wauplin Sep 17, 2026
77b6b2b
[Core] Skip invalid eval results entries instead of raising (#4924)
davanstrien Sep 17, 2026
34ae660
[Cache] Use posix paths for scan_cache_dir ref names (#4925)
gokay-ai Sep 18, 2026
0ba990a
[Download] Measure http_get progress from the initial file position (…
Wauplin Sep 18, 2026
cea607a
[Serialization] Fix device type when map_location is None (#4938)
Wauplin Sep 18, 2026
83ba61a
[CLI] Honor --format / --json / -q on jobs run commands (#4936)
Wauplin Sep 18, 2026
8c9546f
[Quality] Replace `re.match` with `re.fullmatch` / `re.search` and ba…
Wauplin Sep 18, 2026
22b5e87
[Docs] Sync provider task table with the PROVIDERS registry (#4930)
rishabhsai Sep 18, 2026
8efc0a9
[HfFileSystem] Make lpath optional in get_file (fsspec>=2026.9.0) (#4…
assafvayner Sep 21, 2026
4c80e7e
Fix `format_timesince` showing "60 seconds ago" / "60 minutes ago" / …
Sreekant13 Sep 21, 2026
55668e2
Fix Windows conftest erroring test subsets on lazy-loaded file_downlo…
dltsum Sep 21, 2026
7ce68b9
[Webhooks] Allow `bucket` as a watched item type (#4958)
davanstrien Sep 21, 2026
7025391
[MCP] Follow nextCursor when listing tools in MCPClient (#4960)
Kayvan-Zahiri Sep 22, 2026
80aef09
Align the `hf` package's python_requires with huggingface_hub (#4959)
sharziki Sep 22, 2026
94cf392
Remove dots from allowed characters in job labels. (#4962)
evalstate Sep 22, 2026
8d4fba6
[CLI] Honor `CLAUDE_CONFIG_DIR` in `hf skills add -g --claude` (#4957)
hanouticelina Sep 22, 2026
aa8742c
[CLI] Always install skills for Claude Code, deprecate --claude (#4967)
hanouticelina Sep 22, 2026
ce69fdb
[Buckets] Raise when a batch reports failed operations (#4954)
hanouticelina Sep 22, 2026
0f88f32
[Download] Extend tmp .incomplete path on Windows when too long (#4978)
Wauplin Sep 23, 2026
7532272
Fix HfFileSystem.get_file with a bare local filename (#4981)
davanstrien Sep 23, 2026
da5e244
Scope GITHUB_TOKEN permissions per job (#4982)
hf-security-analysis[bot] Sep 23, 2026
1e0c305
Send PrivateLink config under privateService in create_inference_endp…
co42 Sep 23, 2026
b904b94
Apply the repo_id length limit to the namespace as well (#4988)
anishmehta24 Sep 24, 2026
ac3ed2f
[Xet] Show Validating progress during shard finalization (#4478)
seanses Sep 24, 2026
fa26f34
[CI] Support major (pre-)releases in release workflow (#4991)
Wauplin Sep 24, 2026
c2e6e0a
Post-release: bump version to 1.34.0.dev0 (#4993)
huggingface-hub-bot[bot] Sep 24, 2026
9f51867
[Release] Prepare huggingface_hub 2.0 (#4994)
Wauplin Sep 24, 2026
89ae51f
unbump
Wauplin Sep 24, 2026
f5b624c
Post-release: bump version to 2.1.0.dev0 (#4999)
huggingface-hub-bot[bot] Sep 24, 2026
32456ab
[Buckets] Make include/exclude patterns case-sensitive on all platfor…
Wauplin Sep 25, 2026
c749dfa
Support secrets on Job-triggered webhooks (#5009)
moon-bot-app[bot] Sep 25, 2026
ee4f802
[HTTP] Fix repo_id in errors for query strings and download URLs (#5008)
Wauplin Sep 25, 2026
79874bc
[Buckets] Fix copy_files treating lexical siblings as an existing des…
AnishPatel526 Sep 25, 2026
ff0ccab
[CI] Bump doc-builder workflows pin to 63ec6f1 (#5010)
Wauplin Sep 25, 2026
34e8baa
[docs] xet caching and concurrency updates (#4883)
cbensimon Sep 25, 2026
62a1f2f
[HfFileSystem] Download Xet files with hf_xet in get_file (#4990)
hanouticelina Sep 25, 2026
e155ad3
[docs] Fix hf.cos typo in repository guide link (#5016)
pratikgx Sep 28, 2026
a076863
[datasets sql] Run the DuckDB CLI subprocess with UTF-8 encoding (#5020)
825pranav Sep 28, 2026
b15915b
[Auth] Store token names as UTF-8 instead of locale encoding (#5021)
Evolian-o Sep 28, 2026
41222c5
Fix scalar tags splitting into single-character list in card front ma…
zeenat28-ui Sep 28, 2026
14ee250
[CLI] Hint to use --global after local 'hf skills add/update' (#5024)
Wauplin Sep 28, 2026
f151320
[Sandbox] Require scoped credentials for pooled sandboxes (#4782)
Wauplin Sep 28, 2026
f64c26e
[Dataclasses] Accept bare typing.List/Dict/Set in @strict validation …
anishmehta24 Sep 29, 2026
7caeebb
Suggest `hf auth login` when `hf auth whoami` fails (#5033)
moon-bot-app[bot] Sep 29, 2026
5583746
[CLI] Align 'hf auth whoami' not-logged-in message with 'hf auth toke…
Wauplin Sep 29, 2026
862f23d
[Sandbox] Pin sbx-server 0.7.0 (protocol 3) (#5028)
Wauplin Sep 29, 2026
1b8b2fa
[CLI] Escape control characters in listings and table output (#5029)
Wauplin Sep 29, 2026
548f7e3
[Jobs] Allow rescheduling scheduled jobs (#5026)
Wauplin Sep 29, 2026
3fd18f2
[Jobs] Manage public and private exposed ports (#5027)
Wauplin Sep 29, 2026
231a2eb
[Jobs] Add rerun and retry attempts (#5025)
Wauplin Sep 29, 2026
d889712
[Tests] Reduce redundant Hub calls in repo settings and move tests (#…
Wauplin Sep 29, 2026
cc2f368
[Utils] Retry the first pagination page (#5035)
Wauplin Sep 29, 2026
1092497
[Spaces] Update docs for cpu-basic gating and dev mode disabling (#5037)
Wauplin Sep 29, 2026
013afa7
[CLI] Raise if a bare --secrets NAME is not set locally in hf jobs (#…
Wauplin Sep 30, 2026
83a714e
[CLI] Update hf upload help text after upload_large_folder removal (#…
davanstrien Sep 30, 2026
881a6c4
[Docs] Fix links in i18n issue template (#5051)
pratikgx Oct 1, 2026
60fdb08
[Docs] Fix duplicated words in docstring and comments (#5052)
Ad1th Oct 1, 2026
b5e3a12
[HfApi] Support resource_group_id in duplicate_repo (#5038)
Wauplin Oct 1, 2026
3bb08e2
[Endpoints] Migrate catalog to the new /api/v1 API (#4928)
Wauplin Oct 1, 2026
4c88ea6
[Repos] Warn when duplicated repo files are still being copied (#5041)
Wauplin Oct 1, 2026
604252e
[HfApi] Add visibility to create_bucket (#5053)
hanouticelina Oct 1, 2026
0b26b50
[Spaces] Add get_zero_gpu_quota and hf spaces zero-gpu-quota (#5040)
Wauplin Oct 1, 2026
ff61343
Post-release: bump version to 2.2.0.dev0 (#5054)
huggingface-hub-bot[bot] Oct 1, 2026
b3fa3a4
[CLI] Rename 'hf spaces zero-gpu-quota' to 'hf spaces zero-gpu quota'…
Wauplin Oct 1, 2026
7da19ff
[CI] Split tests into api / transfer / inference suites (#5042)
Wauplin Oct 1, 2026
7a54109
[Tests] Always clean up staging repos and buckets created by tests (#…
Wauplin Oct 1, 2026
1a76ad2
[CI] Reduce flaky tests: isolate jedi cache, sleep before read-after-…
Wauplin Oct 1, 2026
c31df86
[Bugfix] Fix wrong exception type and unbound variable guard (#5063)
vector15-05 Oct 2, 2026
37ce057
[CLI] Upload existing paths with glob characters as-is in hf upload (…
Ganesh2325 Oct 2, 2026
0ce4427
[Download] Warn when allow/ignore patterns match no files in snapshot…
Ad1th Oct 2, 2026
a625317
[Repocard] Respect configured endpoint when validating cards (#5069)
Wauplin Oct 2, 2026
d711944
[CLI] Read env/secrets files as UTF-8 and strip the BOM (#5066)
hanouticelina Oct 2, 2026
42f9085
[Setup] Install hf-xet on Windows ARM64 (#5074)
angt Oct 5, 2026
d699913
[Docs] Fix HTTP migration and async session references (#5079)
Yuilona Oct 5, 2026
bc9b691
Bump doc-builder workflow pin (#5087)
paulinebm Oct 5, 2026
3b05c63
[Jobs] Ignore 406 from metrics endpoint for finished jobs (#5091)
hanouticelina Oct 6, 2026
e817417
Bump doc-builder pin to 9fc8a41 in build_documentation.yaml (#5093)
paulinebm Oct 6, 2026
9071b17
[Jobs] Fix no-follow logs hanging while the job keeps printing (#5092)
hanouticelina Oct 6, 2026
68becf4
[CommitScheduler] Fix race that skips the first scheduled push (#5086)
hanouticelina Oct 6, 2026
ff4c989
[CLI] Print [] from hf jobs stats --json when no jobs are running (#5…
hanouticelina Oct 6, 2026
83ce8b0
[Download] Consider port when following /resolve redirects (#4942)
assafvayner Oct 6, 2026
a5277cf
[CLI] Snapshot by default in hf jobs stats, add -f/--follow (#5090)
hanouticelina Oct 6, 2026
14a48e7
[HfFileSystem] fix `ls()` on new empty bucket (#5097)
lhoestq Oct 7, 2026
38e0b3c
[HfFileSystem] Replace cached listings instead of appending to them (…
ChihebBENCHEIKH1 Oct 7, 2026
af74167
[Tests] Wait for bucket listings after writes (#5095)
hanouticelina Oct 7, 2026
cf992a9
[HfFileSystem] Make refresh=True clear cached not-found results (#5106)
hanouticelina Oct 7, 2026
bb3a875
[Docs] Fix from_pretrained integration example (#5105)
Ayushdevo Oct 7, 2026
8abbfe5
Fix race in `_get_version` that makes concurrent downloads skip hf_xe…
BramVanroy Oct 7, 2026
3cfa593
Guard upload_pr_documentation workflow to upstream repo only
g0rdonL Oct 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[Xet] Show Validating progress during shard finalization (huggingface…
…#4478)

* shard upload progress bar

* update _upload_pipeline

* update to simplify UI

* fixes

* fixes

* update UI

* bump hf-xet

* [Xet] Fix validating progress timing and bar positions

* [Xet] Remove validating bar position test

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Celina Hanouti <hanouticelina@gmail.com>
Co-authored-by: Lucain Pouget <lucainp@gmail.com>
  • Loading branch information
4 people authored Sep 24, 2026
commit ac3ed2f7ea0900e8cff6a18f8b5ea6290bb9ec61
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ def get_version() -> str:


# hf-xet version used in both install_requires and extras["hf_xet"]
HF_XET_VERSION = "hf-xet>=1.5.2,<2.0.0"
HF_XET_VERSION = "hf-xet>=1.6.0,<2.0.0"

install_requires = [
"click>=8.4.2,<9.0.0", # 8.4.0/8.4.1 shipped a broken fish completion script
Expand Down
89 changes: 75 additions & 14 deletions src/huggingface_hub/_upload_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,13 +102,19 @@ def _format_bytes(n: float) -> str:


class _LiveDisplay:
"""Three-line live progress display on stderr::
"""Live progress display on stderr::

Preparing ████████████████████ 11,100 / 11,100 ✓
Uploading ██████████████░░░░░░ 580 / 603 files 3.8GB · 19.7MB/s
Validating ████████░░░░░░░░░░░░ 43%
Committing ██████████████████░░ 10,800 / 11,100 14 commits

A small renderer thread redraws the three lines in-place every ~0.5 s on a TTY
The Validating line appears once the v2 shard finalization API starts reporting
progress. Shard upload / validation / sync are folded into this single bar so
users are not exposed to Xet-internal "shard" jargon (and so there is only one
Committing line — the git commit).

A small renderer thread redraws all lines in-place every ~0.5 s on a TTY
(worker threads only update counters under a lock). When stderr is not a TTY,
it falls back to a periodic ``logger.info`` summary instead.

Expand All @@ -117,14 +123,12 @@ class _LiveDisplay:
tailing stderr during a long upload still see periodic progress.
"""

_N_LINES = 3

def __init__(self, total_files: int, enabled: bool = True) -> None:
self._total = total_files
self._tty = enabled and sys.stderr.isatty()
self._active = self._tty or logger.isEnabledFor(logging.INFO)
self._lock = threading.Lock()
self._drawn = False
self._drawn_lines = 0 # number of lines printed in the last _redraw call
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None

Expand All @@ -142,6 +146,13 @@ def __init__(self, total_files: int, enabled: bool = True) -> None:
self._prev_bytes = 0
self._prev_time: float | None = None

# Shard finalization → single Validating bar (v2 API; activated on first event)
self._shard_active = False
self._shard_val_total = 0
self._shard_val_done = 0
self._shards_total = 0
self._shards_done = 0

# -- lifecycle (main thread) ------------------------------------------------

def start(self) -> None:
Expand Down Expand Up @@ -199,18 +210,44 @@ def new_xet_callback(self) -> "Callable | None":
"""
if not self._active:
return None
prev = 0
prev_transfer = 0
prev_shard_val_done = 0
prev_shard_val_total = 0
prev_shards_done = 0
prev_shards_total = 0

def callback(group_report: Any, item_reports: Any) -> None:
nonlocal prev
nonlocal prev_transfer
nonlocal prev_shard_val_done, prev_shard_val_total
nonlocal prev_shards_done, prev_shards_total
with self._lock:
completed = group_report.total_transfer_bytes_completed
self._xet_bytes += max(0, completed - prev)
prev = completed
self._xet_bytes += max(0, completed - prev_transfer)
prev_transfer = completed
for item in item_reports.values():
if item.total_bytes > 0 and item.bytes_completed == item.total_bytes:
self._xet_done.add(item.item_name)

shard = getattr(group_report, "shard", None)
if shard is not None and shard.total_shards > 0:
self._shard_active = True

v = shard.total_shard_validation_entries_completed
self._shard_val_done += max(0, v - prev_shard_val_done)
prev_shard_val_done = v

v = shard.total_shard_validation_entries
self._shard_val_total += max(0, v - prev_shard_val_total)
prev_shard_val_total = v

v = shard.total_shards_completed
self._shards_done += max(0, v - prev_shards_done)
prev_shards_done = v

v = shard.total_shards
self._shards_total += max(0, v - prev_shards_total)
prev_shards_total = v

return callback

# -- rendering (display thread) ----------------------------------------------
Expand All @@ -235,14 +272,19 @@ def _update_speed(self) -> None:
self._prev_bytes = self._xet_bytes

def _redraw(self) -> None:
if self._drawn:
sys.stderr.write(f"\033[{self._N_LINES}A")
if self._drawn_lines > 0:
sys.stderr.write(f"\033[{self._drawn_lines}A")
# Validating sits between Uploading and Committing (before the git commit bar).
lines = [self._line_preparing(), self._line_uploading()]
if self._shard_active:
lines.append(self._line_validating())
lines.append(self._line_committing())
width = shutil.get_terminal_size().columns
for line in (self._line_preparing(), self._line_uploading(), self._line_committing()):
for line in lines:
truncated = line[: width - 4] + "..." if len(line) > width - 1 else line
sys.stderr.write(f"\r\033[K{truncated}\n")
sys.stderr.flush()
self._drawn = True
self._drawn_lines = len(lines)

def _line_preparing(self) -> str:
done = " ✓" if self._prepared >= self._total else ""
Expand All @@ -262,6 +304,17 @@ def _line_uploading(self) -> str:
done = " ✓" if self._prepared >= self._total and n_done >= self._xet_total else ""
return f" Uploading {_bar(n_done, self._xet_total)} {n_done:,} / {self._xet_total:,} files{extra}{done}"

def _line_validating(self) -> str:
# Prefer validation-entry progress (the long phase). Fall back to shard-completion
# counts when entries are absent (e.g. V1 synthetic Result with no Validating frames).
# Show a percentage — raw entry counts are opaque Xet internals.
if self._shard_val_total > 0:
done_n, total_n = self._shard_val_done, self._shard_val_total
else:
done_n, total_n = self._shards_done, self._shards_total
pct = min(100, round(100 * done_n / total_n)) if total_n > 0 else 0
return f" Validating {_bar(done_n, total_n)} {pct}%"

def _line_committing(self) -> str:
effective = self._total - self._ignored
commits_str = f" {self._nb_commits} commits" if self._nb_commits > 1 else ""
Expand All @@ -271,11 +324,19 @@ def _line_committing(self) -> str:
)

def _summary(self) -> str:
return (
summary = (
f"Uploading... {self._prepared:,}/{self._total:,} files checked, "
f"{len(self._xet_done):,}/{self._xet_total:,} uploaded ({_format_bytes(self._xet_bytes)} transferred), "
f"{self._committed:,} committed in {self._nb_commits} commit(s)"
)
if self._shard_active:
if self._shard_val_total > 0:
pct = min(100, round(100 * self._shard_val_done / self._shard_val_total))
summary += f", validating {pct}%"
elif self._shards_total > 0:
pct = min(100, round(100 * self._shards_done / self._shards_total))
summary += f", validating {pct}%"
return summary


class _CommitPacer:
Expand Down
86 changes: 82 additions & 4 deletions src/huggingface_hub/utils/_xet_progress_reporting.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ def _format_speed_postfix(speed: float | None) -> str:
XET_TRANSFER_BAR_FORMAT = "{desc}: {bar}| {n_fmt:>5}B{postfix:>12}"
XET_BYTES_BAR_FORMAT = "{l_bar}{bar}| {n_fmt:>5}B / {total_fmt:>5}B{postfix:>12}"

_NUM_OVERVIEW_BARS = 2 # data_processing + upload (always shown)
_NUM_SHARD_BARS = 1 # single Validating bar (lazy; covers shard upload/validate/sync)


def _set_monotonic_total(bar, total: int | None) -> None:
if total is None or not hasattr(bar, "total"):
Expand Down Expand Up @@ -193,17 +196,33 @@ def __init__(self, n_lines: int = 10, description_width: int = 30, total_files:

self.per_file_progress = is_google_colab() or not is_notebook()

_nrows = (
_NUM_OVERVIEW_BARS + n_lines + _NUM_SHARD_BARS + 1
if self.per_file_progress
else _NUM_OVERVIEW_BARS + _NUM_SHARD_BARS + 1
)

self.tqdm_settings = {
"unit": "B",
"unit_scale": True,
"leave": True,
"unit_divisor": 1000,
"nrows": n_lines + 3 if self.per_file_progress else 3,
"nrows": _nrows,
"miniters": 1,
"bar_format": XET_BYTES_BAR_FORMAT,
}

# Overall progress bars
# Percentage only — validation-entry totals are opaque Xet internals.
self.percent_tqdm_settings = {
"unit": "",
"unit_scale": False,
"leave": True,
"nrows": _nrows,
"miniters": 1,
"bar_format": "{desc}: {bar}| {percentage:3.0f}%{postfix:>12}",
}

# Overview bars (always shown)
self.data_processing_bar = tqdm(
total=0, desc=self.format_desc("Processing Files (0 / 0)", False), position=0, **self.tqdm_settings
)
Expand All @@ -212,14 +231,19 @@ def __init__(self, n_lines: int = 10, description_width: int = 30, total_files:
total=0, desc=self.format_desc("New Data Upload", False), position=1, **self.tqdm_settings
)

# Single Validating bar created lazily on the first shard event.
# Covers shard upload / validation / sync without exposing "shard" jargon.
# Positioned dynamically after the active per-file bars (see _validating_bar_position).
self.validating_bar: tqdm | None = None

self.known_items: set[str] = set()
self.completed_items: set[str] = set()

# Track previous absolute values to compute increments
self._prev_bytes_completed: int = 0
self._prev_transfer_bytes_completed: int = 0

# Item bars (scrolling view)
# Item bars (scrolling view); positions start at _NUM_OVERVIEW_BARS
self.item_state: OrderedDict[str, Any] = OrderedDict()
self.current_bars: list = [None] * self.n_lines

Expand All @@ -241,13 +265,34 @@ def format_desc(self, name: str, indent: bool) -> str:

return f"{padding}{name.ljust(width)}"

def _validating_bar_position(self) -> int:
"""Place Validating right after overview bars and any active per-file bars."""
if not self.per_file_progress:
return _NUM_OVERVIEW_BARS
n_active = sum(1 for bar in self.current_bars if bar is not None)
return _NUM_OVERVIEW_BARS + n_active

def _init_validating_bar(self) -> None:
"""Create the Validating progress bar on first shard event."""
self.validating_bar = tqdm(
total=0,
desc=self.format_desc("Validating", False),
position=self._validating_bar_position(),
**self.percent_tqdm_settings,
)

def reset_for_next_commit(self):
"""Reset per-commit state so the reporter can be reused across multiple upload commits."""
self._prev_bytes_completed = 0
self._prev_transfer_bytes_completed = 0
self.known_items.clear()
self.completed_items.clear()
self.item_state.clear()
# Shard counters are absolute per commit — restart Validating at 0% for the next chunk.
if self.validating_bar is not None:
self.validating_bar.n = 0
self.validating_bar.total = 0
self.validating_bar.refresh()

def update_progress(self, group_report, item_reports: dict):
# Update all the per-item values.
Expand Down Expand Up @@ -287,9 +332,14 @@ def update_progress(self, group_report, item_reports: dict):
in_final_bar_mode = False

if bar is None:
# Move Validating before creating a file bar in its current row.
if self.validating_bar is not None and self.validating_bar.pos == -(_NUM_OVERVIEW_BARS + bar_idx):
self.validating_bar.clear()
self.validating_bar.pos -= 1
self.validating_bar.refresh()
self.current_bars[bar_idx] = tqdm(
desc=self.format_desc(name, True),
position=2 + bar_idx, # Set to the position past the initial bars.
position=_NUM_OVERVIEW_BARS + bar_idx,
total=item.total_bytes,
initial=item.bytes_completed,
**self.tqdm_settings,
Expand Down Expand Up @@ -343,10 +393,38 @@ def update_progress(self, group_report, item_reports: dict):
)
self.upload_bar.update(transfer_inc)

# Single Validating bar for shard finalization (v2 shard API).
# Prefer validation-entry counts (the long phase). Fall back to shard-completion
# counts when entries are absent (e.g. V1 synthetic Result with no Validating frames).
# Set absolute n/total for the current commit; reset_for_next_commit restarts at 0%.
shard = getattr(group_report, "shard", None)
if shard is not None and shard.total_shards > 0:
if self.validating_bar is None:
self._init_validating_bar()

assert self.validating_bar is not None

if shard.total_shard_validation_entries > 0:
done_n, total_n = (
shard.total_shard_validation_entries_completed,
shard.total_shard_validation_entries,
)
else:
done_n, total_n = shard.total_shards_completed, shard.total_shards

self.validating_bar.total = total_n or None
self.validating_bar.n = done_n
# Keep Validating snug under the active file bars as more slots fill in.
self.validating_bar.pos = -self._validating_bar_position()
self.validating_bar.refresh()

def close(self):
self.data_processing_bar.close()
self.upload_bar.close()

if self.validating_bar is not None:
self.validating_bar.close()

if self.per_file_progress:
for bar in self.current_bars:
if bar:
Expand Down
19 changes: 19 additions & 0 deletions tests/test_upload_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,25 @@ def test_scale_down_on_failure_and_bounds(self):


class TestLiveDisplayCounters:
def test_validating_appears_when_shard_finalization_starts(self):
with patch.object(upload_pipeline.logger, "isEnabledFor", return_value=True):
display = _LiveDisplay(total_files=1, enabled=False)
callback = display.new_xet_callback()

shard = SimpleNamespace(
total_shards=0,
total_shards_completed=0,
total_shard_validation_entries=0,
total_shard_validation_entries_completed=0,
)
callback(SimpleNamespace(total_transfer_bytes_completed=0, shard=shard), {})
assert not display._shard_active

shard.total_shards = 1
callback(SimpleNamespace(total_transfer_bytes_completed=0, shard=shard), {})
assert display._shard_active
assert "Validating" in display._line_validating()

def test_xet_callback_sums_increments_across_concurrent_commits(self):
with patch.object(upload_pipeline.logger, "isEnabledFor", return_value=True):
display = _LiveDisplay(total_files=10, enabled=True)
Expand Down