Skip to content

[CLI] Snapshot by default in hf jobs stats, add -f/--follow - #5090

Merged
hanouticelina merged 9 commits into
mainfrom
jobs-stats-follow
Oct 6, 2026
Merged

hanouticelina merged 9 commits into
mainfrom
jobs-stats-follow

Conversation

@hanouticelina

@hanouticelina hanouticelina commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #5088.

hf jobs stats never returns while a job is running, so agents calling it hang until their tool timeout. This aligns it with hf jobs logs and hf spaces logs: print once and exit by default, -f/--follow for the live view.

Summary

  • Default: one snapshot with the latest sample of each job (~1s). Jobs with no sample within 10s get a warning.
  • -f/--follow: the previous live view, unchanged. Only allowed in human mode, since it relies on cursor-up redraws.
  • Snapshot output goes through out, so --format agent|json|quiet now work. Agent and JSON output the raw API metrics.
  • fetch_job_metrics gets the same follow param as fetch_job_logs (default False: current sample only), and the CLI passes it through like hf jobs logs.

Breaking change: hf jobs stats no longer streams by default (use -f), and fetch_job_metrics() now returns only the current sample unless follow=True. Same change as fetch_job_logs in #3783.

Decisions

  • Explicit -f rather than switching on TTY or agent detection: whether a command blocks shouldn't depend on where it runs. kubectl top, nvidia-smi, gh pr checks and docker logs also default to a snapshot.
  • No --tail: the metrics endpoint keeps no history, so there is nothing to tail.

Examples

$ hf jobs stats 6ac3bd9afbc85ba6823a9ad4
JOB ID                CPU % NUM CPU MEM % MEM USAGE      NET I/O         GPU UTIL % GPU MEM % GPU MEM USAGE
--------------------- ----- ------- ----- -------------- --------------- ---------- --------- -------------
6ac3bd9afbc85ba682... 0%    2.0     0.01% 1.0MB / 16.0GB 0.0bps / 0.0bps N/A        N/A       N/A
Hint: Use `hf jobs stats -f celinah/6ac3bd9afbc85ba6823a9ad4` to follow live stats.
$ hf jobs stats --json 6ac3bd9afbc85ba6823a9ad4
[{"job_id": "6ac3bd9afbc85ba6823a9ad4", "cpu_usage_pct": 0, "cpu_millicores": 2000, "memory_used_bytes": 1015808, "memory_total_bytes": 16000000000, "rx_bps": 0, "tx_bps": 0, "gpus": {}, "replica": "8x6jx"}]
$ hf jobs stats -f 6ac3bd9afbc85ba6823a9ad4   # from an agent
Error: `--follow` displays a live terminal view and is not supported with 'agent' output. Run without `--follow` to get a snapshot.

Note

Medium Risk
Breaking CLI and Python API default for job metrics streaming; behavior change affects automation that relied on blocking stats until job completion.

Overview
hf jobs stats now exits after one metrics snapshot by default, fixing hangs for scripts and agents that called it while jobs were still running. Use -f / --follow for the previous live, blocking table refresh until jobs finish; --follow is human-output only (errors on agent/json).

Snapshot mode fetches the latest sample per job in parallel, supports --format json|agent via out.table, and hints how to follow live. fetch_job_metrics(..., follow=False) now yields a single current sample (breaking API change, aligned with fetch_job_logs). Docs for the CLI guide and package reference are updated.

Reviewed by Cursor Bugbot for commit 9d30664. Bugbot is set up for automated code reviews on this repo. Configure here.

@bot-ci-comment

bot-ci-comment Bot commented Oct 5, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/huggingface_hub/cli/jobs.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/huggingface_hub/cli/jobs.py Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 57dad5f. Configure here.

Comment thread src/huggingface_hub/cli/jobs.py Resolved

@hanouticelina hanouticelina left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ all good!

@hanouticelina
hanouticelina marked this pull request as ready for review October 6, 2026 12:56
@hanouticelina
hanouticelina merged commit a5277cf into main Oct 6, 2026
30 of 31 checks passed
@hanouticelina
hanouticelina deleted the jobs-stats-follow branch October 6, 2026 13:33
@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v2.2.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add --follow to hf jobs stats

1 participant