Skip to content

[Jobs] Fix no-follow logs hanging while the job keeps printing - #5092

Merged
hanouticelina merged 7 commits into
mainfrom
jobs-logs-no-follow-return
Oct 6, 2026
Merged

hanouticelina merged 7 commits into
mainfrom
jobs-logs-no-follow-return

Conversation

@hanouticelina

@hanouticelina hanouticelina commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

hf jobs logs <id> (no -f) never returns on a running job that keeps printing. Same agent-hang problem as #5088.

Summary

  • The server streams logs in follow mode and never marks the end of the replay. So while lines keep coming, the 5s read timeout never fires.
  • Without -f, the read now stops at the first log line written after the response's Date header + 1s (Date only has 1s precision).
  • Also fixes hf spaces logs (same helper). -f is unchanged.

Example

Running job printing 20 lines/s:

origin/main: still blocked after 25s (1878 lines)
this PR:     returns in 1.1s (614 lines)

Finished and silent jobs behave as before.

Decisions

  • Uses the server's Date header, not the local clock: a local clock running behind the server would drop the last lines.
  • No time limit on the read: it would cut off the backlog of long jobs.
  • Up to ~1s of new lines can be printed after the request.

Note

Low Risk
Scoped to non-follow log streaming in the shared SSE client; follow mode is untouched, with a small tradeoff of possibly including up to ~1s of live lines in the snapshot.

Overview
Fixes hf jobs logs (and hf spaces logs, same SSE helper) hanging when called without -f on a job or space that keeps emitting logs. The server replays history in a follow-style stream with no clear end, so the client could block indefinitely.

In _stream_sse_events, non-follow mode now treats the HTTP response Date header (+1s) as the end of the historical replay and stops once the first event has a timestamp after that cutoff (server time, not local clock). follow=True behavior is unchanged.

The hf jobs logs CLI always builds job_ref for hints: after a follow stream it still suggests hf jobs inspect; without follow it now nudges users toward hf jobs logs -f if the job may still be running.

Reviewed by Cursor Bugbot for commit 2b1e1c2. Bugbot is set up for automated code reviews on this repo. Configure here.

@bot-ci-comment

bot-ci-comment Bot commented Oct 5, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@hanouticelina
hanouticelina marked this pull request as draft October 5, 2026 16:40
@lhoestq

lhoestq commented Oct 5, 2026

Copy link
Copy Markdown
Member

good catch ! maybe minor but wdyt of adding a hint at the end to suggest --follow ?

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1835c5d. Configure here.

Comment thread src/huggingface_hub/hf_api.py Resolved
Comment thread src/huggingface_hub/cli/jobs.py Resolved
@hanouticelina
hanouticelina marked this pull request as ready for review October 6, 2026 12:45

@hanouticelina hanouticelina left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅

@hanouticelina
hanouticelina merged commit 9071b17 into main Oct 6, 2026
30 of 31 checks passed
@hanouticelina
hanouticelina deleted the jobs-logs-no-follow-return branch October 6, 2026 12:46
@hanouticelina

Copy link
Copy Markdown
Collaborator Author

@lhoestq good point! i've added the hint

@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v2.2.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants