Skip to content

[HfFileSystem] Fix glob crash when the parent listing is outdated - #5237

Merged
hanouticelina merged 1 commit into
mainfrom
fix-bucket-glob-stale-parent-cache
Oct 22, 2026
Merged

hanouticelina merged 1 commit into
mainfrom
fix-bucket-glob-stale-parent-cache

Conversation

@hanouticelina

@hanouticelina hanouticelina commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Since #5097 (v2.2.0), Dataset.push_to_hub("buckets/...") into a new bucket fails with FileNotFoundError: buckets/<id>/data. datasets' bucket integration tests have been failing on this since 2.2.0, unnoticed because those jobs use continue-on-error.

Summary

  • glob lists data/ from the Hub, then info() looks data up in the cached listing of the bucket root, taken before the files were uploaded through HfApi. [HfFileSystem] fix ls() on new empty bucket #5097 started caching empty bucket roots, so the first push to a new bucket always fails. A bucket with files but no data/ folder already failed this way before [HfFileSystem] fix ls() on new empty bucket #5097.
  • After listing a path from the Hub, drop the cached parent listing if it doesn't contain that path.
fs = HfFileSystem()
fs.glob(f"buckets/{bucket_id}/data/train-*")  # [], caches the empty bucket root
api.batch_bucket_files(bucket_id, add=[(b"...", "data/train-00000-of-00001.parquet")])
fs.glob(f"buckets/{bucket_id}/data/*-of-*.*")
# before: FileNotFoundError: buckets/__DUMMY_HUB_USER__/glob-d0c7b3/data
# after:  ['buckets/__DUMMY_HUB_USER__/glob-56393c/data/train-00000-of-00001.parquet']

datasets' tests/test_upstream_hub.py -m integration -k bucket on hub-ci: 5 failed with 2.3.0rc0, 7 passed with this branch.

Decision: fixed in the HfFileSystem cache rather than only in datasets (which could call invalidate_cache() after uploading), since an outdated cache shouldn't make glob fail on a folder it just listed.


Note

Medium Risk
Touches core listing/cache logic used by ls, glob, and info; behavior change is narrow (drop parent cache only when inconsistent) but affects all repo and bucket tree listings.

Overview
Fixes glob / info() failures when HfFileSystem.dircache still holds a stale parent directory listing (e.g. an empty bucket root from before uploads).

After a successful Hub listing in _ls_tree, the code now calls _drop_outdated_parent_listing: if the parent’s cached children don’t include the path that was just listed, that parent entry is removed so the next info() re-lists instead of raising FileNotFoundError. Empty-directory caching (dircache[path] = []) is unchanged when the listing is truly empty.

Reviewed by Cursor Bugbot for commit d7a905c. Bugbot is set up for automated code reviews on this repo. Configure here.

@bot-ci-comment

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@hanouticelina

Copy link
Copy Markdown
Collaborator Author

cc @lhoestq for vis

@hanouticelina
hanouticelina marked this pull request as ready for review October 22, 2026 13:09

@hanouticelina hanouticelina left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me ✅

@hanouticelina
hanouticelina merged commit 03b2828 into main Oct 22, 2026
31 checks passed
@lhoestq

lhoestq commented Oct 22, 2026

Copy link
Copy Markdown
Member

thx !

@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v2.3.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants