Repository navigation
[HfFileSystem] Replace cached listings instead of appending to them - #5099
Merged
hanouticelina merged 3 commits intoOct 7, 2026
Merged
Conversation
_ls_tree appended each fetched entry to its parent's dircache listing. Cached listings got duplicates, and refresh=True kept deleted files. A listing that failed halfway stayed cached as if it were complete. On buckets, ls() on a file cached that file alone as its folder's listing. Build the listings while iterating, then store them once the listing is complete. Skip the entry for the listed path itself, which only buckets return.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
hanouticelina
approved these changes
Oct 7, 2026
hanouticelina
left a comment
Collaborator
There was a problem hiding this comment.
thanks for reporting and fixing!
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5099 +/- ##
==========================================
+ Coverage 75.00% 77.54% +2.54%
==========================================
Files 145 177 +32
Lines 13978 24830 +10852
==========================================
+ Hits 10484 19255 +8771
- Misses 3494 5575 +2081 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
This PR has been shipped as part of the v2.2.0 release. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5098
Summary
_ls_treeappended each fetched entry to its parent'sdircachelisting. Cached listings got duplicates,refresh=Truekept deleted files, and a listing that failed halfway stayed cached. On buckets,ls()on a file cached that file alone as its folder's listing.The script from #5098, on main:
On this branch:
Tests
tests/test_hf_file_system.py: a folder afterfind()andrefresh=True, a listing that fails halfway, and a bucket file then its folder. All 3 fail on main.tests/test_hf_file_system.py: 72 passed. I did not run the staging-backed classes locally.google-bert/bert-base-uncased, which has nested folders:ls,find(withwithdirsandmaxdepth=2),glob("**"),walk,duand the incomplete-cache path offindreturn the same results as on main. Only the duplicatedls()withexpand_info=Truechanges.make style,make quality,mypy srcand the latestty: pass.This touches the same loop as #5097. Happy to rebase if it lands first.
Note
Medium Risk
Changes caching semantics for
ls/find/existsacross repos and buckets; behavior is more correct but any code depending on duplicate or stale cache entries could see different results.Overview
Fixes #5098 by changing how
HfFileSystem._ls_treepopulatesdircache. Directory listings are accumulated in a temporarylistingsmap during the tree walk, then written withdircache.update()only after the listing finishes, replacing prior entries for those parent paths instead of appending.When the listed path is a single bucket file (
cache_path == path), that entry is no longer stored as its parent folder’s listing, so laterls()on the folder is not poisoned by a one-file cache.Adds a regression test that lists a file, runs
find()on the repo root, thenls()on the parent directory withrefresh=Trueand expects the full folder contents.Reviewed by Cursor Bugbot for commit 1179207. Bugbot is set up for automated code reviews on this repo. Configure here.