Skip to content

[HfFileSystem] Replace cached listings instead of appending to them - #5099

Merged
hanouticelina merged 3 commits into
huggingface:mainfrom
ChihebBENCHEIKH1:fix-hffs-dircache-listings
Oct 7, 2026
Merged

hanouticelina merged 3 commits into
huggingface:mainfrom
ChihebBENCHEIKH1:fix-hffs-dircache-listings

Conversation

@ChihebBENCHEIKH1

@ChihebBENCHEIKH1 ChihebBENCHEIKH1 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #5098

Summary

  • _ls_tree appended each fetched entry to its parent's dircache listing. Cached listings got duplicates, refresh=True kept deleted files, and a listing that failed halfway stayed cached. On buckets, ls() on a file cached that file alone as its folder's listing.
  • The listings are now built while iterating, then stored once the listing is complete. They replace what was cached for the same folders.
  • The entry whose path is the listed path itself is no longer cached. Only buckets return it; the repo tree endpoint returns 404 on a file path.

The script from #5098, on main:

ls(onnx): 10 entries, then 20 after find() (10 unique)
ls(repo): 17 entries, then 34 after ls(refresh=True) (17 unique)
ls(bucket folder) after ls() on one file: 1 vs 28 on a fresh instance
exists(sibling file): False

On this branch:

ls(onnx): 10 entries, then 10 after find() (10 unique)
ls(repo): 17 entries, then 17 after ls(refresh=True) (17 unique)
ls(bucket folder) after ls() on one file: 28 vs 28 on a fresh instance
exists(sibling file): True

Tests

  • 3 new offline tests in tests/test_hf_file_system.py: a folder after find() and refresh=True, a listing that fails halfway, and a bucket file then its folder. All 3 fail on main.
  • Offline tests in tests/test_hf_file_system.py: 72 passed. I did not run the staging-backed classes locally.
  • Read-only check on google-bert/bert-base-uncased, which has nested folders: ls, find (with withdirs and maxdepth=2), glob("**"), walk, du and the incomplete-cache path of find return the same results as on main. Only the duplicated ls() with expand_info=True changes.
  • make style, make quality, mypy src and the latest ty: pass.

This touches the same loop as #5097. Happy to rebase if it lands first.


Note

Medium Risk
Changes caching semantics for ls/find/exists across repos and buckets; behavior is more correct but any code depending on duplicate or stale cache entries could see different results.

Overview
Fixes #5098 by changing how HfFileSystem._ls_tree populates dircache. Directory listings are accumulated in a temporary listings map during the tree walk, then written with dircache.update() only after the listing finishes, replacing prior entries for those parent paths instead of appending.

When the listed path is a single bucket file (cache_path == path), that entry is no longer stored as its parent folder’s listing, so later ls() on the folder is not poisoned by a one-file cache.

Adds a regression test that lists a file, runs find() on the repo root, then ls() on the parent directory with refresh=True and expects the full folder contents.

Reviewed by Cursor Bugbot for commit 1179207. Bugbot is set up for automated code reviews on this repo. Configure here.

_ls_tree appended each fetched entry to its parent's dircache listing.
Cached listings got duplicates, and refresh=True kept deleted files.
A listing that failed halfway stayed cached as if it were complete.
On buckets, ls() on a file cached that file alone as its folder's listing.

Build the listings while iterating, then store them once the listing is
complete. Skip the entry for the listed path itself, which only buckets
return.
@bot-ci-comment

bot-ci-comment Bot commented Oct 7, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@hanouticelina hanouticelina left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for reporting and fixing!

@codecov

codecov Bot commented Oct 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.54%. Comparing base (1daa48b) to head (1179207).
⚠️ Report is 772 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5099      +/-   ##
==========================================
+ Coverage   75.00%   77.54%   +2.54%     
==========================================
  Files         145      177      +32     
  Lines       13978    24830   +10852     
==========================================
+ Hits        10484    19255    +8771     
- Misses       3494     5575    +2081     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hanouticelina
hanouticelina merged commit 38e0b3c into huggingface:main Oct 7, 2026
25 of 27 checks passed
@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v2.2.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HfFileSystem caches duplicated, stale or partial directory listings

2 participants