Skip to content

[HfFileSystem] Fix bucket prefix collisions - #4630

Merged
hanouticelina merged 5 commits into
mainfrom
agent/fix-hf-filesystem-bucket-prefix-collision
Aug 7, 2026
Merged

hanouticelina merged 5 commits into
mainfrom
agent/fix-hf-filesystem-bucket-prefix-collision

Conversation

@lewtun

@lewtun lewtun commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

Issue

The Buckets API applies prefix lexically, but HfFileSystem needs path-component semantics. If a bucket contains logs_existing/result.txt but no logs/, listing logs returns the sibling. HfFileSystem then caches it under its real parent, and exists(".../logs/new-result.txt") raises KeyError instead of returning False; ls(".../logs") can also return the unrelated sibling.

Reproducer

from uuid import uuid4

import fsspec
from huggingface_hub import batch_bucket_files, create_bucket, delete_bucket, whoami


namespace = whoami()["name"]
bucket_id = f"{namespace}/hf-fs-prefix-repro-{uuid4().hex[:10]}"
create_bucket(bucket_id, private=True)

try:
    batch_bucket_files(
        bucket_id,
        add=[(b"test", "logs_existing/result.txt")],
    )

    url = f"hf://buckets/{bucket_id}/logs/new-result.txt"
    fs, path = fsspec.core.url_to_fs(url, skip_instance_cache=True)
    print(fs.exists(path))  # Expected: False; before this fix: KeyError
finally:
    delete_bucket(bucket_id, missing_ok=True)

Fix

Filter bucket tree results in _list_bucket_tree_with_folders() so entries are accepted only when the requested prefix is empty, matches exactly, or is followed by /. This keeps HfApi.list_bucket_tree()'s documented lexical behavior unchanged while restoring normal filesystem behavior:

  • exists() returns False
  • glob() returns no matches
  • ls() raises FileNotFoundError

Includes regression coverage using the existing data/ fixture and colliding missing prefix dat.

Validation

  • Prefix-collision regression: 1 passed
  • Full bucket filesystem suite: 38 passed, 2 skipped (2 transient staging 504 reruns)
  • make style: passed
  • make quality: passed

Note

Low Risk
Small, localized filter in bucket listing with a regression test; no auth or API contract changes.

Overview
Fixes bucket path prefix collisions in HfFileSystem when list_bucket_tree returns lexically similar paths (e.g. listing dat also returns data/...).

In _list_bucket_tree_with_folders, results are now filtered so only entries that equal the prefix or sit under prefix/ are kept—restoring normal filesystem behavior for exists, glob, and ls without changing the API’s lexical prefix behavior.

Adds test_prefix_collision against the shared data/ fixture and the missing prefix dat.

Reviewed by Cursor Bugbot for commit fd16996. Bugbot is set up for automated code reviews on this repo. Configure here.

@bot-ci-comment

bot-ci-comment Bot commented Aug 4, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lewtun
lewtun marked this pull request as ready for review August 4, 2026 21:55
@lewtun

lewtun commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

The failing tests seem unrelated to this PR AFAICT

@lewtun
lewtun requested a review from Wauplin August 5, 2026 18:45
@lewtun
lewtun requested a review from hanouticelina August 6, 2026 15:47

@hanouticelina hanouticelina left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks good to me, thanks!

Comment thread src/huggingface_hub/hf_file_system.py Resolved
@hanouticelina
hanouticelina merged commit db2c685 into main Aug 7, 2026
22 of 26 checks passed
@hanouticelina
hanouticelina deleted the agent/fix-hf-filesystem-bucket-prefix-collision branch August 7, 2026 09:05
@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v1.27.0 release.

Sanjays2402 added a commit to Sanjays2402/huggingface_hub that referenced this pull request Aug 27, 2026
The server matches prefix lexically, so a prefix of 'logs' also returns
'logs.json', 'logs_backup/...', and 'logsx/...'.  Every other consumer
in the library (HfFileSystem, _buckets._list_remote_files, hf_api.copy,
cli/_skills) compensates with a path-component filter; the destructive
'remove --recursive' branch was the missing one, causing silent
over-deletion.

Add the same filter already used by HfFileSystem (db2c685, huggingface#4630) so
that only the prefix itself and files under prefix/ are removed.

The test fixture now includes lexical siblings ('logs.json',
'logs_backup/a.log', 'logsx/c.log') so the suite structurally
catches the over-deletion bug.

Closes huggingface#4749.
Sanjays2402 added a commit to Sanjays2402/huggingface_hub that referenced this pull request Aug 27, 2026
The server matches prefix lexically, so a prefix of 'logs' also returns
'logs.json', 'logs_backup/...', and 'logsx/...'.  Every other consumer
in the library (HfFileSystem, _buckets._list_remote_files, hf_api.copy,
cli/_skills) compensates with a path-component filter; the destructive
'remove --recursive' branch was the missing one, causing silent
over-deletion.

Add the same filter already used by HfFileSystem (db2c685, huggingface#4630) so
that only the prefix itself and files under prefix/ are removed.

The test fixture now includes lexical siblings ('logs.json',
'logs_backup/a.log', 'logsx/c.log') so the suite structurally
catches the over-deletion bug.

Closes huggingface#4749.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants