Skip to content

[Buckets] Fix copy_files treating lexical siblings as an existing destination directory - #5001

Merged
Wauplin merged 1 commit into
huggingface:mainfrom
AnishPatel526:fix-copy-dest-lexical
Sep 25, 2026
Merged

Wauplin merged 1 commit into
huggingface:mainfrom
AnishPatel526:fix-copy-dest-lexical

Conversation

@AnishPatel526

@AnishPatel526 AnishPatel526 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Description

HfApi._copy_to_bucket decides whether a non-file destination is an existing directory by checking whether list_bucket_tree(prefix=destination_path, recursive=False) returns anything. The Hub matches prefix lexically, including for non-recursive listings (see #4630 and #4804), so a sibling like logs.json or logs_backup/ makes a missing destination logs look like an existing directory:

  • copy_files("hf://buckets/u/src/a.txt", "hf://buckets/u/dst/logs") writes logs/a.txt instead of logs.
  • copy_files("hf://buckets/u/src/data", "hf://buckets/u/dst/logs") nests the folder as logs/data/... instead of using rename semantics (logs/...).

Fix

Only count listed entries that are destination_path itself or below it (path == destination_path or path.startswith(destination_path + "/")). This is the same path-boundary filter used by _iter_under_prefix (hf buckets ls / rm), HfFileSystem._list_bucket_tree_with_folders, _buckets._list_remote_files, and the source loop of _copy_to_bucket. The destination check was the last caller without it.

Behaviour when the destination really exists as a folder is unchanged: with recursive=False the server returns the logs folder entry itself, which passes the filter.

Tests

Added test_copy_files_to_bucket_ignores_lexical_siblings_of_destination to tests/test_copy_files.py. It is a mocked unit test (no staging needed) where list_bucket_tree reproduces the server's lexical prefix matching. It covers four cases:

  • File to missing logs with only siblings present: expects logs.
  • Folder to missing logs with only siblings present: expects rename, logs/x.txt.
  • File to an existing logs/: expects logs/a.txt (control).
  • Folder to an existing logs/: expects nesting, logs/data/x.txt (control).

The two bug cases fail on main and pass with this change; the two controls pass on both. ruff format and ruff check are clean.

Closes #5000.


Note

Medium Risk
Changes path resolution for bucket copy_files when destinations are ambiguous; behavior is more correct but could differ from what users relied on when siblings falsely implied a directory.

Overview
Fixes bucket copy when the destination path does not exist but lexical prefix siblings do (e.g. copying to logs while logs.json or logs_backup/ is present). HfApi._copy_to_bucket no longer treats any list_bucket_tree(prefix=destination_path) hit as “destination is an existing directory”; it only counts paths that are exactly destination_path or under destination_path/.

That restores expected copy_files behavior: a missing logs gets a file named logs or folder rename semantics (logs/x.txt), instead of wrongly nesting as logs/a.txt or logs/data/.... Copies into a real logs/ folder are unchanged.

Adds a mocked unit test (test_copy_files_to_bucket_ignores_lexical_siblings_of_destination) for file vs folder sources with siblings-only vs real logs/ destination.

Reviewed by Cursor Bugbot for commit 553ff84. Bugbot is set up for automated code reviews on this repo. Configure here.

@Wauplin Wauplin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix!

@Wauplin
Wauplin merged commit 79874bc into huggingface:main Sep 25, 2026
12 of 21 checks passed
@bot-ci-comment

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v2.1.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

copy_files to a bucket treats a missing destination as a directory when a lexical sibling exists

2 participants