Repository navigation
[Buckets] Fix copy_files treating lexical siblings as an existing destination directory - #5001
Merged
Merged
Conversation
…tination directory
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Contributor
|
This PR has been shipped as part of the v2.1.0 release. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
HfApi._copy_to_bucketdecides whether a non-file destination is an existing directory by checking whetherlist_bucket_tree(prefix=destination_path, recursive=False)returns anything. The Hub matchesprefixlexically, including for non-recursive listings (see #4630 and #4804), so a sibling likelogs.jsonorlogs_backup/makes a missing destinationlogslook like an existing directory:copy_files("hf://buckets/u/src/a.txt", "hf://buckets/u/dst/logs")writeslogs/a.txtinstead oflogs.copy_files("hf://buckets/u/src/data", "hf://buckets/u/dst/logs")nests the folder aslogs/data/...instead of using rename semantics (logs/...).Fix
Only count listed entries that are
destination_pathitself or below it (path == destination_path or path.startswith(destination_path + "/")). This is the same path-boundary filter used by_iter_under_prefix(hf buckets ls/rm),HfFileSystem._list_bucket_tree_with_folders,_buckets._list_remote_files, and the source loop of_copy_to_bucket. The destination check was the last caller without it.Behaviour when the destination really exists as a folder is unchanged: with
recursive=Falsethe server returns thelogsfolder entry itself, which passes the filter.Tests
Added
test_copy_files_to_bucket_ignores_lexical_siblings_of_destinationtotests/test_copy_files.py. It is a mocked unit test (no staging needed) wherelist_bucket_treereproduces the server's lexical prefix matching. It covers four cases:logswith only siblings present: expectslogs.logswith only siblings present: expects rename,logs/x.txt.logs/: expectslogs/a.txt(control).logs/: expects nesting,logs/data/x.txt(control).The two bug cases fail on
mainand pass with this change; the two controls pass on both.ruff formatandruff checkare clean.Closes #5000.
Note
Medium Risk
Changes path resolution for bucket
copy_fileswhen destinations are ambiguous; behavior is more correct but could differ from what users relied on when siblings falsely implied a directory.Overview
Fixes bucket copy when the destination path does not exist but lexical prefix siblings do (e.g. copying to
logswhilelogs.jsonorlogs_backup/is present).HfApi._copy_to_bucketno longer treats anylist_bucket_tree(prefix=destination_path)hit as “destination is an existing directory”; it only counts paths that are exactlydestination_pathor underdestination_path/.That restores expected
copy_filesbehavior: a missinglogsgets a file namedlogsor folder rename semantics (logs/x.txt), instead of wrongly nesting aslogs/a.txtorlogs/data/.... Copies into a reallogs/folder are unchanged.Adds a mocked unit test (
test_copy_files_to_bucket_ignores_lexical_siblings_of_destination) for file vs folder sources with siblings-only vs reallogs/destination.Reviewed by Cursor Bugbot for commit 553ff84. Bugbot is set up for automated code reviews on this repo. Configure here.