Skip to content

Recommend upload_folder / hf upload for large uploads - #2832

Merged
davanstrien merged 1 commit into
huggingface:mainfrom
davanstrien:docs/large-upload-guidance
Sep 30, 2026
Merged

davanstrien merged 1 commit into
huggingface:mainfrom
davanstrien:docs/large-upload-guidance

Conversation

@davanstrien

@davanstrien davanstrien commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

upload_large_folder was deprecated in huggingface_hub #4414 and removed in 2.0 (#4955). upload_folder() and hf upload now handle large folders themselves: they split the upload into several commits and resume if interrupted.

This updates the two places in the Hub docs that still point to the old approach:

  • datasets-adding.md: the "Large scale datasets" section now recommends upload_folder() / hf upload, and the link text "how to upload a folder by chunks" becomes "how to upload a large folder" (it already points to that section).
  • datasets-upload-guide-llm.md: "via UI or upload_large_folder" becomes "via the UI, upload_folder or hf upload". This page is aimed at LLM agents, so it shouldn't name a removed method.

🤖 Generated with Claude Code


Note

Low Risk
Documentation-only wording and link text; no runtime or API behavior changes in this repo.

Overview
Updates Hub dataset docs to match huggingface_hub 2.0, where upload_large_folder was removed in favor of upload_folder() and hf upload.

In datasets-adding.md, the Large scale datasets section now tells readers to use those APIs for large folders (multi-commit uploads with automatic resume) and renames the guide link from “upload a folder by chunks” to “how to upload a large folder”.

In datasets-upload-guide-llm.md, the Dataset Viewer note for direct Hub uploads no longer mentions the removed API and instead lists the UI, upload_folder, and hf upload.

Reviewed by Cursor Bugbot for commit 1a3c3dd. Bugbot is set up for automated code reviews on this repo. Configure here.

upload_large_folder is deprecated; upload_folder and hf upload now
split large uploads into several commits and resume if interrupted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davanstrien
davanstrien requested a review from Wauplin September 30, 2026 11:03
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Wauplin Wauplin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay! 🎉

@davanstrien
davanstrien merged commit eba4110 into huggingface:main Sep 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants