Skip to content

Conversation

@icewind1991
Copy link
Member

Multipart copy seems problematic with some backends so avoid it when possible.

@icewind1991 icewind1991 added the 3. to review Waiting for reviews label Nov 29, 2023
@icewind1991 icewind1991 added this to the Nextcloud 29 milestone Nov 29, 2023
@icewind1991 icewind1991 requested review from a team, ArtificialOwl, come-nc, juliusknorr and sorbaugh and removed request for a team November 29, 2023 16:33
@icewind1991
Copy link
Member Author

/backport to stable28

@icewind1991
Copy link
Member Author

/backport to stable27

Copy link
Member

@solracsf solracsf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't test if MultiPart is supported? Something like:

if ($this->getObjectStore() instanceof IObjectStoreMultiPartUpload) {
   // multipart
} else {
  // not multipart
}

Copy link
Member

@juliusknorr juliusknorr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems good, do you know what s3 providers/solutions enforce such a limit?

@juliusknorr
Copy link
Member

Why don't test if MultiPart is supported?

It might need a different interface, I'm not sure if all providers that support multipart upload also support multipart copy

@icewind1991
Copy link
Member Author

do you know what s3 providers/solutions enforce such a limit?

No, but there are at least 2 different cases where the multipart copy is failing

Why don't test if MultiPart is supported? Something like:

I'm not aware of any good way to tell if an S3 implementation supports it

@Ziyann
Copy link

Ziyann commented Dec 13, 2023

For reference, I'm also affected by this issue, using Google Cloud Storage (but haven't tested this patch yet). Some built-in solution would be greatly appreciated, either a configurable limit like copySizeLimit here (which we could set to a very high value, thus never triggering multipart copy), or even better, some enable/disable switches for the various operations, because right now I'm manually reverting #40577 on production after upgrades.
There are many S3 providers with different features supported, so I think it would be great to have as many configuration options as possible (avoiding manual file changes on production).

@Ziyann
Copy link

Ziyann commented Dec 19, 2023

You should propagate copySizeLimit to the S3 SDK (mup_threshold), otherwise, it will use the hardcoded 5 GB threshold at some places: https://github.com/nextcloud/3rdparty/blob/v28.0.0/aws/aws-sdk-php/src/S3/ObjectCopier.php#L96-L103
See: #42068 (comment)

@juliusknorr
Copy link
Member

Pushed an additional commit to be able to fully turn off multipart copy for s3 providers not supporting it at all.

@blizzz blizzz mentioned this pull request Mar 5, 2024
joshtrichards added a commit to nextcloud/documentation that referenced this pull request Nov 22, 2024
backportbot bot pushed a commit to nextcloud/documentation that referenced this pull request Dec 2, 2024
backportbot bot pushed a commit to nextcloud/documentation that referenced this pull request Dec 2, 2024
backportbot bot pushed a commit to nextcloud/documentation that referenced this pull request Dec 2, 2024
susnux pushed a commit to nextcloud/documentation that referenced this pull request Mar 24, 2025
susnux pushed a commit to nextcloud/documentation that referenced this pull request Mar 24, 2025
susnux pushed a commit to nextcloud/documentation that referenced this pull request Mar 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review Waiting for reviews pending documentation This pull request needs an associated documentation update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants