Skip to content

Avoid allocating too much memory for the buffer on s3 uploads #34537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
skjnldsv merged 1 commit into from
Oct 12, 2022
Merged

Avoid allocating too much memory for the buffer on s3 uploads #34537

skjnldsv merged 1 commit into from
Oct 12, 2022

Conversation

@juliusknorr
Copy link
Member

@juliusknorr juliusknorr commented Oct 11, 2022 •
edited
Loading

Otherwise we try to allocate ~500MB in memory which does fail on larger file uploads with our default memory limit. We only need the put file limit configured anyways.

Steps to reproduce on 24+:

  1. use s3 primary storage
  2. upload a 500mb file through the web ui or clients

Before that change the MOVE request ended in an Allowed Memory Size Exhausted error.

@juliusknorr
Copy link
Member Author

juliusknorr commented Oct 11, 2022

Regression from https://github.com/nextcloud/server/pull/28542/files#diff-030e38a9e621019981acf91654c1d8d3a12ab3150dbd9fcc4c280479b8c853dbR147

@juliusknorr juliusknorr requested review from a team, icewind1991 and skjnldsv and removed request for a team October 11, 2022 13:39
@juliusknorr
Copy link
Member Author

juliusknorr commented Oct 11, 2022

/backport to stable25

@juliusknorr
Copy link
Member Author

juliusknorr commented Oct 11, 2022

/backport to stable24

skjnldsv
skjnldsv approved these changes Oct 11, 2022
View reviewed changes
Copy link
Member

@skjnldsv skjnldsv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense! 👍

@juliusknorr juliusknorr changed the title Avoid allocating too much memory for the buffer Avoid allocating too much memory for the buffer on s3 uploads Oct 11, 2022
PVince81
PVince81 approved these changes Oct 11, 2022
View reviewed changes
Copy link
Member

@PVince81 PVince81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@skjnldsv skjnldsv added 4. to release Ready to be released and/or waiting for tests to finish and removed 3. to review Waiting for reviews labels Oct 12, 2022
@skjnldsv skjnldsv merged commit d936694 into master Oct 12, 2022
@skjnldsv skjnldsv deleted the bugfix/noid/s3-upload-memory branch October 12, 2022 08:24
This was referenced Oct 12, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PVince81 PVince81 PVince81 approved these changes

@skjnldsv skjnldsv skjnldsv approved these changes

@CarlSchwan CarlSchwan Awaiting requested review from CarlSchwan

@icewind1991 icewind1991 Awaiting requested review from icewind1991

Assignees

No one assigned

Labels

4. to release Ready to be released and/or waiting for tests to finish bug feature: object storage regression

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@juliusknorr @PVince81 @skjnldsv