Skip to content

Implement multibucket shift for ObjectStore #29622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
PVince81 merged 1 commit into from
Nov 19, 2021
Merged

Implement multibucket shift for ObjectStore #29622

PVince81 merged 1 commit into from
Nov 19, 2021

Conversation

@skjnldsv
Copy link
Member

@skjnldsv skjnldsv commented Nov 10, 2021 •
edited
Loading

On a multibucket setup:

  • As we don't evenly distribute the data but we randomly spread accross all buckets,
  • And when the first buckets are filled

It might be interesting to add new ones without filling the old ones.
The number of buckets stay the same, but we shift the number.

Example given:

'objectstore_multibucket' => array(
        'class' => 'Object\\Storage\\Backend\\Class',
        'arguments' => array(
                // optional, defaults to 64
                'num_buckets' => 128,
                // int, shift the bucket number (default 0)
                'min_bucket' => 64,
                // n integer in the range from 0 to (num_buckets-1) will be appended
                'bucket' => 'nextcloud_',
                ...
        ),
),

This will write to buckets from 64 to 127. Still using 64 buckets

@skjnldsv skjnldsv requested review from a team, CarlSchwan, icewind1991 and nickvergessen and removed request for a team November 10, 2021 11:26
@skjnldsv skjnldsv self-assigned this Nov 10, 2021
@skjnldsv skjnldsv force-pushed the feat/objectstore_multibucket-shift branch 3 times, most recently from 970667e to fe5d705 Compare November 10, 2021 12:05
@icewind1991

This comment has been minimized.

Sign in to view
@skjnldsv skjnldsv force-pushed the feat/objectstore_multibucket-shift branch from fe5d705 to 0244e9e Compare November 10, 2021 21:46
@skjnldsv

This comment has been minimized.

Sign in to view
@skjnldsv skjnldsv force-pushed the feat/objectstore_multibucket-shift branch from 0244e9e to 660c7e1 Compare November 10, 2021 21:52
@skjnldsv skjnldsv force-pushed the feat/objectstore_multibucket-shift branch from 660c7e1 to 6ab2fea Compare November 11, 2021 14:09
@skjnldsv skjnldsv requested a review from a team November 15, 2021 08:43
@skjnldsv skjnldsv added this to the Nextcloud 24 milestone Nov 15, 2021
@skjnldsv
Copy link
Member Author

skjnldsv commented Nov 15, 2021

/backport to stable23

@skjnldsv
Copy link
Member Author

skjnldsv commented Nov 15, 2021

/backport to stable22

@skjnldsv
Copy link
Member Author

skjnldsv commented Nov 15, 2021

/backport to stable21

@skjnldsv
Copy link
Member Author

skjnldsv commented Nov 15, 2021

/backport to stable20

PVince81
PVince81 approved these changes Nov 19, 2021
View reviewed changes
Copy link
Member

@PVince81 PVince81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense 👍

@PVince81 PVince81 merged commit c42a890 into master Nov 19, 2021
@PVince81 PVince81 deleted the feat/objectstore_multibucket-shift branch November 19, 2021 16:26
@backportbot-nextcloud
Copy link

backportbot-nextcloud bot commented Nov 19, 2021

The backport to stable23 failed. Please do this backport manually.

@backportbot-nextcloud
Copy link

backportbot-nextcloud bot commented Nov 19, 2021

The backport to stable22 failed. Please do this backport manually.

@backportbot-nextcloud
Copy link

backportbot-nextcloud bot commented Nov 19, 2021

The backport to stable21 failed. Please do this backport manually.

@backportbot-nextcloud
Copy link

backportbot-nextcloud bot commented Nov 19, 2021

The backport to stable20 failed. Please do this backport manually.

@skjnldsv skjnldsv modified the milestones: Nextcloud 24, Nextcloud 23 Jan 14, 2022
@skjnldsv
Copy link
Member Author

skjnldsv commented Jan 14, 2022

/backport to stable23

@backportbot-nextcloud
Copy link

backportbot-nextcloud bot commented Jan 14, 2022

The backport to stable23 failed. Please do this backport manually.

@skjnldsv
Copy link
Member Author

skjnldsv commented Jan 14, 2022

Weird it works here locally, I'm curious about the bot failure

@skjnldsv
Copy link
Member Author

skjnldsv commented Jan 14, 2022

/backport to stable23

@backportbot-nextcloud
Copy link

backportbot-nextcloud bot commented Jan 14, 2022

The backport to stable23 failed. Please do this backport manually.

This was referenced Jan 14, 2022
Merged
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PVince81 PVince81 PVince81 approved these changes

@icewind1991 icewind1991 icewind1991 approved these changes

@nickvergessen nickvergessen Awaiting requested review from nickvergessen

@CarlSchwan CarlSchwan Awaiting requested review from CarlSchwan

Assignees

@skjnldsv skjnldsv

Labels

3. to review Waiting for reviews enhancement feature: object storage

Projects

None yet

Milestone

Nextcloud 24

Development

Successfully merging this pull request may close these issues.

4 participants

@skjnldsv @icewind1991 @PVince81