Skip to content

Transparent* database sharding #46639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sorbaugh merged 18 commits into from
Aug 28, 2024
Merged

Transparent* database sharding #46639

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
c09ec95
feat: track expected output columns in query builder
icewind1991 Jul 31, 2024
114db05
fix: don't make ICacheFactory depend on database
icewind1991 Aug 8, 2024
c58bdbf
fix: delay calculating global cache prefix untill a cache is created
icewind1991 Aug 20, 2024
f5b3486
feat: add option to automatically partition queries by specific tables
icewind1991 Jun 13, 2024
62f8b65
feat: implement distributing partitioned queries over multiple shards
icewind1991 Jul 31, 2024
ddbeb4c
test: mark share test cleanup as running across all shards
icewind1991 Jul 16, 2024
fc05a67
fix: only allow pre-defined shards
icewind1991 Jul 18, 2024
4d9b563
test: run sharding tests in ci
icewind1991 Jul 18, 2024
390f6a7
fix: hint storage id in more places
icewind1991 Jul 19, 2024
2eaeeee
fix: run mimetype repair query across all shards
icewind1991 Jul 19, 2024
382d102
test: fix share provider tests for sharding
icewind1991 Jul 19, 2024
80a2553
fix: make background scan job compatible with sharding
icewind1991 Jul 25, 2024
e538f46
fix: adjust systemtag orphan cleanup query to work with sharding
icewind1991 Jul 31, 2024
cc091b1
fix: fix share cleanup for deleted groups with sharding
icewind1991 Aug 6, 2024
b21a399
fix: implement sharding compatible cleanup for various bits
icewind1991 Aug 15, 2024
1363e14
fix: make preload custom proterties sharding compatible
icewind1991 Aug 21, 2024
9d02485
fix: mark systemconfig value as not being tainted because they are im…
icewind1991 Aug 22, 2024
2574cbf
chore: Apply php:cs recommendations
artonge Aug 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: hint storage id in more places 
Signed-off-by: Robin Appelman <[email protected]>
  • Loading branch information
@icewind1991 @artonge
icewind1991 authored and artonge committed Aug 28, 2024
commit 390f6a78b48e4c071f1b18d2fc230aa24eb3f3bb
7 changes: 5 additions & 2 deletions lib/private/Files/Cache/Cache.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,7 @@ public function update($id, array $data) {
$query = $this->getQueryBuilder();
$query->update('filecache_extended')
->whereFileId($id)
->hintShardKey('storage', $this->getNumericStorageId())
->andWhere($query->expr()->orX(...array_map(function ($key, $value) use ($query) {
return $query->expr()->orX(
$query->expr()->neq($key, $query->createNamedParameter($value)),
Expand Down Expand Up @@ -523,7 +524,8 @@ public function remove($file) {

$query = $this->getQueryBuilder();
$query->delete('filecache_extended')
->whereFileId($entry->getId());
->whereFileId($entry->getId())
->hintShardKey('storage', $this->getNumericStorageId());
$query->execute();

if ($entry->getMimeType() == FileInfo::MIMETYPE_FOLDER) {
Expand Down Expand Up @@ -567,7 +569,8 @@ private function removeChildren(ICacheEntry $entry) {

$query = $this->getQueryBuilder();
$query->delete('filecache_extended')
->where($query->expr()->in('fileid', $query->createParameter('childIds')));
->where($query->expr()->in('fileid', $query->createParameter('childIds')))
->hintShardKey('storage', $this->getNumericStorageId());

foreach (array_chunk($childIds, 1000) as $childIdChunk) {
$query->setParameter('childIds', $childIdChunk, IQueryBuilder::PARAM_INT_ARRAY);
Expand Down