perf(preview): Split preview data to new table #54543

CarlSchwan · 2025-08-20T15:38:25Z

Resolves: Split previews out of the filecache table #54513

Summary

The new oc_previews table is optimized for storing previews and should decrease significantly the space taken by previews in the filecache table.

This attend to reuse the IObjectStore abstraction over S3/Swift/Azure but currently only support one single bucket configuration.

TODO

Implement multibucket configuration
Implement multibucket configuration where previews are stored in different bucket than the original files
Add migration for local storage
Add migration for object store storage
Add migration for multibucket object store storage
Make migration efficient
Adapt/Add unit tests
Re-thing BackgroundCleanupJob and other jobs and commands

TODO moved to seperate MR

Sharding: we want to do the sharding on the file_id table. Likely could be done with a tool like citus or vitess. Need some research
Server side encryption: currently we don't use SSE on the previews. If we end up implementing SSE for previews, we should use the same key file as the original files and add a seperate IPreviewStorage implementation using the IEncryptionModule. I already added an encrypted bool to the oc_previews table in preparation as it would be needed anyway for the migration

Checklist

Code is properly formatted
Sign-off message is added to all commits
Tests (unit, integration, api and/or acceptance) are included
~~Screenshots before/after for front-end changes~~ Should not contains any visual changes ;)
Documentation (manuals or wiki) has been updated or is not required
~~Backports requested where applicable (ex: critical bugfixes)~~

Altahrim

Quick thoughts on the new table

core/Migrations/Version33000Date20250819110529.php

CarlSchwan · 2025-09-01T18:36:21Z

@icewind1991 some questions for you :)

With allow configuring multiple object store backends #52786 there is now the possibility of using multiple backends per user. The current mapping is done using user configs (oc_preferences) and previews are still stored in the 'root' object store. Should we add a column to the preview table for the backend used (e.g. 'default', 'new', 'old', 'root', ...) to allow an easier migration of the previews in the future? Or do we keeping using only the 'root' backend for previews?
Multi bucket support. Currently either all the previews are stored on the root bucket or with an opt-in flag can be spread out in 256. I keep the exact same mapping in the new system, does it make sense? Or do we want to use the opportunity of having a breaking change to change that?
Sharding: I added a storage_id field to the column with sharding in mind. I didn't configure yet anything else or test it. Should I had it to SHARD_PRESETS in Connection.php?
Generally how do you feel about the direction this is going?

github-actions · 2025-09-10T08:58:15Z

Possible performance regression detected

Show Output

An unhandled exception has been thrown:
TypeError: array_map(): Argument #2 ($array) must be of type array, null given in /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php:35
Stack trace:
#0 /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php(35): array_map()
#1 /home/runner/work/server/server/3rdparty/symfony/console/Command/Command.php(326): OCAProfilerCommandCompare->execute()
#2 /home/runner/work/server/server/core/Command/Base.php(218): SymfonyComponentConsoleCommandCommand->run()
#3 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(1078): OCCoreCommandBase->run()
#4 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(324): SymfonyComponentConsoleApplication->doRunCommand()
#5 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(175): SymfonyComponentConsoleApplication->doRun()
#6 /home/runner/work/server/server/lib/private/Console/Application.php(187): SymfonyComponentConsoleApplication->run()
#7 /home/runner/work/server/server/console.php(91): OCConsoleApplication->run()
#8 /home/runner/work/server/server/occ(33): require_once('...')
#9 {main}

github-actions · 2025-09-11T13:33:27Z

Possible performance regression detected

Show Output

An unhandled exception has been thrown:
TypeError: array_map(): Argument #2 ($array) must be of type array, null given in /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php:35
Stack trace:
#0 /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php(35): array_map()
#1 /home/runner/work/server/server/3rdparty/symfony/console/Command/Command.php(326): OCAProfilerCommandCompare->execute()
#2 /home/runner/work/server/server/core/Command/Base.php(218): SymfonyComponentConsoleCommandCommand->run()
#3 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(1078): OCCoreCommandBase->run()
#4 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(324): SymfonyComponentConsoleApplication->doRunCommand()
#5 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(175): SymfonyComponentConsoleApplication->doRun()
#6 /home/runner/work/server/server/lib/private/Console/Application.php(187): SymfonyComponentConsoleApplication->run()
#7 /home/runner/work/server/server/console.php(91): OCConsoleApplication->run()
#8 /home/runner/work/server/server/occ(33): require_once('...')
#9 {main}

artonge

~~- Do we have a way to garbage collect orphan previews?~~ Was answered during our call.

core/BackgroundJobs/MovePreviewJob.php

The new oc_previews table is optimized for storing previews and should decrease significantly the space taken by previews in the filecache table. This attend to reuse the IObjectStore abstraction over S3/Swift/Azure but currently only support one single bucket configuration. Signed-off-by: Carl Schwan <[email protected]>

Signed-off-by: Carl Schwan <[email protected]>

* Simplify migration by not moving the actual files and just updating the DB * Don't store the storageid in the preview table as it is not needed * Start adding tests Signed-off-by: Carl Schwan <[email protected]>

Signed-off-by: Carl Schwan <[email protected]>

This work similarly to the move preview job to migrate the previews to the new DB table and also reuse some code. So when we are finding files in appdata/preview, try adding them to the oc_previews table and delete them from the oc_filecache table. Signed-off-by: Carl Schwan <[email protected]>

Allow to quickly query all the files from a specific mimetype like in the ResetRenderedTexts command. Signed-off-by: Carl Schwan <[email protected]>

Signed-off-by: Carl Schwan <[email protected]>

And move it to a different table so that we don't have to pay the storage cost when not using it (most of the times). Signed-off-by: Carl Schwan <[email protected]>

Simplify logic Signed-off-by: Carl Schwan <[email protected]>

szaimen · 2025-10-08T14:39:35Z

🎉🎉🎉🎉🎉

ChristophWurst · 2025-10-08T15:13:46Z

core/Migrations/Version33000Date20250819110529.php

+		if (!$schema->hasTable('preview_versions')) {
+			$table = $schema->createTable('preview_versions');
+			$table->addColumn('id', Types::BIGINT, ['autoincrement' => true, 'notnull' => true, 'length' => 20, 'unsigned' => true]);
+			$table->addColumn('file_id', Types::BIGINT, ['notnull' => true, 'length' => 20, 'unsigned' => true]);


Uneducated guess, but an this smells like it will be the primary column to look up rows. Add an index?

The mapper confirms that an index on file_id+version would be beneficial :)

^ @CarlSchwan

CarlSchwan added this to the Nextcloud 33 milestone Aug 20, 2025

CarlSchwan self-assigned this Aug 20, 2025

CarlSchwan added the 2. developing Work in progress label Aug 20, 2025

Altahrim reviewed Aug 21, 2025

View reviewed changes

CarlSchwan force-pushed the preview-db-rework branch 3 times, most recently from 43b105b to 0447f56 Compare August 27, 2025 15:14

CarlSchwan force-pushed the preview-db-rework branch 2 times, most recently from 663dfd5 to e55c56d Compare August 28, 2025 17:42

AndyScherzinger added the performance 🚀 label Aug 30, 2025

CarlSchwan force-pushed the preview-db-rework branch from f6ddec7 to 24dc8a1 Compare September 1, 2025 08:14

CarlSchwan force-pushed the preview-db-rework branch 5 times, most recently from 9b52d71 to 7d3c7e8 Compare September 17, 2025 09:06

CarlSchwan added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Sep 17, 2025

CarlSchwan marked this pull request as ready for review September 17, 2025 09:14

CarlSchwan requested a review from a team as a code owner September 17, 2025 09:14

CarlSchwan requested review from icewind1991, provokateurin and yemkareems and removed request for a team September 17, 2025 09:14

CarlSchwan force-pushed the preview-db-rework branch from 7d3c7e8 to 296b349 Compare September 17, 2025 09:21

artonge reviewed Sep 17, 2025

View reviewed changes

core/BackgroundJobs/MovePreviewJob.php Outdated Show resolved Hide resolved

CarlSchwan force-pushed the preview-db-rework branch from 296b349 to f90075f Compare September 17, 2025 12:03

Carl Schwan and others added 12 commits October 6, 2025 13:37

perf(preview): Add support for multibucket storage

656e33e

Signed-off-by: Carl Schwan <[email protected]>

perf(preview): Migrate previews to the new optimized table

13c35c0

Signed-off-by: Carl Schwan <[email protected]>

feat(preview): Support multibucket storage

6008852

Signed-off-by: Carl Schwan <[email protected]>

perf(preview): Adapt BackgroundCleanupJob to new previews table

bba9667

Signed-off-by: Carl Schwan <[email protected]>

perf(preview): Optimize migration and simplify DB layout

b035766

* Simplify migration by not moving the actual files and just updating the DB * Don't store the storageid in the preview table as it is not needed * Start adding tests Signed-off-by: Carl Schwan <[email protected]>

fix(preview): Fix some tests

6f56dcf

Signed-off-by: Carl Schwan <[email protected]>

refactor(preview): Cleanup the implementation of the new preview backend

324b54b

Signed-off-by: Carl Schwan <[email protected]>

feat(preview): Store original file mimetype in preview table

5802378

Allow to quickly query all the files from a specific mimetype like in the ResetRenderedTexts command. Signed-off-by: Carl Schwan <[email protected]>

refactor: Use Override annotation in new preview code

bd001c9

Signed-off-by: Carl Schwan <[email protected]>

refactor(preview): Use same mimetype ids as filecache

66f50bd

Signed-off-by: Carl Schwan <[email protected]>

CarlSchwan force-pushed the preview-db-rework branch 4 times, most recently from b1c9a14 to 9b63d58 Compare October 8, 2025 08:52

artonge approved these changes Oct 8, 2025

View reviewed changes

CarlSchwan force-pushed the preview-db-rework branch from 9b63d58 to 12b77eb Compare October 8, 2025 09:43

leftybournes approved these changes Oct 8, 2025

View reviewed changes

CarlSchwan force-pushed the preview-db-rework branch from 12b77eb to f91ea9e Compare October 8, 2025 11:24

CarlSchwan added 2 commits October 8, 2025 14:07

fix(preview): Make version column a string

bef3996

And move it to a different table so that we don't have to pay the storage cost when not using it (most of the times). Signed-off-by: Carl Schwan <[email protected]>

refactor(preview-object-store): Refactor object store backend

fed7a33

Simplify logic Signed-off-by: Carl Schwan <[email protected]>

CarlSchwan force-pushed the preview-db-rework branch from f91ea9e to fed7a33 Compare October 8, 2025 12:07

icewind1991 approved these changes Oct 8, 2025

View reviewed changes

Altahrim merged commit 3aa0c23 into master Oct 8, 2025
213 of 221 checks passed

Altahrim deleted the preview-db-rework branch October 8, 2025 14:39

ChristophWurst reviewed Oct 8, 2025

View reviewed changes

artonge mentioned this pull request Oct 14, 2025

fix(Viewer): Provide id instead of fileid to File constructor nextcloud/viewer#2988

Merged

artonge mentioned this pull request Oct 29, 2025

Split previews out of the filecache table #54513

Open

Uh oh!

perf(preview): Split preview data to new table #54543

perf(preview): Split preview data to new table #54543

Uh oh!

Conversation

CarlSchwan commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

TODO

TODO moved to seperate MR

Checklist

Uh oh!

Altahrim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CarlSchwan commented Sep 1, 2025

Uh oh!

github-actions bot commented Sep 10, 2025

Uh oh!

github-actions bot commented Sep 11, 2025

Uh oh!

artonge left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

szaimen commented Oct 8, 2025

Uh oh!

ChristophWurst Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

ChristophWurst Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

AndyScherzinger Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

CarlSchwan commented Aug 20, 2025 •

edited

Loading

artonge left a comment •

edited

Loading