Skip to content

Conversation

@marcelklehr
Copy link
Member

@marcelklehr marcelklehr commented Mar 31, 2025

Summary

These methods are needed for context chat and recognize to efficiently enumerate all files in the nextcloud instance.

In context chat and recognize we need to index all files of all users that fit certain criteria. To do this efficiently, we enumerate all essential mounts like home folders, external storage and groupfolders, excluding stuff like shares and trashbin. For this I implemented the getDistinctMounts method. It allows to filter by mount provider, so we can exclude shares etc. It also allows to adjust (override) home directory mounts to the actual files folder in a home directory, so we don't catch stuff that the user cannot see.

Once we have the distinct mounts, we enumerate all files non-recursively for each mount with getFilesByAncestorInStorage, it can filter by mimetypes and encryption state, and also takes a limit as well as a fileId cursor.

This is currently implemented in the context chat app, but as it makes use of the CacheQuery builder and core tables, the integrations team thinks it would be beneficial to have it in server.

Todo

  • Tests

Checklist

@marcelklehr marcelklehr requested a review from a team as a code owner March 31, 2025 11:44
@marcelklehr marcelklehr requested review from ArtificialOwl, skjnldsv and sorbaugh and removed request for a team March 31, 2025 11:44
@marcelklehr marcelklehr force-pushed the feat/getByAncestorInStorage branch from 4b2548c to 692da35 Compare March 31, 2025 12:53
@marcelklehr marcelklehr changed the title feat(IFileAccess#getByAncestorInStorage): Add new method to retrieve all files in a mount feat: Add new methods to rlist distinct mounts and retrieve all files in a mount Mar 31, 2025
@marcelklehr marcelklehr changed the title feat: Add new methods to rlist distinct mounts and retrieve all files in a mount feat: Add new methods to list distinct mounts and retrieve all files in a mount Mar 31, 2025
Copy link
Contributor

@kyteinsky kyteinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!
just a few nits.

@julien-nc
Copy link
Member

Not relying on OC stuff anymore in context_chat seems way better 👍
I can't say if those new methods are relevant but it looks like they could be used by other apps, in other contexts.

@marcelklehr marcelklehr force-pushed the feat/getByAncestorInStorage branch from 998ddf8 to a8a93e6 Compare April 1, 2025 08:41
@kyteinsky

This comment was marked as outdated.

@marcelklehr

This comment was marked as outdated.

@marcelklehr marcelklehr force-pushed the feat/getByAncestorInStorage branch 2 times, most recently from 4238cc9 to 4588ce8 Compare April 4, 2025 14:05
@marcelklehr marcelklehr force-pushed the feat/getByAncestorInStorage branch from 78fb3b6 to e6ca5c2 Compare April 10, 2025 13:16
@marcelklehr
Copy link
Member Author

Hey @skjnldsv @ArtificialOwl @skjnldsv Any of you perhaps willing to give this a review? It has already been reviewed by Anupam and Julien. I can walk you through the code in a meeting if that would help. It's an important step for our team's projects. Thanks!

@AndyScherzinger AndyScherzinger added this to the Nextcloud 32 milestone May 27, 2025
@skjnldsv
Copy link
Member

@marcelklehr I would ask for @sorbaugh to schedule a review call with fs experts such as Robin

@skjnldsv skjnldsv removed their request for review May 28, 2025 07:00
@kyteinsky
Copy link
Contributor

Marcel is on vacation, sent a ping to Stephan about this.

marcelklehr and others added 18 commits July 15, 2025 09:15
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
…ere is no trashbin mount

Signed-off-by: Julien Veyssier <[email protected]>
…xcludeTrashbinMounts param of getDistinctMounts

Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
@marcelklehr marcelklehr force-pushed the feat/getByAncestorInStorage branch from c91cb48 to 43be97d Compare July 15, 2025 07:15
@marcelklehr
Copy link
Member Author

Awesome. Tests in nextcloud/context_chat#142 pass

@AndyScherzinger AndyScherzinger merged commit af5acc3 into master Jul 15, 2025
232 of 243 checks passed
@AndyScherzinger AndyScherzinger deleted the feat/getByAncestorInStorage branch July 15, 2025 10:14
@nextcloud-bot nextcloud-bot mentioned this pull request Aug 19, 2025
@skjnldsv skjnldsv modified the milestones: Nextcloud 32, Nextcloud 33 Sep 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants