-
-
Notifications
You must be signed in to change notification settings - Fork 4.7k
feat: Add new methods to list distinct mounts and retrieve all files in a mount #51810
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
4b2548c to
692da35
Compare
kyteinsky
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good!
just a few nits.
|
Not relying on OC stuff anymore in context_chat seems way better 👍 |
998ddf8 to
a8a93e6
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
4238cc9 to
4588ce8
Compare
78fb3b6 to
e6ca5c2
Compare
|
Hey @skjnldsv @ArtificialOwl @skjnldsv Any of you perhaps willing to give this a review? It has already been reviewed by Anupam and Julien. I can walk you through the code in a meeting if that would help. It's an important step for our team's projects. Thanks! |
|
@marcelklehr I would ask for @sorbaugh to schedule a review call with fs experts such as Robin |
|
Marcel is on vacation, sent a ping to Stephan about this. |
…all files in a mount Signed-off-by: Marcel Klehr <[email protected]>
…mounts Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Marcel Klehr <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
…ere is no trashbin mount Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
…xcludeTrashbinMounts param of getDistinctMounts Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
…non-user files mounts Signed-off-by: Marcel Klehr <[email protected]>
c91cb48 to
43be97d
Compare
|
Awesome. Tests in nextcloud/context_chat#142 pass |
Summary
These methods are needed for context chat and recognize to efficiently enumerate all files in the nextcloud instance.
In context chat and recognize we need to index all files of all users that fit certain criteria. To do this efficiently, we enumerate all essential mounts like home folders, external storage and groupfolders, excluding stuff like shares and trashbin. For this I implemented the getDistinctMounts method. It allows to filter by mount provider, so we can exclude shares etc. It also allows to adjust (override) home directory mounts to the actual files folder in a home directory, so we don't catch stuff that the user cannot see.
Once we have the distinct mounts, we enumerate all files non-recursively for each mount with getFilesByAncestorInStorage, it can filter by mimetypes and encryption state, and also takes a limit as well as a fileId cursor.
This is currently implemented in the context chat app, but as it makes use of the CacheQuery builder and core tables, the integrations team thinks it would be beneficial to have it in server.
Todo
Checklist