Skip to content

Commit ffd435a

Browse files
artongeartonge
committed
Add high level filesystem documention
From nextcloud/server#26982 Adds some documentation for the filesystem layer, both a high level overview of how the various pieces interact and a high level guide for apps interacting with the filesystem. Hopefully this will be useful to anyone trying to either use the filesystem or work on the filesystem. Signed-off-by: Louis Chemineau <[email protected]>
1 parent 15c3ff2 commit ffd435a

File tree

2 files changed

+243
-22
lines changed

2 files changed

+243
-22
lines changed

‎developer_manual/basics/storage/filesystem.rst‎

Lines changed: 94 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,79 @@
1-
==========
2-
Filesystem
3-
==========
1+
========================
2+
Nextcloud filesystem API
3+
========================
4+
45

56
.. sectionauthor:: Bernhard Posselt <[email protected]>
67

7-
Because users can choose their storage backend, the filesystem should be accessed by using the appropriate filesystem classes.
8+
High level guide to using the Nextcloud filesystem API.
9+
10+
Because users can choose their storage backend, the filesystem should be accessed by using the appropriate filesystem classes. For a simplified filesystem for app specific data see `IAppData <appdata.html>`_
11+
12+
Node API
13+
^^^^^^^^
14+
15+
The "Node API" is the primary api for apps to access the Nextcloud filesystem, each item in the filesystem is
16+
represented as either a File or Folder node with each node providing access to the relevant filesystem information
17+
and actions for the node.
18+
19+
20+
Getting access
21+
--------------
22+
23+
Access to the filesystem is provided by the ``IRootFolder`` which can be injected into your class.
24+
From the root folder you can either access a user's home folder or access a file or folder by its absolute path.
25+
26+
.. code-block:: php
27+
28+
use OCP\Files\IRootFolder;
29+
use OCP\IUserSession;
30+
31+
class FileSystemAccessExample {
32+
private IUserSession $userSession;
33+
private IRootFolder $rootFolder;
34+
35+
public function __constructor(IUserSession $userSession, IRootFolder $rootFolder) {
36+
$this->userSession = $userSession;
37+
$this->rootFolder = $rootFolder;
38+
}
39+
40+
/**
41+
* Create a new file with specified content in the home folder of the current user
42+
* returning the size of the resulting file.
43+
*/
44+
public function getCurrentUserFolder(string $path, string $content): int {
45+
$user = $this->userSession->getUser();
46+
47+
if ($user === null) {
48+
return null;
49+
}
50+
51+
// the "user folder" corresponds to the root of the user visible files
52+
return $this->rootFolder->getUserFolder($user->getUID());
53+
}
54+
}
55+
56+
For more details on the specific methods provided by file and folder nodes see the method documentation from the ``OCP\Files\File`` and ``OCP\Files\Folder`` interfaces.
857

9-
Filesystem classes can be injected automatically with dependency injection. This is the user filesystem.
10-
For a simplified filestystem for app specific data see `IAppData <appdata.html>`_
1158

1259
Writing to a file
1360
-----------------
1461

15-
1662
All methods return a Folder object on which files and folders can be accessed, or filesystem operations can be performed relatively to their root. For instance for writing to file:`nextcloud/data/myfile.txt` you should get the root folder and use:
1763

1864
.. code-block:: php
1965
20-
<?php
21-
namespace OCA\MyApp\Storage;
22-
2366
use OCP\Files\IRootFolder;
2467
25-
class AuthorStorage {
68+
class FileWritingExample {
2669
27-
/** @var IRootStorage */
28-
private $storage;
70+
private IRootStorage $storage;
2971
3072
public function __construct(IRootFolder $storage){
3173
$this->storage = $storage;
3274
}
3375
34-
public function writeTxt($content) {
76+
public function writeContentToFile($content) {
3577
3678
$userFolder = $this->storage->getUserFolder('myUser');
3779
@@ -54,35 +96,32 @@ All methods return a Folder object on which files and folders can be accessed, o
5496
}
5597
}
5698
99+
57100
Reading from a file
58101
-------------------
59102

60103
Files and folders can also be accessed by id, by calling the **getById** method on the folder.
61104

62105
.. code-block:: php
63106
64-
<?php
65-
namespace OCA\MyApp\Storage;
66-
67107
use OCP\Files\IRootFolder;
68108
69-
class AuthorStorage {
109+
class FileReadingExample {
70110
71-
/** @var IRootFolder */
72-
private $storage;
111+
private IRootFolder $storage;
73112
74113
public function __construct(IRootFolder $storage){
75114
$this->storage = $storage;
76115
}
77116
78-
public function getContent($id) {
117+
public function getFileContent($id) {
79118
80119
$userFolder = $this->storage->getUserFolder('myUser');
81120
82121
// check if file exists and read from it if possible
83122
try {
84123
$file = $userFolder->getById($id);
85-
if($file instanceof \OCP\Files\File) {
124+
if ($file instanceof \OCP\Files\File) {
86125
return $file->getContent();
87126
} else {
88127
throw new StorageException('Can not read from folder');
@@ -92,3 +131,36 @@ Files and folders can also be accessed by id, by calling the **getById** method
92131
}
93132
}
94133
}
134+
135+
136+
Direct storage access
137+
---------------------
138+
139+
While it should be generally avoided in favor of the higher level apis,
140+
sometimes an app needs to talk directly to the storage implementation of it's metadata cache.
141+
142+
You can get access to the underlying storage of a file or folder by calling ``getStorage`` on the node or first getting
143+
the mountpoint by calling ``getMountPoint`` and getting the storage from there.
144+
145+
Once you have the storage instance you can use the storage api from ``OCP\Files\Storage\IStorage``, note however that
146+
all paths used in the storage api are internal to the storage, the ``IMountPoint`` returned from ``getMountPoint`` provides
147+
methods for translating between absolute filesystem paths and internal storage paths.
148+
149+
If you need to query the cached metadata directory you can get the ``OCP\Files\Cache\ICache`` from the storage by calling ``getCache``.
150+
151+
Implementing a storage
152+
----------------------
153+
154+
The recommended way for implementing a storage backend is by sub-classing ``OC\Files\Storage\Common`` which provides
155+
fallback implementations for various methods, reducing the amount of work required to implement the full storage api.
156+
Note however that various of these fallback implementations are likely to be significantly less efficient than an
157+
implementation of the method optimized for the abilities of the storage backend.
158+
159+
Adding mounts to the filesystem
160+
-------------------------------
161+
162+
The recommended way of adding your own mounts to the filesystem from an app is implementing ``OCP\Files\Config\IMountProvider``
163+
and registering the provider using ``OCP\Files\Config\IMountProviderCollection::registerProvider``.
164+
165+
Once registered, your provider will be called every time the filesystem is being setup for a user and your mount provider
166+
can return a list of mounts to add for that user.

‎developer_manual/core/architecture/files.rst‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
========================
2+
Nextcloud filesystem API
3+
========================
4+
5+
High level overview
6+
-------------------
7+
8+
The Nextcloud filesystem is roughly based on the unix filesystem, consisting of multiple storages
9+
mounted at various locations.
10+
11+
.. code-block:: txt
12+
13+
┌──────────────────────────────────┐
14+
│Code wanting to use the filesystem│
15+
└─────────┬─────────────────────┬──┘
16+
│ │
17+
│ │
18+
┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
19+
╎Filesystem │ │ ╎
20+
╎layer │new │legacy ╎
21+
╎ │ │ ╎
22+
╎ ▼ ▼ ╎
23+
╎ ┌────────┐ Partly build on ┌─┴──────┐ ╎
24+
╎ │Node API├─────────────────►│View API│ ╎
25+
╎ └───────┬┘ └─┬──────┘ ╎
26+
╎ │ │ ╎
27+
└╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
28+
│ │
29+
┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
30+
╎Storage layer │ │ ╎
31+
╎ ├─────────────────────┤ ╎
32+
╎ │ │ ╎
33+
╎ ▼ ▼ ╎
34+
╎ ┌───────┐ ┌───────┐ ┌──────┐ ╎
35+
╎ │Storage│═══>│Scanner│═══>│Cache │ ╎
36+
╎ └───────┘ └───────┘ └──────┘ ╎
37+
╎ ╎
38+
╎ ╎
39+
└╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
40+
41+
Filesystem layer
42+
^^^^^^^^^^^^^^^^
43+
44+
Any code that wants to use the filesystem has two API options to use, the new ``Node`` api and the old ``View`` api.
45+
New code should preferably use the ``Node`` api as it allows building systems with less overhead than the old api.
46+
47+
Besides the filesystem apis, this layer also manages the available mounts, containing the logic to allow apps
48+
to setup their mounts and translating filesystem paths into a mountpoint + "internal" path.
49+
50+
Storage layer
51+
^^^^^^^^^^^^^
52+
53+
The storage implementation handles the details of communicating with the filesystem or remote storage api
54+
and provide a uniform api for Nextcloud to use the storage.
55+
56+
For each storage a metadata cache/index is maintained to allow reading metadata of the storage without having
57+
to talk to the (potentially) slow storage backend. The scanner is responsible for updating the cache with
58+
information from the storage backend.
59+
60+
Storage/Cache wrappers
61+
----------------------
62+
63+
To allow apps to customize the behavior of a storage without requiring the app to implement this for every
64+
possible storage backend, a ``Wrapper`` system is used.
65+
66+
A ``Wrapper`` encapsulates an inner storage and allows overwriting any method to customize its behavior, with
67+
all other methods being passed through to the inner storage.
68+
69+
Generally search storage wrapper has an equivalent cache wrapper encapsulating the cache of the inner storage
70+
to provide the same behavior modifications when reading metadata from the cache.
71+
72+
Wrappers can be layered to stack the behavior of the wrappers, for example the ``groupfolders`` app works by
73+
stacking a wrapper to provide access to a single folder on the root storage with a wrapper to limit the permissions
74+
of the storage.
75+
76+
.. code-block:: txt
77+
78+
┌───────────────┐ ┌────────────────────┐
79+
│PermissionsMask├─────►│CachePermissionsMask│ PermissionsMask applies a mask to the permissions of a storage
80+
└───────┬───────┘ └─────────┬──────────┘ to provide less-privilaged access to a storage
81+
│ │
82+
▼ ▼
83+
┌───────────────┐ ┌────────────────────┐
84+
│Jail ├─────►│CacheJail │ Jail restricts access to a file or folder of a storage providing
85+
└───────┬───────┘ └─────────┬──────────┘ a limited view into the storage (think unix chroot or bind mount)
86+
│ │
87+
▼ ▼
88+
┌───────────────┐ ┌────────────────────┐
89+
│Base Storage ├─────►│Base Cache │
90+
└───────────────┘ └────────────────────┘
91+
92+
93+
Code Map
94+
--------
95+
96+
Approximate overview of the significant filesystem code
97+
98+
AppData
99+
^^^^^^^
100+
101+
High level api for accessing "appdata" folders, based on the ``Node`` API
102+
103+
Cache
104+
^^^^^
105+
106+
- ``Cache`` implementation
107+
- Cache wrappers
108+
- Scanner and cache update logic
109+
- Search infrastructure
110+
111+
Mount
112+
^^^^^
113+
114+
Mountpoint management and setup
115+
116+
Node
117+
^^^^
118+
119+
``Node`` filesystem api implementation
120+
121+
ObjectStorage
122+
^^^^^^^^^^^^^
123+
124+
Implementation of the various supported object store storage backends
125+
126+
SimpleFS
127+
^^^^^^^^
128+
129+
Simplified version of the Node api, for providing a more limited api for some filesystem bits
130+
131+
Storage
132+
^^^^^^^
133+
134+
Implementation of various storage backends and wrappers
135+
136+
Streams
137+
^^^^^^^
138+
139+
Various low-level php stream wrapper used in storage implementations
140+
141+
Type
142+
^^^^
143+
144+
Mimetype management and detection
145+
146+
View.php
147+
^^^^^^^^
148+
149+
Legacy View api

0 commit comments

Comments
 (0)