Skip to content

Fix DAV stat cache to properly cache 404 #26324

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PVince81 merged 1 commit into from
Nov 8, 2016
Merged

Fix DAV stat cache to properly cache 404 #26324

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 44 additions & 43 deletions lib/private/Files/Storage/DAV.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@
use OCP\Files\StorageNotAvailableException;
use OCP\Util;
use Sabre\DAV\Client;
use Sabre\DAV\Exception\NotFound;
use Sabre\DAV\Xml\Property\ResourceType;
use Sabre\HTTP\ClientException;
use Sabre\HTTP\ClientHttpException;
Expand Down Expand Up @@ -207,7 +206,9 @@ public function opendir($path) {
[],
1
);
$id = md5('webdav' . $this->root . $path);
if ($response === false) {
return false;
}
$content = [];
$files = array_keys($response);
array_shift($files); //the first entry is the current directory
Expand All @@ -225,13 +226,6 @@ public function opendir($path) {
$content[] = $file;
}
return IteratorDirectory::wrap($content);
} catch (ClientHttpException $e) {
if ($e->getHttpStatus() === 404) {
$this->statCache->clear($path . '/');
$this->statCache->set($path, false);
return false;
}
$this->convertException($e, $path);
} catch (\Exception $e) {
$this->convertException($e, $path);
}
Expand All @@ -246,17 +240,13 @@ public function opendir($path) {
*
* @param string $path path to propfind
*
* @return array propfind response
* @return array|boolean propfind response or false if the entry was not found
*
* @throws NotFound
* @throws ClientHttpException
Copy link
Member

@jvillafanez jvillafanez Nov 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc should be updated with the conditions under it returns "false"

*/
protected function propfind($path) {
$path = $this->cleanPath($path);
$cachedResponse = $this->statCache->get($path);
if ($cachedResponse === false) {
// we know it didn't exist
throw new NotFound();
}
// we either don't know it, or we know it exists but need more details
if (is_null($cachedResponse) || $cachedResponse === true) {
$this->init();
Expand All @@ -274,11 +264,15 @@ protected function propfind($path) {
]
);
$this->statCache->set($path, $response);
} catch (NotFound $e) {
// remember that this path did not exist
$this->statCache->clear($path . '/');
$this->statCache->set($path, false);
throw $e;
} catch (ClientHttpException $e) {
Copy link
Member

@jvillafanez jvillafanez Oct 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change intentional? I find strange that we're catching "NotFound" exceptions and removing "ClientHttpException" ones except here 😕

Copy link
Contributor Author

@PVince81 PVince81 Oct 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit messy...

Basically Sabre's propfind itself only ever throws ClientHttpException or ClientException, never NotFound (that one belongs to sabre server, not sabre client btw).

But here we do want to throw a known exception for the consumers of our own $this->propfind, so we throw NotFound and they all catch it.

Maybe it would be cleaner to simply rethrow ClientHttpException as is to avoid confusion ?

Copy link
Member

@jvillafanez jvillafanez Oct 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather keep consistency on the client side: if client expects a "NotFound" exception then we throw that exception.

Copy link
Contributor Author

@PVince81 PVince81 Oct 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The client doesn't expected NotFound exception, only the consumers of $this->propFind do, and they're all within this single class.

Copy link
Contributor Author

@PVince81 PVince81 Oct 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Argh, looks like I can't easily replace with only ClientHttpException, because when a 404 propfind was cached, I'd need to create a virtual ClientHttpException to simulate failure here: https://github.com/owncloud/core/pull/26324/files#diff-eaf59cbf225a9e583d7cda98942ed6abR253

That's the reason why I picked this exception. Not sure if there is a better solution.

Copy link
Member

@jvillafanez jvillafanez Oct 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no easy solution, let's go with this. Just write a comment to remind why we've chosen this decision.

Copy link
Contributor Author

@PVince81 PVince81 Oct 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, will do

Copy link
Contributor Author

@PVince81 PVince81 Oct 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now the tricky part: create a virtual ClientHttpException considering that it takes a ResponseInterface in its constructor... I considered either creating a class that extends ClientHttpException or one that implements ResponseInterface but it all goes too far, too many methods.

I'll see if simply returning false from propfind and handling this outside would be better. (outside being inside the class as propfind is private)

if ($e->getHttpStatus() === 404) {
$this->statCache->clear($path . '/');
$this->statCache->set($path, false);
return false;
}
$this->convertException($e, $path);
} catch (\Exception $e) {
$this->convertException($e, $path);
}
} else {
$response = $cachedResponse;
Expand All @@ -290,17 +284,15 @@ protected function propfind($path) {
public function filetype($path) {
try {
$response = $this->propfind($path);
if ($response === false) {
return false;
}
$responseType = [];
if (isset($response["{DAV:}resourcetype"])) {
/** @var ResourceType[] $response */
$responseType = $response["{DAV:}resourcetype"]->getValue();
}
return (count($responseType) > 0 and $responseType[0] == "{DAV:}collection") ? 'dir' : 'file';
} catch (ClientHttpException $e) {
if ($e->getHttpStatus() === 404) {
return false;
}
$this->convertException($e, $path);
} catch (\Exception $e) {
$this->convertException($e, $path);
}
Expand All @@ -319,13 +311,7 @@ public function file_exists($path) {
return true;
}
// need to get from server
$this->propfind($path);
return true; //no 404 exception
} catch (ClientHttpException $e) {
if ($e->getHttpStatus() === 404) {
return false;
}
$this->convertException($e, $path);
return ($this->propfind($path) !== false);
} catch (\Exception $e) {
$this->convertException($e, $path);
}
Expand Down Expand Up @@ -431,6 +417,9 @@ public function free_space($path) {
try {
// TODO: cacheable ?
$response = $this->client->propfind($this->encodePath($path), ['{DAV:}quota-available-bytes']);
if ($response === false) {
return FileInfo::SPACE_UNKNOWN;
}
if (isset($response['{DAV:}quota-available-bytes'])) {
return (int)$response['{DAV:}quota-available-bytes'];
} else {
Expand All @@ -456,6 +445,9 @@ public function touch($path, $mtime = null) {
$this->client->proppatch($this->encodePath($path), ['{DAV:}lastmodified' => $mtime]);
// non-owncloud clients might not have accepted the property, need to recheck it
$response = $this->client->propfind($this->encodePath($path), ['{DAV:}getlastmodified'], 0);
if ($response === false) {
return false;
}
if (isset($response['{DAV:}getlastmodified'])) {
$remoteMtime = strtotime($response['{DAV:}getlastmodified']);
if ($remoteMtime !== $mtime) {
Expand Down Expand Up @@ -578,15 +570,13 @@ public function copy($path1, $path2) {
public function stat($path) {
try {
$response = $this->propfind($path);
if ($response === false) {
return [];
}
return [
'mtime' => strtotime($response['{DAV:}getlastmodified']),
'size' => (int)isset($response['{DAV:}getcontentlength']) ? $response['{DAV:}getcontentlength'] : 0,
];
} catch (ClientHttpException $e) {
if ($e->getHttpStatus() === 404) {
return [];
}
$this->convertException($e, $path);
} catch (\Exception $e) {
$this->convertException($e, $path);
}
Expand All @@ -597,6 +587,9 @@ public function stat($path) {
public function getMimeType($path) {
try {
$response = $this->propfind($path);
if ($response === false) {
return false;
}
$responseType = [];
if (isset($response["{DAV:}resourcetype"])) {
/** @var ResourceType[] $response */
Expand All @@ -610,11 +603,6 @@ public function getMimeType($path) {
} else {
return false;
}
} catch (ClientHttpException $e) {
if ($e->getHttpStatus() === 404) {
return false;
}
$this->convertException($e, $path);
} catch (\Exception $e) {
$this->convertException($e, $path);
}
Expand Down Expand Up @@ -705,6 +693,9 @@ public function getPermissions($path) {
$this->init();
$path = $this->cleanPath($path);
$response = $this->propfind($path);
if ($response === false) {
return 0;
}
if (isset($response['{http://owncloud.org/ns}permissions'])) {
return $this->parsePermissions($response['{http://owncloud.org/ns}permissions']);
} else if ($this->is_dir($path)) {
Expand All @@ -721,6 +712,9 @@ public function getETag($path) {
$this->init();
$path = $this->cleanPath($path);
$response = $this->propfind($path);
if ($response === false) {
return null;
}
if (isset($response['{DAV:}getetag'])) {
return trim($response['{DAV:}getetag'], '"');
}
Expand Down Expand Up @@ -764,6 +758,13 @@ public function hasUpdated($path, $time) {
// force refresh for $path
$this->statCache->remove($path);
$response = $this->propfind($path);
if ($response === false) {
if ($path === '') {
// if root is gone it means the storage is not available
throw new StorageNotAvailableException(get_class($e) . ': ' . $e->getMessage());
}
return false;
}
if (isset($response['{DAV:}getetag'])) {
$cachedData = $this->getCache()->get($path);
$etag = null;
Expand All @@ -786,7 +787,7 @@ public function hasUpdated($path, $time) {
return $remoteMtime > $time;
}
} catch (ClientHttpException $e) {
if ($e->getHttpStatus() === 404 || $e->getHttpStatus() === 405) {
if ($e->getHttpStatus() === 405) {
if ($path === '') {
// if root is gone it means the storage is not available
throw new StorageNotAvailableException(get_class($e) . ': ' . $e->getMessage());
Expand Down