Fix DAV stat cache to properly cache 404 #26324

PVince81 · 2016-10-10T10:32:52Z

Description

404 errors were not properly cached due to catching the wrong
exception. Now catching ClientHttpException and checking the error
code. In case of 404, adjust the stat cache accordingly.

Since now the local propfind() call throws NotFound, there is no need
to catch ClientHttpException everywhere any more but need to catch
NotFound exception instead and act accordingly.

Related Issue

Fixes #26323

Motivation and Context

Improve performance when creating new file/folders which usually first do a PROPFIND on the remote server to check for file/folder existence. Not caching these 404 would cause at least 6 additional PROPFIND requests to get the status. (this is because we use a lot of file_exists call everywhere...)

How Has This Been Tested?

See issue steps.

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

Backports

TODO: evaluate to which versions to backport this

@DeepDiver1975 @butonic @jvillafanez

mention-bot · 2016-10-10T10:32:54Z

@PVince81, thanks for your PR! By analyzing the history of the files in this pull request, we identified @DeepDiver1975, @icewind1991 and @LukasReschke to be potential reviewers.

PVince81 · 2016-10-10T10:35:23Z

also:

do proper regression testing, I'm not sure this is really covered by CI ...

PVince81 · 2016-10-10T10:38:11Z

Okay, the storage impl is covered by the DAV ext storage tests at least. The bug itself wouldn't trigger any failure since it was a caching issue.

DeepDiver1975 · 2016-10-11T07:32:05Z

21:49:28 There was 1 failure:
21:49:28 
21:49:28 1) OCA\Files_Sharing\Tests\CacheTest::testGetFolderContentsInRoot
21:49:28 Failed asserting that 10 matches expected 3.
21:49:28 
21:49:28 /var/lib/jenkins/workspace/owncloud-core_core_PR-26324-CAYE3ELZQFIMHVKICLMMAFO6AP4X6ZQLHJBL3TNLCB4KDPW5UN2A/apps/files_sharing/tests/CacheTest.php:444
21:49:28 /var/lib/jenkins/workspace/owncloud-core_core_PR-26324-CAYE3ELZQFIMHVKICLMMAFO6AP4X6ZQLHJBL3TNLCB4KDPW5UN2A/apps/files_sharing/tests/CacheTest.php:349

jvillafanez · 2016-10-18T07:34:13Z

lib/private/Files/Storage/DAV.php

-				$this->statCache->clear($path . '/');
-				$this->statCache->set($path, false);
-				throw $e;
+			} catch (ClientHttpException $e) {


Is this change intentional? I find strange that we're catching "NotFound" exceptions and removing "ClientHttpException" ones except here 😕

It's a bit messy...

Basically Sabre's propfind itself only ever throws ClientHttpException or ClientException, never NotFound (that one belongs to sabre server, not sabre client btw).

But here we do want to throw a known exception for the consumers of our own $this->propfind, so we throw NotFound and they all catch it.

Maybe it would be cleaner to simply rethrow ClientHttpException as is to avoid confusion ?

I'd rather keep consistency on the client side: if client expects a "NotFound" exception then we throw that exception.

The client doesn't expected NotFound exception, only the consumers of $this->propFind do, and they're all within this single class.

Argh, looks like I can't easily replace with only ClientHttpException, because when a 404 propfind was cached, I'd need to create a virtual ClientHttpException to simulate failure here: https://github.com/owncloud/core/pull/26324/files#diff-eaf59cbf225a9e583d7cda98942ed6abR253

That's the reason why I picked this exception. Not sure if there is a better solution.

If there is no easy solution, let's go with this. Just write a comment to remind why we've chosen this decision.

Sounds good, will do

Now the tricky part: create a virtual ClientHttpException considering that it takes a ResponseInterface in its constructor... I considered either creating a class that extends ClientHttpException or one that implements ResponseInterface but it all goes too far, too many methods.

I'll see if simply returning false from propfind and handling this outside would be better. (outside being inside the class as propfind is private)

PVince81 · 2016-10-20T14:27:12Z

Oh, PHP 7.1 specific failure... fancy!

PVince81 · 2016-10-20T15:47:41Z

hmm, tests pass for me in PHP 7.0.12... will need to setup PHP 7.1 then

PVince81 · 2016-10-21T11:08:22Z

I've setup 7.1.0RC4 with phpenv (yay, it worked!) but the tests do pass.
On Jenkins it's 7.1.0RC1, not sure if related.
Let's restart the build and see...

If it fails again, then we can try upgrading Jenkins to 7.1.0RC4.
And if it still fails, then it's possibly due to test order...

PVince81 · 2016-10-21T11:09:13Z

This is the failure on Jenkins:

15:10:51 1) OCA\Files_Sharing\Tests\CacheTest::testGetFolderContentsInRoot
15:10:51 Failed asserting that 10 matches expected 3.
15:10:51 
15:10:51 /var/lib/jenkins/workspace/owncloud-core_core_PR-26324-CAYE3ELZQFIMHVKICLMMAFO6AP4X6ZQLHJBL3TNLCB4KDPW5UN2A/apps/files_sharing/tests/CacheTest.php:444
15:10:51 /var/lib/jenkins/workspace/owncloud-core_core_PR-26324-CAYE3ELZQFIMHVKICLMMAFO6AP4X6ZQLHJBL3TNLCB4KDPW5UN2A/apps/files_sharing/tests/CacheTest.php:349
15:10:51

PVince81 · 2016-10-21T14:20:58Z

Rebased, let's hope the test magically passes now...

PVince81 · 2016-10-24T14:31:35Z

Even with PHP 7.1.0RC1 the problem doesn't appear on my local env.

So it seems that all I can do now is guessing and trial and error.

PVince81 · 2016-10-24T14:49:50Z

The failure itself seems completely unrelated to this change.
I could imagine that the failure is rather related to another PR that was merged into master related to etag propagation. But then it should all also fail on master then.

PVince81 · 2016-10-24T15:06:20Z

Jenkins PR: #26466

Let's see if it also fails if restarted in that separate test env...

PVince81 · 2016-10-25T09:33:59Z

Ha, the Jenkins PR healed it...

PVince81 · 2016-10-25T09:35:02Z

@jvillafanez I decided to use the return false approach for the private propfind method when an entry was not found. It seems to work fine.

Please rereview.

PVince81 · 2016-10-28T08:24:33Z

@jvillafanez please re-review

jvillafanez · 2016-11-07T08:41:39Z

lib/private/Files/Storage/DAV.php

 	 * @return array propfind response
 	 *
-	 * @throws NotFound
+	 * @throws ClientHttpException


This doc should be updated with the conditions under it returns "false"

jvillafanez · 2016-11-07T08:43:20Z

I don't see anything strange so 👍

PVince81 · 2016-11-07T09:08:08Z

Adjusted comment, squashed and rebased.

PVince81 · 2016-11-07T16:22:09Z

Grrrr, that unrelated failure again:

10:13:15 1) OCA\Files_Sharing\Tests\CacheTest::testGetFolderContentsInRoot
10:13:15 Failed asserting that 10 matches expected 3.
10:13:15 
10:13:15 /var/lib/jenkins/workspace/owncloud-core_core_PR-26324-CAYE3ELZQFIMHVKICLMMAFO6AP4X6ZQLHJBL3TNLCB4KDPW5UN2A/apps/files_sharing/tests/CacheTest.php:444
10:13:15 /var/lib/jenkins/workspace/owncloud-core_core_PR-26324-CAYE3ELZQFIMHVKICLMMAFO6AP4X6ZQLHJBL3TNLCB4KDPW5UN2A/apps/files_sharing/tests/CacheTest.php:349
10:13:15

Jenkins PR for the rescue: #26564

404 errors were not properly cached due to catching the wrong exception. Now catching ClientHttpException and checking the error code. In case of 404, adjust the stat cache accordingly.

PVince81 · 2016-11-08T09:00:50Z

Tests passed here #26574 and travis failure unrelated.

PVince81 · 2016-12-07T11:45:54Z

stable9.1: #26791

PVince81 · 2016-12-07T11:50:31Z

stable9: #26792

lock · 2019-08-04T06:02:29Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

PVince81 added 3 - To Review backport-request labels Oct 10, 2016

PVince81 added this to the 9.2 milestone Oct 10, 2016

PVince81 mentioned this pull request Oct 10, 2016

DAV client / fed share client doesn't properly stat cache 404 #26323

Closed

jvillafanez reviewed Oct 18, 2016

View reviewed changes

PVince81 force-pushed the fed-statcache-404 branch from 651f3e3 to af14eed Compare October 20, 2016 13:05

PVince81 force-pushed the fed-statcache-404 branch from af14eed to 19f38c2 Compare October 21, 2016 14:20

PVince81 mentioned this pull request Oct 24, 2016

[jenkins] Fix DAV stat cache to properly cache 404 #26466

Closed

jvillafanez reviewed Nov 7, 2016

View reviewed changes

PVince81 force-pushed the fed-statcache-404 branch from 19f38c2 to 0ab699f Compare November 7, 2016 09:08

PVince81 added 4 - To release and removed 3 - To Review labels Nov 7, 2016

PVince81 mentioned this pull request Nov 7, 2016

[jenkins] Fix DAV stat cache to properly cache 404 #26564

Closed

Fix DAV stat cache to properly cache 404

48093ef

404 errors were not properly cached due to catching the wrong exception. Now catching ClientHttpException and checking the error code. In case of 404, adjust the stat cache accordingly.

PVince81 force-pushed the fed-statcache-404 branch from 0ab699f to 48093ef Compare November 7, 2016 17:19

This was referenced Nov 7, 2016

[jenkins] Fix DAV stat cache to properly cache 404 #26573

Closed

[jenkins] Fix DAV stat cache to properly cache 404 #26574

Closed

PVince81 merged commit 0a6fc26 into master Nov 8, 2016

PVince81 deleted the fed-statcache-404 branch November 8, 2016 09:01

PVince81 mentioned this pull request Dec 7, 2016

[stable9.1] Fix DAV stat cache to properly cache 404 #26791

Merged

PVince81 mentioned this pull request Dec 7, 2016

[stable9] Fix DAV stat cache to properly cache 404 #26792

Merged

1 task

PVince81 removed the backport-request label Dec 7, 2016

rullzer mentioned this pull request Dec 19, 2016

[downstream] Fix DAV stat cache to properly cache 404 nextcloud/server#2744

Merged

lock bot locked as resolved and limited conversation to collaborators Aug 4, 2019

Fix DAV stat cache to properly cache 404 #26324

Fix DAV stat cache to properly cache 404 #26324

Uh oh!

Conversation

PVince81 commented Oct 10, 2016

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Backports

Uh oh!

mention-bot commented Oct 10, 2016

Uh oh!

PVince81 commented Oct 10, 2016

Uh oh!

PVince81 commented Oct 10, 2016

Uh oh!

DeepDiver1975 commented Oct 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PVince81 commented Oct 20, 2016

Uh oh!

PVince81 commented Oct 20, 2016

Uh oh!

PVince81 commented Oct 21, 2016

Uh oh!

PVince81 commented Oct 21, 2016

Uh oh!

PVince81 commented Oct 21, 2016

Uh oh!

PVince81 commented Oct 24, 2016

Uh oh!

PVince81 commented Oct 24, 2016

Uh oh!

PVince81 commented Oct 24, 2016

Uh oh!

PVince81 commented Oct 25, 2016

Uh oh!

PVince81 commented Oct 25, 2016

Uh oh!

PVince81 commented Oct 28, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jvillafanez commented Nov 7, 2016

Uh oh!

PVince81 commented Nov 7, 2016

Uh oh!

PVince81 commented Nov 7, 2016

Uh oh!

PVince81 commented Nov 8, 2016

Uh oh!

PVince81 commented Dec 7, 2016

Uh oh!

PVince81 commented Dec 7, 2016

Uh oh!

lock bot commented Aug 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects