Skip to content

Conversation

@artonge
Copy link
Contributor

@artonge artonge commented Jul 27, 2021

Folders contained in S3 bucket's are displayed with a size of "Pending".

The following command can get the real size to be displayed, but only for one refresh: php occ files:scan --path "<S3 bucket path>"

Rational (I might be wrong):

  1. When running the stat method on the S3 folder, the mtime is always set to 'now'.
  2. This leads to the hasUpdated method from Files/Storage/Common.php to always returns true,
  3. And to the needsUpdate method in Files/View.php to always returns true too.
  4. So the cache gets updated by Files/Cache/Scanner.php every time the folder is acceceed. But it gets updated with a size of -1 because the size for S3 folders is not getting computed and default to -1

Need guidance on how and where we should address this issue :).

@artonge artonge force-pushed the investigate/pending_size_for_s3_buckets_master branch 4 times, most recently from 1c04f22 to e05aa2c Compare July 27, 2021 09:24
@CarlSchwan
Copy link
Member

I investigated a similar issue with using Group Folders with the S3 storage with Vincent last week:

  • OC\File\Cache\Updater: the update function is never called for the folder itself, only for the preview. It's correctly called for folders outside the group folder
  • A S3ObjectStorage uses a NoopScanner, but the GroupFolderStorage uses a normal Scanner. This will probably cause troubles when the Updater::update method is called. Maybe it's the reason why Update is not called but I doubt it.

@PVince81
Copy link
Member

ref: nextcloud/groupfolders#1630

but probably not related because the latter is for group folders and the problem with S3 primary happens also without

@artonge artonge force-pushed the investigate/pending_size_for_s3_buckets_master branch 2 times, most recently from aea7a2e to bdd1a35 Compare August 19, 2021 10:28
@artonge artonge changed the title Investigate pending size for s3 buckets Fix folder size contained in S3 buckets Aug 19, 2021
@artonge artonge requested a review from juliusknorr August 19, 2021 10:31
@artonge artonge added this to the Nextcloud 23 milestone Aug 19, 2021
@artonge
Copy link
Contributor Author

artonge commented Aug 19, 2021

/backport to stable22

@artonge
Copy link
Contributor Author

artonge commented Aug 19, 2021

/backport to stable21

If 'filesystem_check_changes' was set to never, the cached size was alway set to -1 (Pending) on every access

Signed-off-by: Louis Chemineau <[email protected]>
@artonge artonge force-pushed the investigate/pending_size_for_s3_buckets_master branch from bdd1a35 to 735fd94 Compare August 19, 2021 13:03
Copy link
Member

@PVince81 PVince81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, since the solution covers all the interesting cases let's move forward 👍

@juliusknorr juliusknorr merged commit 041fbda into master Aug 20, 2021
@juliusknorr juliusknorr deleted the investigate/pending_size_for_s3_buckets_master branch August 20, 2021 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants