Skip to content

Large federated shares #28059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PVince81 merged 5 commits into from
Aug 7, 2017
Merged

Large federated shares #28059

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6add3d3
get rid of External Scanner
butonic Jun 1, 2017
5aaed29
remove support for non OCS endpoint to add remote shares
butonic Jun 1, 2017
6b73884
deprecate shareapi.php
butonic Jun 1, 2017
c138ee2
Bring back ajax/external.php, sadly
Jul 7, 2017
bb71b69
Remove scanAll from ajax/external
Jul 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 15 additions & 18 deletions apps/files_sharing/ajax/external.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,31 +122,28 @@
\OCP\JSON::error(['data' => ['message' => $l->t('Storage not valid')]]);
exit();
}
$result = $storage->file_exists('');
if ($result) {
try {
$storage->getScanner()->scanAll();
\OCP\JSON::success();
} catch (\OCP\Files\StorageInvalidException $e) {
\OCP\Util::writeLog(
'files_sharing',
'Invalid remote storage: ' . get_class($e) . ': ' . $e->getMessage(),
\OCP\Util::DEBUG
);
\OCP\JSON::error(['data' => ['message' => $l->t('Storage not valid')]]);
} catch (\Exception $e) {

try {
$result = $storage->file_exists('');
if (!$result) {
$externalManager->removeShare($mount->getMountPoint());
\OCP\Util::writeLog(
'files_sharing',
'Invalid remote storage: ' . get_class($e) . ': ' . $e->getMessage(),
'Couldn\'t add remote share',
\OCP\Util::DEBUG
);
\OCP\JSON::error(['data' => ['message' => $l->t('Couldn\'t add remote share')]]);
}
} else {
$externalManager->removeShare($mount->getMountPoint());
} catch (\OCP\Files\StorageInvalidException $e) {
\OCP\Util::writeLog(
'files_sharing',
'Couldn\'t add remote share',
'Invalid remote storage: ' . get_class($e) . ': ' . $e->getMessage(),
\OCP\Util::DEBUG
);
\OCP\JSON::error(['data' => ['message' => $l->t('Storage not valid')]]);
} catch (\Exception $e) {
\OCP\Util::writeLog(
'files_sharing',
'Invalid remote storage: ' . get_class($e) . ': ' . $e->getMessage(),
\OCP\Util::DEBUG
);
\OCP\JSON::error(['data' => ['message' => $l->t('Couldn\'t add remote share')]]);
Expand Down
11 changes: 10 additions & 1 deletion apps/files_sharing/ajax/shareinfo.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,17 @@
*
*/

/**
* @deprecated fetches all metadata, kept for compatibility with older instances
*/

OCP\JSON::checkAppEnabled('files_sharing');

OC::$server->getLogger()->warning(
Copy link
Contributor

@PVince81 PVince81 Jun 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if we can afford this... likely to flood logs. Switch to debug maybe ?

Also an admin might not know what hundreds of different remotes are connecting to this instance as this is controlled by the users.

Copy link
Member Author

@butonic butonic Aug 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intention is to annoy admins to get them to upgrade older versions. And if they have to annoy another admin I am fine with it. Saves everyone problems and time in the long run.

Copy link
Contributor

@PVince81 PVince81 Aug 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

following MS' footsteps with Win10 upgrades ? not sure...

Copy link

@bcutter bcutter Oct 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly vote for reverting those 4 lines (34-37). It´s annoying and - guess what - really helps NOTHING, especially when all the remote servers are hosted instances without admin access or even when they run Nextcloud.

'Deprecated api access from '.OC::$server->getRequest()->getRemoteAddress().
'. Ask remote to upgrade.', ['app' => 'files_sharing']
);

if (!isset($_GET['t'])) {
\OC_Response::setStatus(400); //400 Bad Request
exit;
Expand Down Expand Up @@ -101,4 +110,4 @@ function getChildInfo($dir, $view, $sharePermissions) {
$result['children'] = getChildInfo($rootInfo, $rootView, $sharePermissions);
}

OCP\JSON::success(['data' => $result]);
OCP\JSON::success(['data' => $result, 'message' => 'DEPRECATED API']);
127 changes: 0 additions & 127 deletions apps/files_sharing/lib/External/Scanner.php
View file
Open in desktop

This file was deleted.

112 changes: 19 additions & 93 deletions apps/files_sharing/lib/External/Storage.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,11 @@
use GuzzleHttp\Exception\ClientException;
use GuzzleHttp\Exception\ConnectException;
use OC\Files\Storage\DAV;
use OC\ForbiddenException;
use OCA\FederatedFileSharing\DiscoveryManager;
use OCA\Files_Sharing\ISharedStorage;
use OCP\AppFramework\Http;
use OCP\Files\NotFoundException;
use OCP\Files\StorageInvalidException;
use OCP\Files\StorageNotAvailableException;
use Sabre\DAV\Client;

class Storage extends DAV implements ISharedStorage {
/** @var string */
Expand Down Expand Up @@ -89,7 +87,9 @@ public function __construct($options) {
'host' => $host,
'root' => $root,
'user' => $options['token'],
'password' => (string)$options['password']
'password' => (string)$options['password'],
// Federated sharing always uses BASIC auth
'authType' => Client::AUTH_BASIC
]);
}

Expand Down Expand Up @@ -139,21 +139,6 @@ public function getCache($path = '', $storage = null) {
return $this->cache;
}

/**
* @param string $path
* @param \OC\Files\Storage\Storage $storage
* @return \OCA\Files_Sharing\External\Scanner
*/
public function getScanner($path = '', $storage = null) {
if (!$storage) {
$storage = $this;
}
if (!isset($this->scanner)) {
$this->scanner = new Scanner($storage);
}
return $this->scanner;
}

/**
* check if a file or folder has been updated since $time
*
Expand Down Expand Up @@ -207,29 +192,25 @@ public function test() {
public function checkStorageAvailability() {
// see if we can find out why the share is unavailable
try {
$this->getShareInfo();
} catch (NotFoundException $e) {
// a 404 can either mean that the share no longer exists or there is no ownCloud on the remote
if ($this->testRemote()) {
// valid ownCloud instance means that the public share no longer exists
// since this is permanent (re-sharing the file will create a new token)
// we remove the invalid storage
$this->manager->removeShare($this->mountPoint);
$this->manager->getMountManager()->removeMount($this->mountPoint);
throw new StorageInvalidException();
} else {
// ownCloud instance is gone, likely to be a temporary server configuration error
throw new StorageNotAvailableException();
if ( ! $this->propfind('') ) {
// a 404 can either mean that the share no longer exists or there is no ownCloud on the remote
if ($this->testRemote()) {
// valid ownCloud instance means that the public share no longer exists
// since this is permanent (re-sharing the file will create a new token)
// we remove the invalid storage
$this->manager->removeShare($this->mountPoint);
$this->manager->getMountManager()->removeMount($this->mountPoint);
throw new StorageInvalidException();
} else {
// ownCloud instance is gone, likely to be a temporary server configuration error
throw new StorageNotAvailableException();
}
}
} catch (ForbiddenException $e) {
} catch (StorageInvalidException $e) {
// auth error, remove share for now (provide a dialog in the future)
$this->manager->removeShare($this->mountPoint);
$this->manager->getMountManager()->removeMount($this->mountPoint);
throw new StorageInvalidException();
} catch (\GuzzleHttp\Exception\ConnectException $e) {
throw new StorageNotAvailableException();
} catch (\GuzzleHttp\Exception\RequestException $e) {
throw new StorageNotAvailableException();
throw $e;
} catch (\Exception $e) {
throw $e;
}
Expand Down Expand Up @@ -286,61 +267,6 @@ private function testRemoteUrl($url) {
return $returnValue;
}

/**
* Whether the remote is an ownCloud, used since some sharing features are not
* standardized. Let's use this to detect whether to use it.
*
* @return bool
*/
public function remoteIsOwnCloud() {
if(defined('PHPUNIT_RUN') || !$this->testRemoteUrl($this->getRemote() . '/status.php')) {
return false;
}
return true;
}

/**
* @return mixed
* @throws ForbiddenException
* @throws NotFoundException
* @throws \Exception
*/
public function getShareInfo() {
$remote = $this->getRemote();
$token = $this->getToken();
$password = $this->getPassword();

// If remote is not an ownCloud do not try to get any share info
if(!$this->remoteIsOwnCloud()) {
return ['status' => 'unsupported'];
}

$url = rtrim($remote, '/') . '/index.php/apps/files_sharing/shareinfo?t=' . $token;

// TODO: DI
$client = \OC::$server->getHTTPClientService()->newClient();
try {
$response = $client->post($url, [
'body' => ['password' => $password],
'timeout' => 10,
'connect_timeout' => 10,
]);
} catch (\GuzzleHttp\Exception\RequestException $e) {
if ($e->getCode() === Http::STATUS_UNAUTHORIZED || $e->getCode() === Http::STATUS_FORBIDDEN) {
throw new ForbiddenException();
}
if ($e->getCode() === Http::STATUS_NOT_FOUND) {
throw new NotFoundException();
}
// throw this to be on the safe side: the share will still be visible
// in the UI in case the failure is intermittent, and the user will
// be able to decide whether to remove it if it's really gone
throw new StorageNotAvailableException();
}

return json_decode($response->getBody(), true);
}

public function getOwner($path) {
list(, $remote) = explode('://', $this->remote, 2);
return $this->remoteUser . '@' . $remote;
Expand Down
Loading