Huge download: File gets completely read even after download is cancelled

[LukeLR]

Steps to reproduce

1. Download a huge file using the webinterface
2. Cancel that download
3. Check Server IO on that file

Expected behaviour

After the user cancels the download of that file, the server should stop reading the contents of that file.

Actual behaviour

The server keeps reading the contents of that file until it's completely read.

Server configuration

Operating system:
Arch Linux, 4.14.15-1-ARCH
Web server:

Server version: Apache/2.4.29 (Unix)
Server built:   Oct 21 2017 12:45:02

Database:

mysql  Ver 15.1 Distrib 10.1.30-MariaDB, for Linux (x86_64) using readline 5.1

PHP version:

PHP 7.2.2 (cli) (built: Jan 30 2018 19:18:38) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.2, Copyright (c) 1999-2018, by Zend Technologies

Nextcloud version: (see Nextcloud admin page)
13.0.0 RC 4

Updated from an older Nextcloud/ownCloud or fresh install:
Fresh setup on 12.x

Where did you install Nextcloud from:
Official updater

Signing status:

Signing status

``` Login as admin user into your Nextcloud and access http://example.com/index.php/settings/integrity/failed paste the results here. ``` No errors have been found. ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your Nextcloud installation folder ``` ``` Enabled: - activity: 2.6.1 - bruteforcesettings: 1.0.3 - calendar: 1.5.7 - comments: 1.3.0 - contacts: 2.0.1 - dav: 1.4.6 - federatedfilesharing: 1.3.1 - federation: 1.3.0 - files: 1.8.0 - files_external: 1.4.1 - files_pdfviewer: 1.2.0 - files_sharing: 1.5.0 - files_texteditor: 2.5.1 - files_trashbin: 1.3.0 - files_versions: 1.6.0 - files_videoplayer: 1.2.0 - firstrunwizard: 2.2.1 - gallery: 18.0.0 - logreader: 2.0.0 - lookup_server_connector: 1.1.0 - metadata: 0.6.0 - nextcloud_announcements: 1.2.0 - notifications: 2.1.2 - oauth2: 1.1.0 - password_policy: 1.3.0 - previewgenerator: 1.0.9 - provisioning_api: 1.3.0 - serverinfo: 1.3.0 - sharebymail: 1.3.0 - survey_client: 1.1.0 - systemtags: 1.3.0 - theming: 1.4.1 - twofactor_backupcodes: 1.2.3 - updatenotification: 1.3.0 - workflowengine: 1.3.0 Disabled: - admin_audit - encryption - user_external - user_ldap ```

Nextcloud configuration:

Config report

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "192.168.2.101"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "http:\/\/192.168.2.101\/nextcloud",
        "dbtype": "mysql",
        "version": "13.0.0.13",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "apps_paths": [
            {
                "path": "***REMOVED SENSITIVE VALUE***\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "***REMOVED SENSITIVE VALUE***\/apps2",
                "url": "\/apps2",
                "writable": true
            }
        ],
        "loglevel": 2,
        "maintenance": false,
        "updater.release.channel": "beta",
        "theme": ""
    }
}

Are you using external storage, if yes which one: local

Are you using encryption: no

Are you using an external user-backend, if yes which one: None

Client configuration

Browser:
Mozilla Firefox 58.0.1

Operating system:
Arch Linux 4.14.15-1-ARCH

Logs

Web server error log

Web server error log

No relevant log entries

Nextcloud log (data/nextcloud.log)

Nextcloud log

No relevant log entries

Browser log

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

Request headers

Host: 192.168.2.101
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Cookie: ***REMOVED SENSITIVE VALUE***
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Response Headers

Date: Fri, 02 Feb 2018 19:43:03 GMT
Server: Apache
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Powered-By: PHP/7.2.2
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Security-Policy: default-src 'none';
X-Frame-Options: SAMEORIGIN
Set-Cookie: ocDownloadStarted=***REMOVED SENSITIVE VALUE***; expires=Fri, 02-Feb-2018 19:43:24 GMT; Max-Age=20; path=/
Last-Modified: Sat, 07 Oct 2017 15:30:29 GMT
ETag: ***REMOVED SENSITIVE VALUE***
Content-Length: 26874923280
Content-Disposition: attachment; filename*=UTF-8''IMG_2328.MOV; filename="IMG_2328.MOV"
OC-ETag: ***REMOVED SENSITIVE VALUE***
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Robots-Tag: none
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: video/quicktime

[MorrisJobke]

@icewind1991 @nickvergessen Any ideas?

[nickvergessen]

not really, but I guess we run a transaction/locking etc. so we can not simply quit out of the request. and the cancel command is not continued.

[LukeLR]

Thanks a lot for taking this serious :) Not all open source projects are open to improvement requests. Keep up the great work!

[pupboss]

@kesselb Hi kesselb, as you replied in my previous issue #22334 , this weekend I did some testing.

Definitely video playback is using streaming because I saw clearly Range header and 206 status code in HTTP request, however there was still a high IO load on server side. The detailed status changing will be given below:

• For NextCloud video playback

  1. At the beginning php-fpm takes all disk reading bandwidth, say 120MB/s
  2. After it is loaded, there is no reading load even I drag the progress bar, there is write load instead (20-30 MB/s)

• For Nginx static file hosting

  1. There are only reading loads no more than 30MB/s anytime

Meanwhile, I noticed that web video player was using WebDAV protocol upon file request, I checked https://github.com/nextcloud/server/blob/master/remote.php in master branch, noticed there is a line require_once $file;, is it possible that file gets completely read caused by this line of code? Would like to know why there is a require_once $file; at the end of this function.

[berrnd]

I also faced this issue.

On my Nextcloud server I have "not so much" space, so bigger files I want share I put into Nextcloud via "Local external storage" which points to a mounted NFS share from another server.

After a download of such a shared file gets canceled, the file continues to be transfered from the NFS share - so this also produces a lot of network overhead for canceled downloads...

Found that this problem was also already reported in #15055.