Web Based File Upload Chunking

[matart]

Steps to reproduce

Web based large file upload

Expected behaviour

File gets chunked and multiple ajax requests send the chunks to the server for reassembly

Actual behaviour

File is uploaded as a whole file. A single request is made

Server configuration

Operating system: RHEL 6

Web server: NGINX

Database: MySQL

PHP version: 5.5

ownCloud version: 6.0.3

Updated from an older ownCloud or fresh install:

List of activated apps: Files, Activity, Contacts, Pictures

The content of config/config.php:

I had to remove the identifying URLs so it seems a little empty.

<?php
$CONFIG = array (
  'instanceid' => 'oc8fdcdb7be6',
  'passwordsalt' => '',
  'datadirectory' => '/var/www/html/macdrop_assets',
  'dbtype' => 'mysql',
  'version' => '6.0.3.1',
  'dbname' => '',
  'dbhost' => '',
  'dbtableprefix' => '',
  'dbuser' => '',
  'dbpassword' => '',
  'installed' => true,
  'forcessl' => true,
  'theme' => 'macdrop',
  'proxy' => '',
  'loglevel' => '2',
  'maxZipInputSize' => 1073741824,
  'allowZipDownload' => true,
  'mail_domain' => '',
  'mail_smtphost' => '',
  'mail_smtpname' => 'macDrop',
  'maintenance' => false,
  'trusted_domains' => 
  array (
    0 => 'url1',
    1 => 'url2',
  ),
);

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: yes/no

Client configuration

Browser: Chrome

Operating system: Windows

Browser log

This is a 1.7GB file. One request taking 3.2 minutes.

If this feature does not exist I am willing to look into implementing it and making a pull request.

[karlitschek]

I think this feature doesn´t exist at the moment. So a pull request would be awesome. @butonic Haven´t you looked into that last year?

AFAIK there is already a discussion / feature request for chunked file uploading available at the bugtracker but i can't find it at the moment. Will see if i can dig this out tomorrow.

[matart]

That would be great. Thank you! Thank you,Mathew A. From: RealRancorSent: Monday, June 9, 2014 4:53 PMTo: owncloud/coreReply To: owncloud/coreCc: Mathew ArtemenkoSubject: Re: [core] Web Based File Upload Chunking (#8955)AFAIK there is already a discussion / feature request for chunked file uploading available at the bugtracker but i can't find it at the moment. Will see if i can dig this out tomorrow.

—Reply to this email directly or view it on GitHub.

[PVince81]

I believe I heard that some app, maybe this one http://apps.owncloud.com/content/show.php/Large+File+Upload?content=159291 was using HTML5 file chunking of some sort.

[tylla]

Yes that app was working until OC 8.0. Now it is defunct and the maintainer has some problems fixing it.
I believe too that this functionality should be implemented as a core functionality, so someone with enough knowledge should take a look at that app's code and merge it to the core. Unfortunately I'm not that person.:(

[PVince81]

Yes, would be a nice addition.

Note that there are plans to make the web UI use WebDAV first: #12353

This means it will become possible to use chunking too through the WebDAV endpoint.

[tylla]

As I found out, OC 8 uses jQuery-File-Upload and as I discovered here jQuery-File-Upload already supports chunked uploading. One only has to override the maxChunkSize variable with some sane value instead of the default "undefined", and after that the upload starts to send chunks with the specified size instead of whole files.
For testing I changed to
maxChunkSize: 1000000, // 1MB

One problem exists though. At the end of the upload there is an error message that asks whether I want to overwrite the same file with the same file. If it's gibberish what I say, there is a screenshot attached. There you can see that the already uploaded image is only the first chunk (there is some missing data from the image too) and asks me to overwrite it with the full file. But if I choose the newer or both files it just hangs after I click the Continue button. After this I can click the Cancel, but then I'm also only with the partial file.
BTW it throws a console error when I click the Continue button:
TypeError: data.formData.push is not a function
data.formData.push({name:'resolution', value:'autorename'}); //hack for ie8
file-upload.js?v=8ca31e8dc30c45c0bf2c0abdfb1746bd (line 155, col 42)

Hmm, this starts to look as a bug report... maybe I should file it as such.

[PVince81]

But also I believe that the ajax/upload.php endpoint does not support chunks, so it probably won't work.
Ideal would be to use the WebDAV endpoint instead.
Note that the WebDAV endpoint uses custom chunking (so does the sync client), so it might not work either with a "standard" solution.

Either way, it will need a bit of work to make this.

Still, any information you post here could be useful for that, so thanks 😄

[PVince81]

I guess it asks you for overwrite because for each chunk sent to upload.php, it saves it as a new file. Because the ajax/upload.php endpoint doesn't understand chunks. And since the file exists already, it shows you that dialog. (just guessing)

[PVince81]

PR here #26306