Iterated mega file upload - invented wheel wanted

[john_ramsden]

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

I'm aware that file upload limits can be increased by config
changes; but I'd rather minimize the need for changes like
this, both for admin reasons and to avoid possible (?) memory problems
or even bugs.

In principle there's a simple solution whereby the user runs
an initial page, in which they input the file name, and the
Submit button then runs JavaScript in the page.

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Anyway, the question I have is whether anyone has written
anything similar, ideally with a progress bar?

The only problem I anticipate is possible delays, and even
timeouts, while the code (client or server) positions to
the next offset to read or write (respectively) the next
chunk of a very large, e.g. 50 GByte, file.

Any suggestions gratefully received.

(It's a shame Apache doesn't seem to have an embedded FTP
server in some way. I guess FTP would be the obvious choice,
but the client doesn't wish to have a separate FTP server
running.)


Cheers

John R Ramsden (jramsden at glassouse.com)

 

[A. Sinan Unur]

(e-mail address removed) wrote in @z14g2000cwz.googlegroups.com:

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Do you have a Perl question?

Sinan

 

[john_ramsden]

(e-mail address removed) wrote in @z14g2000cwz.googlegroups.com:

I have to prepare an Apache/PHP based (or perl if easier)
[...]

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Do you have a Perl question?

Read the first line, numbnuts: "(or perl if easier)"

 

[Anno Siegel]

Read the first line, numbnuts: "(or perl if easier)"

So long then.

Anno

 

[Richard Gration]

(It's a shame Apache doesn't seem to have an embedded FTP server in some
way.

Apache 2.x does, I think. At least the MPM architecture allows for it, I
don't know if one has been written. As has been pointed out, FTP
belongs to the age of a nicer internet, but httpd2 will let you do it if
you really want to ...

R

 

[J. Gleixner]

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes. [...]
Anyway, the question I have is whether anyone has written
anything similar, ideally with a progress bar? [...]
Any suggestions gratefully received.

Out of the 3 newsgroups you chose, all 3 are off topic. This isn't a
"programming" question.

The wheel is to build a Web page, in HTML, and put it on your Web site
and let the browser do it for you. That's it, problem solved.

Just have a link to the file and the client clicks on the link or does a
right click...Save Link as. That's it. You could use PHP/CGI to build
the list of files, but there's probably no need to call a script to
serve the file, just use HTTP. You don't need to do anything to Apache
or your PHP installation.

Forget about a progress bar. Most (All?) browsers create their own
download manager and they likely have their own progress display. If you
really want to make your own, then an Applet is your best bet.

 

[Mr. Wilson]

[idiotic non-answer snipped]

Didn't watch enough Sesame Street to learn the difference between up and
down, eh?

Heh, what a maroon.

--Mr. Wilson

 

[Csaba2000]

This is an interesting question, but I must be missing something:
If I understand you right, you're trying to get a large file from
the client (browser) side to your (the server) side. And you
want the user's browser (possibly with the aid of code you
provide) to cooperate in this endeavor so you can get past
the usual (including timeout) limitations.

But javascript (really the only thing available generically) on
the client side does not have access to the user's disk. So
I am curious to know what I am not understanding here.
If browsers are not involved, then I would turn to cURL
within command line PHP (on the client side) or a similar
variant and have the PHP on the Apache end reassemble
the file. Haven't done it, but it seems straightforward.
Probably would try with a command line version of cURL.

As for progress bar, on the client side, you can use COM
methods on recent Windows systems to put up an IE and
keep changing the IE page as you progress along (see the
php.net/com area for starter examples).

Csaba Gabor from Vienna

 

[juglesh]

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

maybe this
http://jupload.biz/

havent tried it for large files (working great so far with smallish images),
but the website says no limit...

 

[john_ramsden]

[idiotic non-answer snipped]

Didn't watch enough Sesame Street to learn the difference
between up and down, eh?

Heh, what a maroon.

--Mr. Wilson

My post was about copying files from a client to a server,
as should have been clear to anyone with a 3-digit IQ who
skimmed it in more than a second.

I bow to your superior knowledge of kids' TV. But Sesame
Street must use a different definition of "upload" to the
one in http://www.vivtek.com/rfc1867.html.