Large file uploads using Java

[heather.fraser]

I am well aware of several file upload servlets & libraries (including
the one from Apache/Jakarta).

However, how does one go about creating a web-based file upload
similar to Whalemail (www.whalemail.com) or many of the AJAX web2.0
sites I see where they show the status of the upload (in KB and %age)
as the upload progresses ?

Also I now find myself with the need to upload files as large as 500
MB over a Local Area Network - is the servlet going to be able to
handle this or will it try to cache the uploaded data in memory and
therefore run into memory limits on the server ?

Thank you for any advice,

Heather

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

I am well aware of several file upload servlets & libraries (including
the one from Apache/Jakarta).

However, how does one go about creating a web-based file upload
similar to Whalemail (www.whalemail.com) or many of the AJAX web2.0
sites I see where they show the status of the upload (in KB and %age)
as the upload progresses ?

Also I now find myself with the need to upload files as large as 500
MB over a Local Area Network - is the servlet going to be able to
handle this or will it try to cache the uploaded data in memory and
therefore run into memory limits on the server ?

500 MB in memory should be OK on todays HW.

But let us assume DVD sise instead of CD size.

I don't think Jakarta Common FileUpload can stream directly from
request to final destination.

But I think Com Oreilly Servlet can.

I have some code somewhere that can show uploaded bytes
being incremented in a separate window (a refreshing window - this
was made before AJAX became popular).

Arne

 

[heather.fraser]

Thank you for the input, Arne.

You're right actually - we WILL need to be able to upload DVD-sized
objects at some stage. Architecturally, one of the key considerations
has been whether we should use the web browser (and HTTP) for file-
upload/transfer; or FTP separately & then regularly ingest the files
on the server - but this introduces the problem of synchronizing the
uploaded file with the user & the adding of meta data. In the long
term, we believe all things are going the web-route and it is easier
for the end users.

We already use the Oreilly servlet actually so we'll see what happens
with massive gargantuan file uploads and whether the JVM & Tomcat
choke. However, how does one show & dynamically update the percentage
uploaded with this?

Thanks again,

Heather

 

[Roedy Green]

Also I now find myself with the need to upload files as large as 500
MB over a Local Area Network - is the servlet going to be able to
handle this or will it try to cache the uploaded data in memory and
therefore run into memory limits on the server ?

See the code for download at
http://mindprod.com/products1.html#FILETRANSFER

It uses use a read method that reads in chunks. You could attach your
progress bar to that to jump x ticks for every byte read in the chunk.
You have to change the read to a write, but the skeletons you need are
there.

see http://mindprod.com/jgloss/progress.html

You need something on the server side to decide when to accept files.

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

We already use the Oreilly servlet actually so we'll see what happens
with massive gargantuan file uploads and whether the JVM & Tomcat
choke. However, how does one show & dynamically update the percentage
uploaded with this?

If you read directly from network stream to file stream, then it should
not require much memory.

Just save percentages in session and have a refreshing window that
calls something that retrieves that value.

Arne

 

[Roedy Green]

See the code for download at
http://mindprod.com/products1.html#FILETRANSFER

for giant files, you need ether to be able to restart part way through
or you need to transfer in chunks, perhaps a meg each. Then combine
them when all done. You also want to compress them.

 

[heather.fraser]

See the code for download at
Thank you. I shall take a look.

for giant files, you need ether to be able to restart part way through

Yes, that's a good point. This is turning out to be not such a trivial
upload question after all.

or you need to transfer in chunks, perhaps a meg each. Then combine
them when all done. You also want to compress them.

Giant files tend to be binaries so I'm not sure the compression is
going to help that much, is it ?

Thanks for the help,

Heather.

 

[heather.fraser]

If you read directly from network stream to file stream, then it should
not require much memory.

Just save percentages in session and have a refreshing window that
calls something that retrieves that value.

Wow, that's a fantastic idea and so simple. Thank you so much, Arne,
I always thought it was so much more complicated than that

 

[Lew]

Giant files tend to be binaries so I'm not sure the compression is
going to help that much, is it ?

Binary files compress. Typical binaries, like visual images or program code,
display enough regularity for compression to be effective.

Encrypted or already-compressed files are another matter.