How to find uploaded data size from content-length

A

Asterbing

Considering a multipart/form-data form, is there a formula to calculate
the size of an uploaded file from within $ENV{'CONTENT_LENGTH'} which
represent the size of the entire data in STDIN (including all name/value
pairs, boundary between parts, content-disposition & type lines, some
blank lines) ?
 
X

xhoster

Asterbing said:
Considering a multipart/form-data form, is there a formula to calculate
the size of an uploaded file from within $ENV{'CONTENT_LENGTH'} which
represent the size of the entire data in STDIN (including all name/value
pairs, boundary between parts, content-disposition & type lines, some
blank lines) ?

No.

Xho
 
J

John Bokma

Asterbing said:
Considering a multipart/form-data form, is there a formula to calculate
the size of an uploaded file from within $ENV{'CONTENT_LENGTH'} which
represent the size of the entire data in STDIN (including all name/value
pairs, boundary between parts, content-disposition & type lines, some
blank lines) ?

Why not -s filename assuming you store it in a file system?
 
J

John Bokma

Asterbing said:
Because I would like to avoid too big files

Ok, so your question actually is: how do I avoid the uploading of big
files:

$CGI::pOST_MAX

If set to a non-negative integer, this variable puts a ceiling on the
size of POSTings, in bytes. If CGI.pm detects a POST that is greater than
the ceiling, it will immediately exit with an error message. This value
will affect both ordinary POSTs and multipart POSTs, meaning that it
limits the maximum size of file uploads as well. You should set this to a
reasonably high value, such as 1 megabyte.


(documentation of CGI.pm, Avoiding Denial of Service Attacks)
 
X

xhoster

John Bokma said:
Ok, so your question actually is: how do I avoid the uploading of big
files:

$CGI::pOST_MAX

If set to a non-negative integer, this variable puts a ceiling on the
size of POSTings, in bytes. If CGI.pm detects a POST that is greater than
the ceiling, it will immediately exit with an error message.

That should probaby be changed. It does not immediately exit with
an error message. It first reads (and discards) the entire contents
of the POST, which apparently the OP doesn't want to happen althought it
is kind of hard to figure out exactly what he wants. Also, it doesn't
exit at all. It causes param to return an empty param list, and it causes
cgi_error to return an error message--no exiting involved.

Xho
 
X

xhoster

Asterbing said:

Why isn't the moon made out of green cheese?

You pretty much covered why in the post. You can't figure out how the
sizes of the different parts of the post (which sum up to CONTENT_LENGTH)
are distributed until you read all but one of them. Since you can't
gaurantee that the part which contains the file upload is the last part,
you can't figure out how big that part is until you have read it.

I have no idea why you think you need to do this.....

Xho
 
A

Asterbing

I have no idea why you think you need to do this.....

Because when CONTENT_LENGTH has been informed by server itself after
receiving of entire POST, I can effectively read STDIN, extract uploaded
data and do the substraction.

But, when CONTENT_LENGTH comes from original request, how to proceed ?

Say, I wish to avoid any upload (I mean write on server, since data may
already be in STDIN) above 1MB, how to proceed ?
 
J

John Bokma

That should probaby be changed. It does not immediately exit with
an error message. It first reads (and discards) the entire contents
of the POST, which apparently the OP doesn't want to happen althought
it is kind of hard to figure out exactly what he wants. Also, it
doesn't exit at all. It causes param to return an empty param list,
and it causes cgi_error to return an error message--no exiting
involved.

You mean it reads until it detects that the max size has been reached?
Yes, that's what I expect, since one can keep streaming bytes, and only
when the max has been reached the script knows it was too much.

The second part: if the documentation is wrong, it must be fixed. Did you
submit this?
 
A

Asterbing

$CGI::pOST_MAX

If set to a non-negative integer, this variable puts a ceiling on the
size of POSTings, in bytes. If CGI.pm detects a POST that is greater than
the ceiling, it will immediately exit with an error message. This value
will affect both ordinary POSTs and multipart POSTs, meaning that it
limits the maximum size of file uploads as well. You should set this to a
reasonably high value, such as 1 megabyte.

Without CGI.pm !

However, reading what you posted from CGI.pm help, it seems that CGI.pm
doesn't really limit the file size precisely but rather the entire
POSTed data. So, how if I want to be free to choose the file limit not
as 1MB but as 5KB ?
 
A

Asterbing

That should probaby be changed. It does not immediately exit with
an error message. It first reads (and discards) the entire contents
of the POST, which apparently the OP doesn't want to happen althought it
is kind of hard to figure out exactly what he wants. Also, it doesn't
exit at all. It causes param to return an empty param list, and it causes
cgi_error to return an error message--no exiting involved.

Xho

OK, so it doesn't specifically limit the size of uploadable file (but
size of entire POST) and it doesn't avoid sending of entire POST data
from web browser to server's STDIN ?
 
A

Asterbing

You mean it reads until it detects that the max size has been reached?
Yes, that's what I expect, since one can keep streaming bytes, and only
when the max has been reached the script knows it was too much.

When CONTENT_LENGTH not given during request, server has to compute it
from real data received in STDIN : so, in this case how does CGI.pm is
able to stop receiving to STDIN precisely when $CGI::pOST_MAX is reached
?

Not sure it's able to do that. What do you think ?
 
J

John Bokma

Asterbing said:
Without CGI.pm !

Why? Look how "they" do it, and copy it.
However, reading what you posted from CGI.pm help, it seems that
CGI.pm doesn't really limit the file size precisely but rather the
entire POSTed data. So, how if I want to be free to choose the file
limit not as 1MB but as 5KB ?

Good question, set it to 50KB, and check the actual filesize.
 
D

Denver

It would make sense to me to send an error message, close the connection, and exit.
Otherwise, a DoS attack indeed keeps streaming bits to the server.

OK, so it doesn't specifically limit the size of uploadable file
(but size of entire POST)
and it doesn't avoid sending of entire POST data from web browser to server's STDIN ?

STDIN isn't the right term.
Apparently the incoming data continue filling the TCP receive buffer for that socket.
 
X

xhoster

John Bokma said:
You mean it reads until it detects that the max size has been reached?

No. It compares CONTENT_LENGTH to POST_MAX. If CONTENT_LENGTH is greater
than POST_MAX, then it reads the entire post, but doesn't do anything with
it (doesn't parse it, doesn't save it to disk, just reads and throws away
in chunks.) If CONTENT_LENGTH is less than POST_MAX, it reads the post
and parses it (saving parts of it to disk, if appropriate.)

Yes, that's what I expect, since one can keep streaming bytes, and only
when the max has been reached the script knows it was too much.

But AFAICT, it doesn't do it this way. It seems to trust CONTENT_LENGTH
absolutely, and if CONTENT_LENGTH is wrong it does nothing to detect this.
The second part: if the documentation is wrong, it must be fixed. Did you
submit this?

I haven't yet. I'm not sure how I would recommend changing it. The
read-but-discard thing may be an (only sometimes important) implementation
detail that does not need to be documented. The "immediate exit" is
clarified later on the documention, and I'm not sure how to clarify here
also without causing bloat.


Xho
 
X

xhoster

Asterbing said:
Without CGI.pm !

However, reading what you posted from CGI.pm help, it seems that CGI.pm
doesn't really limit the file size precisely but rather the entire
POSTed data. So, how if I want to be free to choose the file limit not
as 1MB but as 5KB ?

Do you really consider 5KB to be a denial of service attack?

Set one limit on the whole post of, say, 10MB, to prevent DoS attacks.
If the size is over 10MB, send a nasty message about DoS not being welcome.
If it is under 10MB, process it, save the upload to a temp file, and
check its size. If the size is more than 5KB, send a polite message
saying the file was a tad too big, and delete the temp file. Otherwise,
do whatever it is you want to do.

Xho
 
X

xhoster

Asterbing said:
Because when CONTENT_LENGTH has been informed by server itself after
receiving of entire POST, I can effectively read STDIN, extract uploaded
data and do the substraction.

Once you've extracted the data, why would you need to do the subtraction?
Once you've extracted the data, you already have the data, just ask the
data how big it is. I thought the whole point was to avoid extracting the
data in the first place.
But, when CONTENT_LENGTH comes from original request, how to proceed ?

Say, I wish to avoid any upload (I mean write on server, since data may
already be in STDIN) above 1MB, how to proceed ?

You have enough memory to load a 1 MB of non-file-upload form-data into
memory, but not enough disk to temporarily save 1 MB of file-upload data?
That just doesn't make sense. If you computer will break with 1MB of
posted data, don't allow that size of post, whether it is file upload or
not. If your computer won't break, then what's the problem? Uploaded it,
ask how big each form-part is, and do the appropriate thing.

Having said that, you could hack/subclass the read_multipart method
of CGI.pm to have it, at a certain size, stop copying the data into the
file, something like:

while (defined($data = $buffer->read)) {
if (defined $self->{'.upload_hook'})
{
$totalbytes += length($data);
&{$self->{'.upload_hook'}}($filename ,$data, $totalbytes,
$self->{'.upload_data'});
}
##print $filehandle $data;
print $filehandle $data unless $totalbytes > $FOO::Asterbing_size;
}

Of course, you would have to set a flag or something so that you know that
this has happened.

Xho
 
S

Sherm Pendley

Asterbing said:
Because when CONTENT_LENGTH has been informed by server itself after
receiving of entire POST, I can effectively read STDIN, extract uploaded
data and do the substraction.

Don't read from stdin. Read from the filehandle you get from CGI.pm.
Say, I wish to avoid any upload (I mean write on server, since data may
already be in STDIN) above 1MB, how to proceed ?

Wish for something that's possible.

sherm--
 
S

Sherm Pendley

Asterbing said:
When CONTENT_LENGTH not given during request, server has to compute it
from real data received in STDIN

The server isn't receiving data from stdin, it's receiving it from a net-
work connection.
: so, in this case how does CGI.pm is
able to stop receiving to STDIN precisely when $CGI::pOST_MAX is reached

It doesn't make the client stop sending data, it just ignores anything
beyond POST_MAX.
Not sure it's able to do that. What do you think ?

It can and does stop receiving data. Your confusion is the result of
thinking that it *also* causes the client to stop *sending* data.

sherm--
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,997
Messages
2,570,239
Members
46,827
Latest member
DMUK_Beginner

Latest Threads

Top