xmlhttprequest to get dynamically generated data seems to have a size limit

[ted benedict]

hi everybody, i hope this is the right place to discuss this weird
behaviour. i am getting dynamically generated text or xml from the
server side using xmlhttprequest. if the server side data is STATIC, i
can have whatever size of data i want.

but if the data (xml or text) is generated dynamically using php, then
there seems to be a size limit! xmlhttprequest's responseText is
truncated, and the xml therefore not well fromed. in border cases (just
about a little above some file size limit), there is even RANDOM
behaviour, sometimes it works, and when refreshing it does not work,
and the next time it works again.

anybody having similar issues? if you're interested, i can also post
code and a detailed summary of when this behaviour occurs.

greetings to all of you,
ted

 

[VK]

hi everybody, i hope this is the right place to discuss this weird
behaviour. i am getting dynamically generated text or xml from the
server side using xmlhttprequest. if the server side data is STATIC, i
can have whatever size of data i want.

but if the data (xml or text) is generated dynamically using php, then
there seems to be a size limit! xmlhttprequest's responseText is
truncated, and the xml therefore not well fromed. in border cases (just
about a little above some file size limit), there is even RANDOM
behaviour, sometimes it works, and when refreshing it does not work,
and the next time it works again.

anybody having similar issues? if you're interested, i can also post
code and a detailed summary of when this behaviour occurs.

greetings to all of you,
ted

I bet my sugar against your coffee that you are hitting the GET limits
imposed by IE:
<http://www.ajaxtoolbox.com/forum/viewtopic.php?t=26>
derived from
<http://support.microsoft.com/default.aspx?scid=KB;en-us;q208427>

You should use only clj endorsed well-formed libraries like AjaxToolBox
to be protected from the known, mal-known and unknown issues of Ajax
technologies sets.

IMHighlyHO

 

[Martin Honnen]

ted benedict wrote:

but if the data (xml or text) is generated dynamically using php, then
there seems to be a size limit! xmlhttprequest's responseText is
truncated, and the xml therefore not well fromed.

Consider not to use responseText at all, rather make your PHP script do
header('Content-Type: application/xml');
before it sends any XML data and that way the XMLHttpRequest object you
use will automatically parse the response body and build responseXML.

in border cases (just
about a little above some file size limit), there is even RANDOM
behaviour, sometimes it works, and when refreshing it does not work,
and the next time it works again.

Is that happening with all browsers or just one particular browser
respectively one particular implementation of XMLHttpRequest?
What does
httpRequestObject.status
httpRequestObject.statusText
httpRequestObject.getAllResponseHeaders()
look like when you have problems?

How long does (e.g. some seconds or a minute or even several minutes?)
the PHP script need to write the complete data, perhaps the browser
times out before the script has finished?

 

[Thomas 'PointedEars' Lahn]

hi everybody, i hope this is the right place to discuss this weird
behaviour. i am getting dynamically generated text or xml from the
server side using xmlhttprequest. if the server side data is STATIC, i
can have whatever size of data i want.

but if the data (xml or text) is generated dynamically using php, then
there seems to be a size limit! xmlhttprequest's responseText is
truncated, and the xml therefore not well fromed.

in border cases (just
about a little above some file size limit), there is even RANDOM
behaviour, sometimes it works, and when refreshing it does not work,
and the next time it works again.
[...]

I bet my sugar against your coffee that you are hitting the GET limits
imposed by IE:

You lose again. He was talking about the response text (responseText
property) being truncated, which does not have anything to do with the
URI length limit IE imposes.


PointedEars

 

[Thomas 'PointedEars' Lahn]

[...]
but if the data (xml or text) is generated dynamically using php, then
there seems to be a size limit!

Maybe you generate a wrong Content-Length header. The reason of this
could be that you use a flawed mod_gzip version, or a flawed cgi_buffer
version or use the latter in the wrong way. Whatever the reason is,
the error is unlikely to be on the client side.

xmlhttprequest's responseText is truncated, and the xml therefore not well
fromed. [...]

Why are you using `responseText' in the first place? If you serve an XML
document (text/xml) as response, use `responseXML' and parse the resulting
element tree instead.


HTH

PointedEars

 

[VK]

You lose again. He was talking about the response text (responseText
property) being truncated, which does not have anything to do with the
URI length limit IE imposes.

You lose again. The truncation happens on the client side *before*
request is being sent - if it's GET and if it's longer than
GET-maxlength.

You really should use the software you're advising about - because
ECMA/W3C text data is mainly a source of fun rather than information.
;-)

 

[ted benedict]

thanks for the answer, i only use responseText for checking. (because
if it is truncated, i won't get a responseXML object (because the xml
is malformed if the text of it is truncated).

the border case fun RANDOM behavior happens with IE (what a surprise.
;-) )

thanks for the hints, i'll check these

httpRequestObject.status
httpRequestObject.statusText
httpRequestObject.getAllResponseHeaders()

the php only has some seconds, not really much.

 

[ted benedict]

Thomas 'PointedEars' Lahn, thanks for your answer.

i don't really generate any header information myself. i just use a php
script and build the xml using echo('....') commands. (because i have
to use php4 and don't want to use the deprecated xml dom lib)

 

[ted benedict]

ok i decided to post more detailed information.

1. dynamically generated TEXT from a server side php script.
---------------------------------------------------------------------------------------
the server side script looks like this:
<?php
echo('text ');
?>

if i use xmlhttprequest to GET this file, asynchronously, and then use
(AFTER the request object's state is 4) .responseText, i can read out
'text'. now, depending on the size of this file, i'll get the complete
generated text or not.

case 1: echo('text '); repeated for about 80kB of space.
in firefox, no problems occured. in internet explorer, the
..responseText returned the full generated text.

case 2: echo('text '); repeated for about 120kB of space.
in firefox, no problems occured. in internet explorer, the
..responseText only returns a truncated version, not everything.

2. dynamically generated XML from a server side php script.
---------------------------------------------------------------------------------------
the server side script looks like this:
<?php
header("Content-type: text/xml; charset=iso-8859-1");
echo('<entries>');
echo('<entry>');
// more elements
echo('</entry>');
echo('</entries>');
?>

case 1: copy pasted the entry generating echo statements, repeated for
about 55kB of space.
using .responseText, in firefox, no problems occured. in internet
explorer, the full generated xml is returned
using .responseXML and counting the entry elements for example, in
firefox, no problems, in ie no problems

case 2: copy pasted the entry generating echo statements, repeated for
about 220kB of space.
using .responseText, in firefox and in internet explorer, only partial
data is displayed if i print it to a <div> for example
using .responseXML and counting the entry elements for example, in
firefox and in ie, the responseXML cannot be accessed as it received
truncated data and did not build up correctly.


now this was a little a simplification because i did more experiments
to get the file sizes and cases more exactly (ie fails to receive and
build the responseXML object sooner than firefox, for example)


i hope this explained my problem? i hope it was interesting for you to
see what weird things happen

greetings and all the best to you all

 

[Dag Sunde]

I bet my sugar against your coffee that you are hitting the GET limits
imposed by IE:
<http://www.ajaxtoolbox.com/forum/viewtopic.php?t=26>
derived from
<http://support.microsoft.com/default.aspx?scid=KB;en-us;q208427>

And what does the response have to do with IE's GET limit?

Read the OP properly:
"..., there is even RANDOM behaviour, sometimes it works,
and when refreshing it does not work, and the next time
it works again."

Refreshing implies the *same* GET URI... and sometimes it works!
IE's GET limit is fixed, and does *not* vary between calls.

To the OP:

Can you post your response function, so we don't have to guess
what goes wrong?

My guess is that you post or get async, but doesn't wait until
readyState and status is set properly, so you get a race with
the server...:

if (req.readyState == 4) {
if (req.status == 200) {
// ...processing...
} else {
alert(\"There was a problem retrieving the XML data:\n\");
}
}

 

[VK]

ok i decided to post more detailed information.

1. dynamically generated TEXT from a server side php script.
---------------------------------------------------------------------------------------
the server side script looks like this:
<?php
echo('text ');
?>

if i use xmlhttprequest to GET this file, asynchronously, and then use
(AFTER the request object's state is 4) .responseText, i can read out
'text'. now, depending on the size of this file, i'll get the complete
generated text or not.

case 1: echo('text '); repeated for about 80kB of space.
in firefox, no problems occured. in internet explorer, the
.responseText returned the full generated text.

case 2: echo('text '); repeated for about 120kB of space.
in firefox, no problems occured. in internet explorer, the
.responseText only returns a truncated version, not everything.

2. dynamically generated XML from a server side php script.
---------------------------------------------------------------------------------------
the server side script looks like this:
<?php
header("Content-type: text/xml; charset=iso-8859-1");
echo('<entries>');
echo('<entry>');
// more elements
echo('</entry>');
echo('</entries>');
?>

case 1: copy pasted the entry generating echo statements, repeated for
about 55kB of space.
using .responseText, in firefox, no problems occured. in internet
explorer, the full generated xml is returned
using .responseXML and counting the entry elements for example, in
firefox, no problems, in ie no problems

case 2: copy pasted the entry generating echo statements, repeated for
about 220kB of space.
using .responseText, in firefox and in internet explorer, only partial
data is displayed if i print it to a <div> for example
using .responseXML and counting the entry elements for example, in
firefox and in ie, the responseXML cannot be accessed as it received
truncated data and did not build up correctly.


now this was a little a simplification because i did more experiments
to get the file sizes and cases more exactly (ie fails to receive and
build the responseXML object sooner than firefox, for example)


i hope this explained my problem? i hope it was interesting for you to
see what weird things happen

As I pointed in my first post:
<http://support.microsoft.com/default.aspx?scid=KB;en-us;q208427>

Please take your time to read it though. It is nasty (the fix), but it
is forced after numerous Address Buffer Overflow attacks. Firefox is so
generous just because it is still in the position of "Uncachable Joe"
("-Why is this Joe so uncachable? - Because why the hell anyone would
want to cache him?!")

Change you submission method to POST (using say www.ajaxtoolbox.com)
and be happy ever after. ;-)

 

[Martin Honnen]

ted benedict wrote:

<?php
header("Content-type: text/xml; charset=iso-8859-1");

Note that MSXML used by IE/Win does not look at charset parameters in
HTTP headers, rather it looks for the XML declaration. If you want to
send ISO-8859-1 encoded XML then make sure that your PHP script sends a
proper XML declaration at the very beginning e.g.
echo '<?xml version="1.0" encoding="ISO-8859-1"?>' . "\n";
Of course then you need to make sure you send ISO-8859-1 encoded data,
if that

echo('<entries>');
echo('<entry>');
// more elements
echo('</entry>');
echo('</entries>');

is all you send then there should not be a problem usually, unless you
are doing some exotic encodings with PHP. But of course depending on
where you get your data from and what you do in PHP your output can have
any encoding, simply setting the HTTP response header does not ensure

case 2: copy pasted the entry generating echo statements, repeated for
about 220kB of space.
using .responseText, in firefox and in internet explorer, only partial
data is displayed if i print it to a <div> for example
using .responseXML and counting the entry elements for example, in
firefox and in ie, the responseXML cannot be accessed as it received
truncated data and did not build up correctly.

What does MSXML/IE show for
request.responseXML.parseError.reason
?

If that is a GET requests, what happens if you simply load the URL in an
IE or Firefox browser window?

 

[Thomas 'PointedEars' Lahn]

You lose again. The truncation happens on the client side *before*
request is being sent - if it's GET and if it's longer than
GET-maxlength.

You really should use the software you're advising about - because
ECMA/W3C text data is mainly a source of fun rather than information.
;-)

Idiot. The problem is about the truncated *R*E*S*P*O*N*S*E*!


PointedEars

 

[Thomas 'PointedEars' Lahn]

[...]
if i use xmlhttprequest to GET this file, asynchronously, and then use
(AFTER the request object's state is 4) .responseText, i can read out ^^^^^^^^^^^^
'text'. now, depending on the size of this file, i'll get the complete
generated text or not.

case 1: echo('text '); repeated for about 80kB of space.
in firefox, no problems occured. in internet explorer, the
.responseText returned the full generated text. ^^^^^^^^
case 2: echo('text '); repeated for about 120kB of space.
in firefox, no problems occured. in internet explorer, the
.responseText only returns a truncated version, not everything. ^^^^^^^^
[...]
case 1: copy pasted the entry generating echo statements, repeated for
about 55kB of space.
using .responseText, in firefox, no problems occured. in internet ^^^^^^^^
explorer, the full generated xml is returned
using .responseXML and counting the entry elements for example, in ^^^^^^^^
firefox, no problems, in ie no problems

case 2: copy pasted the entry generating echo statements, repeated for
about 220kB of space.
using .responseText, in firefox and in internet explorer, only partial ^^^^^^^^
data is displayed if i print it to a <div> for example
using .responseXML and counting the entry elements for example, in ^^^^^^^^
firefox and in ie, the responseXML cannot be accessed as it received ^^^^^^^^
truncated data and did not build up correctly.

now this was a little a simplification because i did more experiments
to get the file sizes and cases more exactly (ie fails to receive and
build the responseXML object sooner than firefox, for example) ^^^^^^^^^^^
[...]

As I pointed in my first post:
<http://support.microsoft.com/default.aspx?scid=KB;en-us;q208427>

Are you dumb and blind/deaf? R-E-S-P-O-N-S-E. Spell it.
Then spell R-E-Q-U-E-S-T, as in "HTTP GET request" ...


PointedEars

 

[Thomas 'PointedEars' Lahn]

2. dynamically generated XML from a server side php script.
---------------------------------------------------------------------------------------
the server side script looks like this:
<?php
header("Content-type: text/xml; charset=iso-8859-1");
echo('<entries>');
echo('<entry>');
// more elements
echo('</entry>');
echo('</entries>');
?>

One more comment: Such coding is seldom wise.

1. `echo' is not a PHP function, but a language feature. Good code style
calls for not using the parantheses, unless the expression used is more
complicated.

2. You do not have to feed to PHP's parser what is not PHP code. Therefore
it is PHP, the PHP _Hypertext Preprocessor_

Consider this:

<?php
// PHP code
?><entries>
<entry>
<?php
// PHP code that generates more elements
?>
</entry>
</entries><?php
PHP code
?>

See also <URL:http://www.php.net/manual/en/tutorial.firstpage.php>


X-Post & F'up2 comp.lang.php

PointedEars

 

[VK]

Idiot. The problem is about the truncated *R*E*S*P*O*N*S*E*!

Wash your tong, baby ;-)

URI limitation has been introduced in IE to protect it from malformed
*responses*.

<http://support.microsoft.com/default.aspx?scid=KB;en-us;q208427>
<quote>
Internet Explorer has a maximum uniform resource locator (URL) length
of 2,083 characters, with a maximum path length of 2,048 characters.
This limit applies to both POST and GET request URLs.
</quote>

This protection works in both sides.

If your GET *request* is longer than the limit, it will:
1) Raise "access violation" error on IE 5.x with "hot fix" installed
or
2) Will be silently truncated to 2,083 chars on IE 5.5 SP1 or higher

If the GET *response* is longer than allowed, it will be silently
truncated to the allowed maximum of 2,083 chars.

P.S. To Thomas: please either install Windows + IE for testing or do
not participate in Windows/IE related questions - just listen for now.

 

[Thomas 'PointedEars' Lahn]

Wash your tong, baby ;-)

Power on your brain, in case you really have one.

URI limitation has been introduced in IE to protect it from malformed
*responses*.

The "malformed response" received then is but due to the malformed request!


PointedEars

 

[Thomas 'PointedEars' Lahn]

Wash your tong, baby ;-)

Power on your brain, in case you really have one.

URI limitation has been introduced in IE to protect it from malformed
*responses*.

The "malformed response" (i.e. the 4xx status message and the like) received
then is only due to the _wrong (because truncated) URI_ submitted for the
corresponding request!

However, the request is OK with the OP, yet the response is still broken,
and only if the resource is a dynamically created one.

After having a pillowful of good night's sleep you will maybe able to
recognize that. However, I estimate the statistical probability of that
to be near 0.


PointedEars

 

[Matt Kruse]

If the GET *response* is longer than allowed, it will be silently
truncated to the allowed maximum of 2,083 chars.

I'd like to see one single test case for this.

A "GET response" can be anything. When I load any url in my browser, it's a
GET request, and the response can be several kb or hundreds of megabytes. If
you're talking about something else, you're using the wrong terms.

 

[VK]

A "GET response" can be anything. When I load any url in my browser, it's a
GET request, and the response can be several kb or hundreds of megabytes. If
you're talking about something else, you're using the wrong terms.

Sorry, but this is a kind of paranoia now. I show you *official* MSDN
behavior documentation, but *I* myself also have to prove that it is
not a conspiracy between VK and Microsoft.

As someone said in other thread, it is their browser and it acts as it
was programmed by developers. You don't like it - you don't fn use it.
But for cross-browser development you are forced to deal with the fn
reality, not with your own ideas of right and wrong behavior. Or even
do not deal with the reality - but do not call your solution
cross-browser then.

<http://support.microsoft.com/default.aspx?scid=KB;en-us;q208427> says
that
<quote>
RFC 2616, Hypertext Transfer Protocol -- HTTP/1.1
(ftp://ftp.isi.edu/in-notes/rfc2616.txt), does not specify any
requirement for URL length.
</quote>

- so Microsoft imposed their own requitement of 2,081 character per
URI. You don't like, you don't agree with it - so write a petition, go
to Microsoft One Str. for a "Give our URL's back" manifestation, do
whatever you want.

Nevertheless here some startup test to check locally and online: play
with the url.length. What will be next required burden of proof? That
document.getElementById returns DOM reference?

(You create empty URL2.html file yourselve, and below is URL1.html for
initial load)

<html>
<head>
<title>OMFG</title>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<script type="text/Jscript">
function OMFG() {
var url = '?';
for (var i=0; i<1000; i++) {
url+= 'OMFG';
}
alert(url.length);
try {
window.location.href = 'URL2.html' + url;
}
catch (e) {
alert(e.message);
}
}

window.onload = OMFG;
</script>
</head>

<body>