Probelm to post XML data in a loop. First time XML is posted, second time data is getting truncated.

[vamsi.aluru]

Hi,
I am posting an XML document to an asp applicaiton. This XML
document, i am sending in a loop. That is, if the XML document is big,
I am sending the first 50 records first time, then next 50 records and
so on.

The problem is, I am able to send first 50, but next 50 is
causing problem and the asp application is receiving the truncated XML.
When I send 20 after 20, first two times the xml goes fine, but the
third time the XML is truncating.

Any idea what is the problem? I am using post, so the size
should not be a problem. And more over the XML size is not more than
50k each time.

I am using the below code. I am psing only part of the code
where I am posting. if i post the full code, it will be very big.


for($x=0;$x<=$count-1;$x++){

# getting the XML content.
$XML_OUT=$xmlcnt[$x];


print 'xml==========='.$XML_OUT;
$ua = LWP::UserAgent->new;
$ua->timeout(36000);
$ua->max_size(1000000000);
# post the XML to the below url.
$req = HTTP::Request->new(POST =>
'http://viz-ams-001/facility/mapping.aspx');
# set the conetnt type of the post.
$req->content_type('application/x-www-form-urlencoded');
$req->content_length(1000000000);

#while posting, assign the XML to Doc variable. This variable acts
a form variable.
$req->content('Doc='.$XML_OUT);
# Pass request to the user agent and get a response back
$res = $ua->request($req);

# if response is available do the below.
if ($res->is_success) {

print 'success';
}
else{

print 'error';
}

 

[vamsi.aluru]

I forgot to add the first lines of the code. i am using these at the
begning.

use CGI;
use DBI;
use DBD::Oracle;
use LWP::UserAgent;
use HTTP::Request::Common;
use HTTP::Headers;
use HTTP::Headers::Util qw(split_header_words);

 

[J. Gleixner]

The problem is, I am able to send first 50, but next 50 is
causing problem and the asp application is receiving the truncated XML.
When I send 20 after 20, first two times the xml goes fine, but the
third time the XML is truncating.

Where is it being truncated? Before you send it or on the server's side?

Any idea what is the problem? I am using post, so the size
should not be a problem. And more over the XML size is not more than
50k each time.

I am using the below code. I am psing only part of the code
where I am posting. if i post the full code, it will be very big.


for($x=0;$x<=$count-1;$x++){

# getting the XML content.
$XML_OUT=$xmlcnt[$x];


print 'xml==========='.$XML_OUT;
$ua = LWP::UserAgent->new;
$ua->timeout(36000);
$ua->max_size(1000000000);
# post the XML to the below url.
$req = HTTP::Request->new(POST =>
'http://viz-ams-001/facility/mapping.aspx');
# set the conetnt type of the post.
$req->content_type('application/x-www-form-urlencoded');
$req->content_length(1000000000);

#while posting, assign the XML to Doc variable. This variable acts
a form variable.
$req->content('Doc='.$XML_OUT);
# Pass request to the user agent and get a response back
$res = $ua->request($req);

# if response is available do the below.
if ($res->is_success) {

print 'success';
}
else{

print 'error';
}

Untested...

my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(
POST=>'http://viz-ams-001/facility/mapping.aspx');
$req->content_type('application/x-www-form-urlencoded');

for my $XML_OUT ( @xmlcnt )
{
print "xml===========$XML_OUT\n";

$req->content( "Doc=$XML_OUT" );
my $res = $ua->request( $req );
if ($res->is_success)
{
print "success\n";
}
else
{
print 'error: ', $res->as_string, "\n";
}
}


Possibly XML_OUT contains other data? From perldoc HTTP::Request:

"Note that the content should be a string of bytes. Strings in perl
can contain characters outside the range of a byte. The Encode
module can be used to turn such strings into a string of bytes."

 

[vamsi.aluru]

ASP application is receiving the truncated xml string. so it should
have got truncated when the post happens.


Possibly XML_OUT contains other data?

I dont understand this statement. XML_OUT contains characters, which
for a XML string. If the problem is with XML, then first 50 shd also
fail.

I think size is what creating the problem. but again, post sholud not
have a size limit.

is there a way to flush the data after the post happens? or is there a
way to create a new request each time, closing the old one.?

The problem is, I am able to send first 50, but next 50 is
causing problem and the asp application is receiving the truncated XML.
When I send 20 after 20, first two times the xml goes fine, but the
third time the XML is truncating.

Where is it being truncated? Before you send it or on the server's side?

Any idea what is the problem? I am using post, so the size
should not be a problem. And more over the XML size is not more than
50k each time.

I am using the below code. I am psing only part of the code
where I am posting. if i post the full code, it will be very big.


for($x=0;$x<=$count-1;$x++){

# getting the XML content.
$XML_OUT=$xmlcnt[$x];


print 'xml==========='.$XML_OUT;
$ua = LWP::UserAgent->new;
$ua->timeout(36000);
$ua->max_size(1000000000);
# post the XML to the below url.
$req = HTTP::Request->new(POST =>
'http://viz-ams-001/facility/mapping.aspx');
# set the conetnt type of the post.
$req->content_type('application/x-www-form-urlencoded');
$req->content_length(1000000000);

#while posting, assign the XML to Doc variable. This variable acts
a form variable.
$req->content('Doc='.$XML_OUT);
# Pass request to the user agent and get a response back
$res = $ua->request($req);

# if response is available do the below.
if ($res->is_success) {

print 'success';
}
else{

print 'error';
}

Untested...

my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(
POST=>'http://viz-ams-001/facility/mapping.aspx');
$req->content_type('application/x-www-form-urlencoded');

for my $XML_OUT ( @xmlcnt )
{
print "xml===========$XML_OUT\n";

$req->content( "Doc=$XML_OUT" );
my $res = $ua->request( $req );
if ($res->is_success)
{
print "success\n";
}
else
{
print 'error: ', $res->as_string, "\n";
}
}


Possibly XML_OUT contains other data? From perldoc HTTP::Request:

"Note that the content should be a string of bytes. Strings in perl
can contain characters outside the range of a byte. The Encode
module can be used to turn such strings into a string of bytes."

 

[J. Gleixner]

ASP application is receiving the truncated xml string. so it should
have got truncated when the post happens.

Why are you top posting??

Since you are printing out the data, is it 'truncated' before you send it?

Possibly XML_OUT contains other data?

I dont understand this statement. XML_OUT contains characters, which
for a XML string. If the problem is with XML, then first 50 shd also
fail.

What's 'shd'? Please use full words.

I was only posting a possible issue. You know the data, so if you're
sure it's OK, then that's fine with me. It seems odd that the
application would work for a bunch of requests, then seem to fail. That
usually points to an issue with the data.

I think size is what creating the problem. but again, post sholud not
have a size limit.

is there a way to flush the data after the post happens? or is there a
way to create a new request each time, closing the old one.?

That's not needed. This is an HTTP request. It sends the data and
receives a response. You are creating a new request each time.

If you can verify that what you're sending is accurate [ by debugging,
maybe .. use Data:umper; print Dumper( $req ) or print
$req->content(); after you set it, and looking at what's being POSTed. ]
then it sounds like an issue on the other side.

 

[Ian Wilson]

Please don't top-post, please intersperse your replies after the
relevant parts of the quotation - like this ...

(note I had to reformat the posting after applying "rewrap" in my
newsreader - consequently the perl scripts may be laid out differently
than originally posted)

ASP application is receiving the truncated xml string. so it should
have got truncated when the post happens.

It would help if you could better describe the difference between the
output you expected and the output you actually got.

I guess your XML is too long to post here (50 lines) - do you get a
similar problem when you decrease your chunk size, e.g. to 5 lines?

If you read the posting guidelines, you will find that it is recommended
that you edit your program to produce the smallest working program that
can be used by others to reproduce your problem.

for($x=0;$x<=$count-1;$x++){
# getting the XML content.
$XML_OUT=$xmlcnt[$x];

I think I'd write the above as
foreach my $XML_OUT (@xmlcnt) {

Did this print what you expected? If not, can you describe the difference?

That seems bogus to me. An Internet maxim is "be conservative in what
you produce but be generous in what you accept". If I could, I'd put the
correct size into the content_length.

I dont understand this statement. XML_OUT contains characters, which
for a XML string. If the problem is with XML, then first 50 shd also
fail.

I'm not sure I understand it either, it may be a reference to multi-byte
character encodings such as UTF-8. Unless I was certain that my XML is
in a single-byte encoding such as ISO-8859-1 (Latin-1), I'd read the
HTTP::Request docs carefully and try to find out whether HTTP standards
have anything to say on the subject.

This illustrates why it is useful to post a (small) sample set of data
that others can examine and/or use for testing your code.

I think size is what creating the problem. but again, post sholud not
have a size limit.

You've promised the ASP service that you are sending it one gigabyte of
XML. Maybe the ASP service hasn't been tested with a gigabyte of data,
maybe it has some internal limit? I wouldn't assume it is just going to
ignore the content-length header.

is there a way to flush the data after the post happens? or is there
a way to create a new request each time, closing the old one.?

Unless I am mistaken, your original code was doing this.

It should be relatively simple to create a Perl CGI script that can be
targetted instead of the ASP URI - this might help in testing.

 

[vamsi.aluru]

I figured out the problem.

I am having an '&' symbol in one of the data elements. I am actually
replacing that with '&amp;'. But the actual problem was, I did not
escpae the xml before I post to asp application. Because of this, the
xml is getting truncated from the place where & symbol is there.

I changed the code like this.

$req->content('Doc='.CGI::escape($XML_OUT));

CGI::escacpe was not there previously.

Now the code works.

Thanks everybody for replies.

Please don't top-post, please intersperse your replies after the
relevant parts of the quotation - like this ...

(note I had to reformat the posting after applying "rewrap" in my
newsreader - consequently the perl scripts may be laid out differently
than originally posted)

ASP application is receiving the truncated xml string. so it should
have got truncated when the post happens.

It would help if you could better describe the difference between the
output you expected and the output you actually got.

I guess your XML is too long to post here (50 lines) - do you get a
similar problem when you decrease your chunk size, e.g. to 5 lines?

If you read the posting guidelines, you will find that it is recommended
that you edit your program to produce the smallest working program that
can be used by others to reproduce your problem.

for($x=0;$x<=$count-1;$x++){
# getting the XML content.
$XML_OUT=$xmlcnt[$x];

I think I'd write the above as
foreach my $XML_OUT (@xmlcnt) {

Did this print what you expected? If not, can you describe the difference?

That seems bogus to me. An Internet maxim is "be conservative in what
you produce but be generous in what you accept". If I could, I'd put the
correct size into the content_length.

I dont understand this statement. XML_OUT contains characters, which
for a XML string. If the problem is with XML, then first 50 shd also
fail.

I'm not sure I understand it either, it may be a reference to multi-byte
character encodings such as UTF-8. Unless I was certain that my XML is
in a single-byte encoding such as ISO-8859-1 (Latin-1), I'd read the
HTTP::Request docs carefully and try to find out whether HTTP standards
have anything to say on the subject.

This illustrates why it is useful to post a (small) sample set of data
that others can examine and/or use for testing your code.

I think size is what creating the problem. but again, post sholud not
have a size limit.

You've promised the ASP service that you are sending it one gigabyte of
XML. Maybe the ASP service hasn't been tested with a gigabyte of data,
maybe it has some internal limit? I wouldn't assume it is just going to
ignore the content-length header.

is there a way to flush the data after the post happens? or is there
a way to create a new request each time, closing the old one.?

Unless I am mistaken, your original code was doing this.

It should be relatively simple to create a Perl CGI script that can be
targetted instead of the ASP URI - this might help in testing.

 

[A. Sinan Unur]

(e-mail address removed) wrote in

[ Please do not top-post and please do quote only the relevant parts of
previous messages ]

I figured out the problem.

We are happy for you (honestly) but you have failed. The reason you
posted here was because you someone else to find the problem, thereby
saving you time and effort. That did not happen for a number of reasons.

I am having an '&' symbol in one of the data elements. I am actually
replacing that with '&amp;'. But the actual problem was, I did not
escpae the xml before I post to asp application. Because of this, the
xml is getting truncated from the place where & symbol is there.

Because you did not post a short but complete program that could run
easily by others, and a good sample of the data, a lot of us did not
respond to your question.

Those who responded had a hard time getting information from you.

I hope you'll read and follow the posting guidelines the next time.

Sinan