what to feed UserAgent?

[dan baker]

I have a little script that has been working fine for a long time
which feeds some parameters to a script on a remote server and
extracts some information from the response. Recently, the UserAgent
portion of the script has been returning an error response 302 "server
temporarily unavailable..."

however, if I go to the remote server and manually use their input
FORM, I get a results... so I am suspecting that the site is doing
some other check on where the request is originating from. I tried
grabbing a copy of their form and making sure that all the field names
and values my UserAgent is sending are the same as what their FORM is
sending.

is there some other information I can pass along thru UserAgent to
make it look more like the request originates from the remote server
rather than from my domain? Do variables like HTTP_HOST and others get
passed along? Can they be set by UserAgent?

A snippet of my code looks like:

use LWP::UserAgent ;
use HTTP::Request::Common ;
my $ua = LWP::UserAgent->new;
my $req = POST 'http://www.remotehost.com/cgi-bin/theirscript.jsp',
[
Selection => 1,
Submit.x => 0,
Submit.y => 0,
address => $Address ,
zipcode => $Zip
] ;
my $response = $ua->request($req);
unless ( $response->is_success ) { # request failed for some reason
$tempString = $response->status_line ;
print STDERR "server gave error 302 \n" ;
}

 

[Matt Garrish]

I have a little script that has been working fine for a long time
which feeds some parameters to a script on a remote server and
extracts some information from the response. Recently, the UserAgent
portion of the script has been returning an error response 302 "server
temporarily unavailable..."

however, if I go to the remote server and manually use their input
FORM, I get a results... so I am suspecting that the site is doing
some other check on where the request is originating from. I tried
grabbing a copy of their form and making sure that all the field names
and values my UserAgent is sending are the same as what their FORM is
sending.

Maybe it's a sign you should stop doing whatever it is you are doing.

Matt

 

[Richard Trahan]

dan baker wrote:
(snip)

I had this problem with a script I wrote to download a dictionary
that didn't want to be downloaded. The admin fought me by looking at
my HTTP_USER_AGENT value, which has a distinctive look when it comes
from a Perl script. You can set its value in Perl to spoof any browser.
To see what yours looks like, enter this address:
www.richsniches.com/cgi-bin/what.pl
This script echoes all the environment variables it sees when it runs,
including your HTTP_USER_AGENT. Set your user agent object to that
value and see if that works.

HTH

 

[dan baker]

dan baker wrote: ....
This script echoes all the environment variables it sees when it runs,
including your HTTP_USER_AGENT. Set your user agent object to that
value and see if that works.
--------------

do you just add definitions for the HTTP_ variable to the list?
something like:
my $ua = LWP::UserAgent->new;
my $req = POST 'http://somedomain.com/cgi-bin/somescript.jsp',
[
Selection => 1,
Submit.x => 17,
Submit.y => 10,
HTTP_USER_AGENT => 'whatever...' ];
or , do you have to do something different to pass the ENV vars in a
header or something?

d

 

[dan baker]

Maybe it's a sign you should stop doing whatever it is you are doing.
--------------

it *should* be ok, the resource I am attempting to automate is not a
fee for service type thing, I just want to automate the extraction of
a portion of the response to condense some information.

d

 

[Richard Trahan]

dan baker wrote:

Here's the actual line I used:

my $ua = LWP::UserAgent->new(agent => "Mozilla/4.76 [en] (Win98; U)");

 

[Ben Morrow]

Quoth (e-mail address removed) (dan baker):

dan baker wrote: ...
This script echoes all the environment variables it sees when it runs,
including your HTTP_USER_AGENT. Set your user agent object to that
value and see if that works.
--------------

do you just add definitions for the HTTP_ variable to the list?
something like:
my $ua = LWP::UserAgent->new;
my $req = POST 'http://somedomain.com/cgi-bin/somescript.jsp',
[
Selection => 1,
Submit.x => 17,
Submit.y => 10,
HTTP_USER_AGENT => 'whatever...' ];
or , do you have to do something different to pass the ENV vars in a
header or something?

Read the docs for LWP::UserAgent, specifically for LWP::UserAgent->new.

Ben

 

[Tore Aursand]

I have a little script that has been working fine for a long time which
feeds some parameters to a script on a remote server and extracts some
information from the response. Recently, the UserAgent portion of the
script has been returning an error response 302 "server temporarily
unavailable..."

The URL you're trying to reach can do a number of things do ensure that
only "valid" requests are coming through, the most often one is a) find
out _what_ is doing the request and b) find out _from where_ the request
is coming.

Everything you need to know is in the LWP::UserAgent documentation (or
related documents).

 

[Ilmari Karonen]

my $response = $ua->request($req);
unless ( $response->is_success ) { # request failed for some reason
$tempString = $response->status_line ;
print STDERR "server gave error 302 \n" ;
}

How about printing the _actual_ error? You know, that thing you're
putting in $tempString. I'm willing to bet it's _not_ 302 anything.

(Just FYI, HTTP status code 302 means temporary redirection, it's not
even an error. Real errors have code numbers of 400 or more.)

 

[dan baker]

dan baker wrote:

Here's the actual line I used:

my $ua = LWP::UserAgent->new(agent => "Mozilla/4.76 [en] (Win98; U)");

-------------
darn, didn't work.

I looked thru the syntax in the docs, but didnt see anything about how
to overide what gets passed for values like HTTP_REFERER and such. Is
this possible?

d

 

[Tore Aursand]

my $ua = LWP::UserAgent->new(agent => "Mozilla/4.76 [en] (Win98; U)");

darn, didn't work.

Let me emphasize that the code above work _as intended_. It didn't solve
your problem, possibly.

I looked thru the syntax in the docs, but didnt see anything about how
to overide what gets passed for values like HTTP_REFERER and such. Is
this possible?

Yes. What part of the documentation didn't you read? The HTTP_REFERER is
a part of the HTTP headers, and LWP::UserAgent's documentation has a
reference to the HTTP::Headers module (which the new() method takes as
input argument).

Read the documentation for HTTP::Headers, and create a object of it which
you feed to LWP::UserAgent. Never tried it myself, but it ought to work.