M
mark1.thompson45
Hi,
I am quite new to LWP but familiar with Perl so be kind. I am trying
to pull some data from a webpage using LWP, the actual page I need the
data from is accessed by clicking on a link off the main page, this
is:-
URL1 =
http://www.youthemanager.co.uk/Clev...me=YouTheManager2006&language=MIRROR&aff_id=7
If I take the link I need it starts a servlet and when I view the URL
for the servlet it gives me the following:-
URL2 =
http://www.youthemanager.co.uk/Clever/servlet/PlayerList?gameid=184&catidx=4
So I set about writing some code to get the information from URL2:-
-------------------------------------------------------------------
#!C:\\Perl\\bin\\perl -w
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);
use HTTP::Cookies;
my $ua= LWP::UserAgent->new(
requests_redirectable => ['GET', 'HEAD', 'POST'],
cookie_jar => HTTP::Cookies->new(
file => 'mycookies.txt',
autosave => 1
),
);
# Define user agent type
$ua->agent('Mozilla/8.0');
# Request object
my $req = GET
'http://www.youthemanager.co.uk/Clev...=YouTheManager2006&language=MIRROR&aff_id=7';
#my $req = GET
'http://www.youthemanager.co.uk/Clever/servlet/PlayerList?catidx=3';
# Make the request
my $res = $ua->request($req);
# Check the response
if ($res->is_success) {
print ">>".$res->content;
} else {
print ">> FAILED <<\n";
print $res->status_line ."\n";
}
exit 0;
-------------------------------------------------------------------
When I enable URL1 in my code and run it I get the contents back from
the webpage, when I enable URL2 in the code (this is the page I want)
then I get the following when the code is run:-
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("/");</SCRIPT></HEAD></HTML>
I think this maybe something to do with redirects but I thought the
code above would handle that, I know that the GET is used as apposed to
POST for this page as I have snooped the activity on my LAN.
Can anyone offer any advice on where to go from here ....
cheers, Mark.
I am quite new to LWP but familiar with Perl so be kind. I am trying
to pull some data from a webpage using LWP, the actual page I need the
data from is accessed by clicking on a link off the main page, this
is:-
URL1 =
http://www.youthemanager.co.uk/Clev...me=YouTheManager2006&language=MIRROR&aff_id=7
If I take the link I need it starts a servlet and when I view the URL
for the servlet it gives me the following:-
URL2 =
http://www.youthemanager.co.uk/Clever/servlet/PlayerList?gameid=184&catidx=4
So I set about writing some code to get the information from URL2:-
-------------------------------------------------------------------
#!C:\\Perl\\bin\\perl -w
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);
use HTTP::Cookies;
my $ua= LWP::UserAgent->new(
requests_redirectable => ['GET', 'HEAD', 'POST'],
cookie_jar => HTTP::Cookies->new(
file => 'mycookies.txt',
autosave => 1
),
);
# Define user agent type
$ua->agent('Mozilla/8.0');
# Request object
my $req = GET
'http://www.youthemanager.co.uk/Clev...=YouTheManager2006&language=MIRROR&aff_id=7';
#my $req = GET
'http://www.youthemanager.co.uk/Clever/servlet/PlayerList?catidx=3';
# Make the request
my $res = $ua->request($req);
# Check the response
if ($res->is_success) {
print ">>".$res->content;
} else {
print ">> FAILED <<\n";
print $res->status_line ."\n";
}
exit 0;
-------------------------------------------------------------------
When I enable URL1 in my code and run it I get the contents back from
the webpage, when I enable URL2 in the code (this is the page I want)
then I get the following when the code is run:-
<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("/");</SCRIPT></HEAD></HTML>
I think this maybe something to do with redirects but I thought the
code above would handle that, I know that the GET is used as apposed to
POST for this page as I have snooped the activity on my LAN.
Can anyone offer any advice on where to go from here ....
cheers, Mark.
