I need to read web pages without LWP

[Mark Healey]

For years I was running a web host out of my living room to parse and
digest some web pages for display on my Palm Pilot. It got hacked and I
got tired of doing all that maintenance so I bought a hosting account.

The problem is that they only have the modules that came with Apache
installed.

The modules I used are CGI, LWP::UserAgent, HTTP::Request, HTTP::Response,
URI::Escape.

So what I need to do is have a script that takes form data and passes that
to another sites form submission, takes the response, parses it for the
data I want and responds with a page formatted for a really small screen.

Can I do this without any modules and how hard would it be?

 

[Keith Keller]

For years I was running a web host out of my living room to parse and
digest some web pages for display on my Palm Pilot. It got hacked and I
got tired of doing all that maintenance so I bought a hosting account.

The problem is that they only have the modules that came with Apache
installed.

The modules I used are CGI, LWP::UserAgent, HTTP::Request, HTTP::Response,
URI::Escape.

You should read perldoc -q library, then maintain your own library
directory, rather than try to rewrite all that functionality yourself.

--keith

 

[Ric]

For years I was running a web host out of my living room to parse and
digest some web pages for display on my Palm Pilot. It got hacked and I
got tired of doing all that maintenance so I bought a hosting account.

The problem is that they only have the modules that came with Apache
installed.

The modules I used are CGI, LWP::UserAgent, HTTP::Request, HTTP::Response,
URI::Escape.

So what I need to do is have a script that takes form data and passes that
to another sites form submission, takes the response, parses it for the
data I want and responds with a page formatted for a really small screen.

Can I do this without any modules and how hard would it be?

You can do it with Sockets, pretty easy if you have some experience with
HTTP Protocol. But why not use HTTP::Request etc. upload them along with
your script all you need to do is to put them into INC

 

[Gunnar Hjalmarsson]

You should read perldoc -q library, then maintain your own library
directory, rather than try to rewrite all that functionality yourself.

That's one way. But even better, IMO, would be to have your web host
install those modules. Or, if they refuse to do so, leave them and get a
hosting account with a decent Perl installation available.

 

[John Bokma]

That's one way. But even better, IMO, would be to have your web host
install those modules. Or, if they refuse to do so, leave them and get
a hosting account with a decent Perl installation available.

Yup, can't agree more. I am sure that even the sites I have hosted for one
buck a month come with at least a decent Perl install. If not, me asking
is often sufficient

 

[zentara]

For years I was running a web host out of my living room to parse and
digest some web pages for display on my Palm Pilot. It got hacked and I
got tired of doing all that maintenance so I bought a hosting account.

The problem is that they only have the modules that came with Apache
installed.

The modules I used are CGI, LWP::UserAgent, HTTP::Request, HTTP::Response,
URI::Escape.

So what I need to do is have a script that takes form data and passes that
to another sites form submission, takes the response, parses it for the
data I want and responds with a page formatted for a really small screen.

Can I do this without any modules and how hard would it be?

For pure Sockets, these worked when I last tested them.


#!/usr/bin/perl -w
# webget-client 192.168.0.1 /~zentara/index.html
use IO::Socket;
unless (@ARGV > 1) { die "usage: $0 host document ..." }
$host = shift(@ARGV);
foreach $document ( @ARGV ) {
$remote = IO::Socket::INET->new( Proto => "tcp",
PeerAddr => $host,
PeerPort => "http(80)",
);
unless ($remote) { die "cannot connect to http daemon on $host" }
$remote->autoflush(1);
print $remote "GET $document HTTP/1.0\n\n";
while ( <$remote> ) { print }
close $remote;
}
__END__



#!/usr/bin/perl
# Usage: wwwgrab http://site.name.edu/ [filename] #
# #
# ex. wwwgrab http://www.engr.wisc.edu/~ballard/ #
# will get you the source behind my homepage and #
# wwwgrab http://www.engr.wisc.edu/~ballard/pics/me.gif #
# will get you a .gif image of my picture. #
# #
#########################################################
# Written by Jeff Ballard ([email protected]) 7/2/95 #
# URL: http://www.engr.wisc.edu/~ballard/ #
#########################################################
use warnings;
use Socket;

# Given an http address, rip it into its coresponding parts.

($_, $savefilename) = @ARGV;

if (!$_) {
print "Usage: $0 http://www.any.site/location [outputfilename]\n";
print " Where [outputfilename] is the optional filename to
create.\n";
print " (ommitting this means that it will mean that the data will
be\n";
die " sent to STDOUT. If this file exists, it will
beoverwritten)\n";
}

/http:\/\/([^\/]*)\/*([^ ]*)/;
$site = $1;
$file = "/".$2;

if (!$site) {die "$0: Fatal Error. You appear to have munged your URL
address.\nIt must be in the form of http://www.an\$"
}

$_ = $site;
/^([^:]*):*([^ ]*)/;
$site = $1;
$port = $2;
$port = 80 unless $port;

$hostname = $site;

#print STDERR "[$site] [$port] [$file]\n";

# Open a socket and get the data
($sockaddr,$there,$response,$tries) = ("Snc4x8");
$there = pack($sockaddr,2,$port, &getaddress($hostname));
($a, $b, $c, $d) = unpack('C4', $hostaddr);

$proto = (getprotobyname ('tcp'))[2];

if (!socket(S,AF_INET,SOCK_STREAM,$proto)) { die "$0: Fatal
Error.$!\n"; }
if (!connect(S,$there)) { die "$0: Fatal Error. $!\n"; }
select(S);$|=1;
select(STDOUT);
print S "GET $file\r\n";
if ($savefilename) {
open(OUT,">".$savefilename) || die "$0: Fatal error. Cannot create
$savefilename.\n";
}
while($line = <S>) {
if ($savefilename) {
print OUT $line;
} else {
print $line;
}
}
close(S);
if ($savefilename) {
close(OUT);
}

sub getaddress {
local($host) = @_;
local(@ary);
@ary = gethostbyname($host);
return(unpack("C4",$ary[4]));
}
__END__