LWP::Simple Not Working

[Larry Snider]

My web host has Perl 5.6.1 and LWP::Simple is installed.

When I execute the following code on my web host's server, the getstore()
part doesn't do anything and no errors are produced.

######################
#!/usr/bin/perl -w

use LWP::Simple;
my $document =
getstore('http://www.computercops.biz/ipbans.txt','ipbans.txt');
######################

Furthermore, the following always results in "Bad.":

######################
#!/usr/bin/perl

use strict;
use LWP::Simple;

my $url = 'http://computercops.biz/ipbans.txt';

head($url) ? print "Good.\n" : print "Bad.\n";
######################

What could be causing this lack of proper behavior? I did notice that I
cannot ping anything while in a shell. I get "ping: icmp open socket:
Operation not permitted". Could this be what's stopping it from working?

 

[Matt Garrish]

What could be causing this lack of proper behavior? I did notice that I
cannot ping anything while in a shell. I get "ping: icmp open socket:
Operation not permitted". Could this be what's stopping it from working?

Sounds like a firewall issue...

Matt

 

[Tim Shoppa]

My web host has Perl 5.6.1 and LWP::Simple is installed.

Furthermore, the following always results in "Bad.":

######################
#!/usr/bin/perl

use strict;
use LWP::Simple;

my $url = 'http://computercops.biz/ipbans.txt';

head($url) ? print "Good.\n" : print "Bad.\n";
######################

What could be causing this lack of proper behavior?

Knowing the http response code would help... but LWP::Simple doesn't
let you get at that. See the example from LWP::UserAgent where you
can get the response code number which will then help track down the
problem.

I did notice that I
cannot ping anything while in a shell. I get "ping: icmp open socket:
Operation not permitted". Could this be what's stopping it from working?

What OS/Platform?

Tim.

 

[Larry Snider]

Knowing the http response code would help... but LWP::Simple doesn't
let you get at that. See the example from LWP::UserAgent where you
can get the response code number which will then help track down the
problem.

I'll try that. Thx.

What OS/Platform?

Tim.

uname returns Linux running kernel 2.4.20-24.7. I am assuming Red Hat.

 

[Larry Snider]

Sounds like a firewall issue...

Matt

Hmmm... That's not reassuring. Would opening direct sockets be a way
around this? I've never done any socket programming.

 

[pkent]

When I execute the following code on my web host's server, the getstore()
part doesn't do anything and no errors are produced. ....
#!/usr/bin/perl -w

use LWP::Simple;
my $document =
getstore('http://www.computercops.biz/ipbans.txt','ipbans.txt');

You need to read the docs:

getstore($url, $file)
Gets a document identified by a URL and stores it in
the file. The return value is the HTTP response code.

There's also the is_error() routine to explicitly check the return code.
You may also want to check file permissions on 'ipbans.txt'.

You may also find it useful to say:

use LWP:ebug qw(+);

P

 

[Larry]

You need to read the docs:

getstore($url, $file)
Gets a document identified by a URL and stores it in
the file. The return value is the HTTP response code.

I knew that. ;-) I getting an HTTP response of 500.

There's also the is_error() routine to explicitly check the return code.
You may also want to check file permissions on 'ipbans.txt'.

You may also find it useful to say:

use LWP:ebug qw(+);

Here's the output:

LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET http://www.computercops.biz/ipbans.txt
LWP::UserAgent::_need_proxy: Not proxied
LWP:rotocol::http::request: ()
LWP::UserAgent::request: Simple response: Internal Server Error

I'm not sure what this is telling me. Is it saying that since my web host
is not proxied go ahead with the http request?

 

[Larry]

There's also the is_error() routine to explicitly check the return code.
You may also want to check file permissions on 'ipbans.txt'.

I failed to mention that the above code works when I execute it from my home
computer.

 

[Tad McClellan]

I failed to mention that the above code works when I execute it from my home
computer.

Since the code works, you do not have a Perl problem.

Time to start looking for OS or web server problems (in some
other newsgroup, where they discuss such things).

 

[Tim Shoppa]

Since the code works, you do not have a Perl problem.

No, but he does have a problem with LWP::Simple on a machine that's not
under his full control. There are a lot of
"gotchas" hidden under LWP::Simple's hood, and if for example someone
has set his http_proxy environment variable to something inappropriate he'll
get exactly the symptoms being reported.

Admittedly the proxy gotcha is mentioned at the bottom of the LWP::Simple
documentation.

Time to start looking for OS or web server problems (in some
other newsgroup, where they discuss such things).

I do agree it's not our job to find the right http_proxy setting for
everyone's PeeCee. But we should teach them how to read the documentation
and teach how to check for the minutae (like environment variables) that
can interact in unexpected (to use a polite term) ways common Perl modules.

Tim.

 

[Larry]

(e-mail address removed) (Tad McClellan) wrote in message
news:<[email protected]>...

Admittedly the proxy gotcha is mentioned at the bottom of the LWP::Simple
documentation.

In Perldoc? I couldn't find it. To which documentation are you referring?

I do agree it's not our job to find the right http_proxy setting for
everyone's PeeCee. But we should teach them how to read the documentation
and teach how to check for the minutae (like environment variables) that
can interact in unexpected (to use a polite term) ways common Perl
modules.

So, what would be the appropriate approach? Test to see if LWP::Simple's
getstore() works and if not use sockets? As stated before I've never done
socket programming so a little help in this area would be nice.

Does anyone know of some code that could gather a lot of information about
what's going on? I used LWP::Simple qw(+) and got limited information.
Here's the information:

LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET http://www.computercops.biz/ipbans.txt
LWP::UserAgent::_need_proxy: Not proxied
LWP:rotocol::http::request: ()
LWP::UserAgent::request: Simple response: Internal Server Error

I knew about the 500 since that was what getstore was returning. I'm not
sure what this is telling me. Is it saying that since my web host
is not proxied go ahead with the http request?

 

[Larry]

what's going on? I used LWP::Simple qw(+) and got limited information.
Here's the information:

Oops! That should read LWP:ebug qw(+);

 

[Larry]

From LWP::Simple::
The user agent created by this module will identify itself as
``LWP::Simple/#.##'' (where ``#.##'' is the libwww-perl version number)
and will initialize its proxy defaults from the environment (by calling
$ua->env_proxy).

This wording is considered a gotcha? I read it before and didn't get that
out of it.

If you should be using a proxy, but aren't...

So what does LWP:ebug mean when it says, "LWP::UserAgent::_need_proxy: Not
proxied"? Does that mean that I didn't set my proxy variable or that the
server does not require a proxy to be set?

Did you not read my whole post?

 

[Larry]

I'd answer you again, but you obviously didn't read my post (or you're
trying to be a troll). If you can't put two and two together, I'd suggest
posting to k12.ed.math for additional help.

Matt

Matt,

I'm not trying to be rude. It's just that it's frustrating when people
answer your questions by not answering them. Maybe I'm not phrasing my
question properly or in a vernacular that's common to Perl programmers. I
thought that my question above was very on-topic and worded properly but I
may be wrong. If I were in the wrong I apologise if not, then your insult
is not warranted.

 

[Matt Garrish]

In Perldoc? I couldn't find it. To which documentation are you referring?

From LWP::Simple::
The user agent created by this module will identify itself as
``LWP::Simple/#.##'' (where ``#.##'' is the libwww-perl version number) and
will initialize its proxy defaults from the environment (by calling
$ua->env_proxy).

If you should be using a proxy, but aren't...

It may be that your host wasn't expecting anyone to be retrieving pages from
the server (or doesn't want anyone to) and so never set the environment
variables to route requests through the proxy. You'd have to check with your
host to find out if you should be using one, and what settings to use. As
Tad noted, this really isn't something that anyone here can help you with.

Matt

 

[Jay Tilton]

: I'm not trying to be rude. It's just that it's frustrating when people
: answer your questions by not answering them.

The key part of Matt's article was:

this really isn't something that anyone here can help you with

There just aren't many ways to elaborate on that. Nobody here can know
why a specific machine cannot get a successful response from a specific
HTTP server. Only the people running those machines can help now.

: Maybe I'm not phrasing my
: question properly or in a vernacular that's common to Perl programmers.

It's kosher, but there is no answer to give.

: thought that my question above was very on-topic

It once was, but Perl has been eliminated as being the cause of the
problem. More discussion of Perl's involvement will not help to resolve
the problem.

 

[Matt Garrish]

So what does LWP:ebug mean when it says, "LWP::UserAgent::_need_proxy: Not
proxied"? Does that mean that I didn't set my proxy variable or that the
server does not require a proxy to be set?

Did you not read my whole post?

I'd answer you again, but you obviously didn't read my post (or you're
trying to be a troll). If you can't put two and two together, I'd suggest
posting to k12.ed.math for additional help.

Matt

 

[Helgi Briem]

In Perldoc? I couldn't find it. To which documentation are you referring?

The Perl documentation comes packaged with every distribution
of Perl. It can be read, browsed and queried using the perldoc
program, which also comes bundled with Perl.

Run it from the command line. You may find the following pages
useful:

perldoc perl
perldoc perldoc
perldoc perltoc
perldoc LWP::Simple

Set your environment variable http_proxy to the
appropriate value. The appropriate value can be
found in your browser settings and is likely to
be of the form:

http://yourproxy.yourdomain:portnumber

 

[Matt Garrish]

I'm not trying to be rude. It's just that it's frustrating when people
answer your questions by not answering them. Maybe I'm not phrasing my
question properly or in a vernacular that's common to Perl programmers. I
thought that my question above was very on-topic and worded properly but I
may be wrong. If I were in the wrong I apologise if not, then your insult
is not warranted.

Ever heard of DoS attacks?

A couple of steps conscientious sys admins can take to prevent their servers
from being used for (and prone to) such attacks is to restrict ping access
and not allow web access. Looks like you have a conscientious sys admin,
doesn't it?

So, who knows if you have a proxy you can use to get around the default
settings for the server?
a) You
b) Posters to clpm
c) Your sys admin

(I know you can get this one!)

Matt

 

[Larry]

Thanks for the input everyone.

One of the admins says that it's intentionally blocked by Jail Shell. I
didn't confirm that but I did some tests and could perform the function in
a jail shell using sockets, via the web in cgi-bin or just in a cron job,
which is where I was going to execute it anyway.