https with LWP::UserAgent

[one man army]

Hi All-

I am an infrequent user of Perl, but have succeeded in a few endeavors.
I have a script that reads data from a series of web pages. But now I
want to modify it to handle an HTTPS page.

I looked briefly in UserAgent.pm, most of the code is beyond my skill
level. I found the lines:

use Carp ();

if ($ENV{PERL_LWP_USE_HTTP_10}) {
require LWP:rotocol::http10;
LWP:rotocol::implementor('http', 'LWP:rotocol::http10');
eval {
require LWP:rotocol::https10;
LWP:rotocol::implementor('https', 'LWP:rotocol::https10');
};
}

And there is an option described, $ua->protocols_allowed(). So I added
the following line after init'ing my $ua

$ua->protocols_allowed( [ 'http', 'https'] ); # to set


Alas, still an error connecting to the page. The very same urls pasted
in to Firefox result in no error.

What to do?

thanks much

 

[Richard Gration]

Hi All-

I am an infrequent user of Perl, but have succeeded in a few endeavors.
I have a script that reads data from a series of web pages. But now I
want to modify it to handle an HTTPS page.

I looked briefly in UserAgent.pm

That way lies pain and death. Well, pain anyway. You haven't specified
your OS, so I'll say how it is for mine, Linux.

Using LWP with the https protocol is as simple as

my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => 'https://secure.tld');
my $res = $ua->request($req);

The key on *nix is to have some SSL libraries installed to do the hard
work. LWP itself doesn't have any SSL code.

If LWP finds a 3rd party SSL library, such as openssl, when it is
installed then it will be capable of fetching from https URLs. If not,
then you will see an error like 'unknown protocol' or similar.

This could well be different on winders et al.

HTH
Rich

 

[one man army]

Aha! I am using Perl 5.8.1 on the FreeBSD derivative, Mac OS X/Darwin.
How do I install the SSL Library? I have a folder called /Library/Perl,
then :
5.8.1
File
lwptut.pod
lwpcook.pod
Regexp
Net
LWP.pm
LWP
WWW
URI.pm
URI
Bundle
HTTP
HTML
darwin-thread-multi-2level
HTML-LinkExtractor-0.13
HTML-Tagset-3.04
HTML-Parser-3.45
URIC-2.02
libwww-perl-5.69
Regexp-Common-2.120
libnet-1.19
URI-1.35

 

[Richard Gration]

Aha! I am using Perl 5.8.1 on the FreeBSD derivative, Mac OS X/Darwin.
How do I install the SSL Library? I have a folder called /Library/Perl,
then :
5.8.1
File
lwptut.pod
lwpcook.pod
Regexp
Net
LWP.pm
LWP
WWW
URI.pm
URI
Bundle
HTTP
HTML
darwin-thread-multi-2level
HTML-LinkExtractor-0.13
HTML-Tagset-3.04
HTML-Parser-3.45
URIC-2.02
libwww-perl-5.69
Regexp-Common-2.120
libnet-1.19
URI-1.35

[ Please don't top post ]

SSL isn't part of perl. It's a thing in and of itself. I know next to
nothing about Macs so I can't tell you specifically how to do what you
need to but I would:

1. Find out if you have SSL libraries. On my system I have
/usr/lib/libssl.*

2. If not, then install openssl or equivalent.

3. Once you have ssl on your machine, the most likely point of failure is
that libwww-perl doesn't know it's there. Once you are certain you have an
ssl implementation, reinstall libwww-perl. Make sure that it notices ssl
during configuration / installation. You might have to give some options
when configuring to specify the location of your ssl.

If none of this helps, then I'm afraid I'm at a bit of a loss.

Rich

 

[one man army]

I found Crypy::SSLeay on CPAN. I brought it down, and in the readme it
says that you must have OpenSSL installed before compiling/building.

I searched for openSSL (case-insensitive) anywhere on my system, but
found nothing.

How to install OpenSSL on MacOS 10.3? thanks

 

[Richard Gration]

How to install OpenSSL on MacOS 10.3? thanks

This thread has now lost even a tenuous connection to perl, I'm afraid ;-)

You should try asking in a MacOS newsgroup.

Cheers
Rich

 

[Sherm Pendley]

I searched for openSSL (case-insensitive) anywhere on my system, but
found nothing.

How to install OpenSSL on MacOS 10.3?

You don't need to - it's included with the OS. Honestly, couldn't you have at
least *tried* to install the module before assuming it would fail?

sherm--

 

[Sherm Pendley]

Aha! I am using Perl 5.8.1 on the FreeBSD derivative, Mac OS X/Darwin.
How do I install the SSL Library?

Same way you install it on any other *nix, by following the instructions found
in "perldoc perlmodinstall".

I'm not being snide - the instructions there are far better and more detailed
than any usnet posting could be. Be sure to follow the Unix instructions - the
Mac instructions are for MacPerl running on MacOS 9 and earlier.

sherm--

 

[Babacio]

I found Crypy::SSLeay on CPAN. I brought it down, and in the readme it
says that you must have OpenSSL installed before compiling/building.

I searched for openSSL (case-insensitive) anywhere on my system, but
found nothing.

How to install OpenSSL on MacOS 10.3? thanks

Have a look on darwinports. There is a port 'openssl' which sure gives
you all what you need. There is even a package named p5-net-ssleay
which installs Net::SSLeay, which may be of interest.

 

[one man army]

SUMMARY:
With the addition of Perl Module Crypt-SSLeay-0.51, LWP will use
OpenSSL to provide HTTPS (perl 5.8 / Mac OS X Darwin).

OpenSSL is built in to Mac OS X. The Finder's find does not look in
system libraries by default, so it will not find OpenSSL. /usr/lib has a
few versions of ssl with various encoded names.

I built Crypt::SSLeay with only a minor error (the man files don't get
put into a place the make expects)

I rebuilt libwww-perl just in case it needs to know SSLeay is present.
The makefile seems confused about the Mac's .DS_Store files and
generates a hard error. But the library runs and code that uses it runs.

<someone should probably add a check for .DS_Store files in the make,
since there are so many Darwin machines out there now>

All in all, CPAN is the best feature of the Perl library system IMHO.
The system of pieces being built is not yet as bullet-proof as it could
be, since so many smart people use this stuff all the time.

thanks for <most of> the replies

 

[Babacio]

one man army.

I rebuilt libwww-perl just in case it needs to know SSLeay is present.
The makefile seems confused about the Mac's .DS_Store files and
generates a hard error. But the library runs and code that uses it runs.

<someone should probably add a check for .DS_Store files in the make,
since there are so many Darwin machines out there now>

Well, one of my favorite things is
find . -name '.DS_Store' -exec rm -i {} \;
....