read a webpage with username/password

[jiehuang001]

I am trying to use perl to extract the content of a webpage, such as an
online newspaper. For example, I might want to have the program to send
me any material with the word of "computer".

I know that I can use Get("url") > output.txt to do that. But the
problem is that to reach to the "url", it needs a username/password to
login first. Of course, I know the username/password. My question is,
how to pass the username/password from perl?

Thank you very much!

JH

 

[Paul Lalli]

I am trying to use perl to extract the content of a webpage, such as an
online newspaper. For example, I might want to have the program to send
me any material with the word of "computer".

I know that I can use Get("url") > output.txt to do that

Assuming you mean the get() method of the LWP::Simple module, first
note that case matters. Second note that LWP::Simple also provides
getstore() which puts the output in a file for you, rather than messing
with shell redirection.

. But the
problem is that to reach to the "url", it needs a username/password to
login first. Of course, I know the username/password. My question is,
how to pass the username/password from perl?

I don't think you can with LWP::Simple. However, the lwpcook
documentation shows this example, using LWP::UserAgent ...
http://search.cpan.org/~gaas/libwww-perl-5.805/lwpcook.pod#ACCESS_TO_PROTECTED_DOCUMENTS

Does that help?

Paul Lalli

 

[jiehuang001]

Hi, Paul:

Thank you very much. But as you will see, the following code does not
work. Any help would be deeply appreciated.

#! /Perl/bin/perl

use LWP;

my $url = 'http://www.dpgn.com/resultDetail.cfm?resultID=4543';
my $ua = new LWP::UserAgent;
my $req = new HTTP::Request (GET => $url);

$req->authorization_basic ('jieacs', 'cancer');
my $request = $ua->request ($req);
die unless $request->code eq '200';
my $page = $request->content;
print $page;

 

[David Dorward]

I know that I can use Get("url") > output.txt to do that. But the
problem is that to reach to the "url", it needs a username/password to
login first. Of course, I know the username/password. My question is,
how to pass the username/password from perl?

Depends how the server expects the username and password to be presented. It
could be as post data, or cookies, or http basic auth or ...

 

[Ben Morrow]

Quoth (e-mail address removed):

$req->authorization_basic (<username/pwd snipped>);

Go and change your password. Now.
Don't post passwords in public forums.

Ben

 

[Jahagirdar Vijayvithal S]

I am trying to use perl to extract the content of a webpage, such as an
online newspaper. For example, I might want to have the program to send
me any material with the word of "computer".

I know that I can use Get("url") > output.txt to do that. But the
problem is that to reach to the "url", it needs a username/password to
login first. Of course, I know the username/password. My question is,
how to pass the username/password from perl?

Thank you very much!

JH

try WWW:Mechanize
Quoting From http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize.pm
{QUOTE}
WWW::Mechanize, or Mech for short, helps you automate interaction with a
website. It supports performing a sequence of page fetches including
following links and submitting forms. Each fetched page is parsed and
its links and forms are extracted. A link or a form can be selected,
form fields can be filled and the next page can be fetched. Mech also
stores a history of the URLs you've visited, which can be queried and
revisited.
{/QUOTE}

Regards
Jahagirdar Vijayvithal S

 

[Paul Lalli]

Hi, Paul:

Thank you very much.

You're welcome. For what, may I ask? Please quote context when
posting a reply.

But as you will see, the following code does not
work.

The code works perfectly for what it was designed to do. The problem
is that you incorrectly stated your problem description.

Any help would be deeply appreciated.

You incorrectly stated your problem description. You said that you had
a URL, but that a user/password was required to access that URL. That
turns out to not be the case. Instead, you have a URL whose page
contains a form. That form has user and password fields. Submission
of that form will take you to the actual URL you wanted to go to.

Use WWW::Mechanize.

Paul Lalli