Perl code to fill-in online form

[Babul]

Hello everyone,

I am trying to write a program in perl (windows) that I will use to
fill out online form.

Here are the steps that I follow to get the online form:
1. I go to the web site first and get a Login form
2. I fill in the Login form and submit that form. Then a page open
with a link "Choose a period". When I click on that link, a small
window pops up with links for different weeks. I click on my desired
week.
3. Now a page display with some project names, dates of my choosen
week and box field to enter time that I spent on that progect.

I tried WWW::Mechanize to get the contents of the web page.

Here are the steps I followed to grab the contents of the web page:
#! /usr/local/bin/perl

use Tk;
use Cwd;
use WWW::Mechanize;
$currentDir = getcwd;

$mech = WWW::Mechanize->new();
$mech->get("web address from the 3rd step above");
$contents = $mech->content();
open WF, ">$currentDir/temp.txt";
print WF "$contents\n";
close WF;

But the contents I get is the source code of the Login page (from step
1 above). My purpose is to get the source code of the web page which
is displayed in step 3.

If any one can help me I would appreciate that.

Note: I also noticed that If I go to View->Source on the Internet
Explorer to view the HTML source code of that web page, it does not
display. But I can view source code using View->Source for any other
web sites.

Regards,

Babul

 

[Joost Diepenmaat]

Hello everyone,
hi!

I am trying to write a program in perl (windows) that I will use to
fill out online form.

2. I fill in the Login form and submit that form. Then a page open
with a link "Choose a period". When I click on that link, a small
window pops up with links for different weeks. I click on my desired
week.
ok

I tried WWW::Mechanize to get the contents of the web page.

That's usually the best module to use for this, yes. but it sounds like
the popup is opened via javascript and WWW::Mechanize doesn't support
javascript.

You either have to figure out how to get at the popup's url via
WWW::Mechanize (probably using some hand-written rules), and/or you'll
need to "mechanize" a javascript capable browser. Since you're on
windows, you may want to try Win32::IE::Mechanize.

See <http://search.cpan.org/~abeltje/Win32-IE-Mechanize-0.009/lib/Win32/IE/Mechanize.pm>

I think this is all mentioned in the docs for WWW::Mechanize too.

[ ...]

open WF, ">$currentDir/temp.txt";

You don't need to specify the current directory. That's what a current
directory is for.

 

[Ben Bullock]

Hello everyone,

I am trying to write a program in perl (windows) that I will use to fill
out online form.

Here are the steps that I follow to get the online form: 1. I go to the
web site first and get a Login form 2. I fill in the Login form and
submit that form. Then a page open with a link "Choose a period". When I
click on that link, a small window pops up with links for different
weeks. I click on my desired week.
3. Now a page display with some project names, dates of my choosen week
and box field to enter time that I spent on that progect.

I tried WWW::Mechanize to get the contents of the web page.

Here are the steps I followed to grab the contents of the web page: #!
/usr/local/bin/perl

use Tk;
use Cwd;
use WWW::Mechanize;
$currentDir = getcwd;

$mech = WWW::Mechanize->new();
$mech->get("web address from the 3rd step above"); $contents =
$mech->content();
open WF, ">$currentDir/temp.txt";
print WF "$contents\n";
close WF;

But the contents I get is the source code of the Login page (from step 1
above). My purpose is to get the source code of the web page which is
displayed in step 3.

If any one can help me I would appreciate that.

You probably need to actually log in via Perl and get the cookies (login
cookie, session cookie) via Perl. Even if your browser is logged in, Perl
isn't able to share the cookies from the browser to access the web page.

Here are some code fragments from a similar automated login which might
help get you started:

# modules I used

use LWP::UserAgent;
use HTML::Form;
use HTTP::Cookies;

# from the generic user agent starter for my login:

sub my_module::start_user_agent
{
my $ua = LWP::UserAgent->new;

$ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt",
autosave => 1));

return $ua;
}

# from the automated login script:

my $name = $form->find_input ("Name", "text")->value;
my $remember = $form->find_input ("Remember", "checkbox")->value;
my $password;# = $form->find_input ("Password", "text")->value;

$form->value ("Name", "myusername");
$form->value ("Remember", "1");
$form->value ("Password", "mypassword");

$request = $form->click;
$response = $ua->request($request);

# end

I ran the login script as a different script since once I had the cookies
I didn't need to run it again.

Alternatively, if you want to automatically fill in the form from your
browser, don't use Perl, use greasemonkey instead.

 

[Ben Bullock]

That's usually the best module to use for this, yes. but it sounds like
the popup is opened via javascript and WWW::Mechanize doesn't support
javascript.

It's possible but highly unlikely. I think you didn't read where the
poster said that he keeps getting the source code for the login window
when he tries to access the page he wants. The problem is lack of
authentication. If he has the URL for the popup window, window 3, which
he said that he does:

$mech->get("web address from the 3rd step above");

it doesn't matter how window 2 opens window 3. Being opened with
JavaScript has nothing to do with it.

You either have to figure out how to get at the popup's url via
WWW::Mechanize (probably using some hand-written rules), and/or you'll
need to "mechanize" a javascript capable browser. Since you're on
windows, you may want to try Win32::IE::Mechanize.

No, no, no, this is really bad advice. You didn't read the poster's
question properly.

 

[Joost Diepenmaat]

It's possible but highly unlikely. I think you didn't read where the
poster said that he keeps getting the source code for the login window
when he tries to access the page he wants. The problem is lack of
authentication. If he has the URL for the popup window, window 3, which
he said that he does:

$mech->get("web address from the 3rd step above");

it doesn't matter how window 2 opens window 3. Being opened with
JavaScript has nothing to do with it.

Although you state your case a bit confusingly, after re-reading the
original post I agree that my interpretation is probably wrong and yours
is probably correct.

No, no, no, this is really bad advice. You didn't read the poster's
question properly.

I do wonder why this is bad advice. AFAIK it should "work" either way.

 

[Ben Bullock]

I do wonder why this is bad advice. AFAIK it should "work" either way.

I'm sorry to have written like that. I had this vision of the original
poster spending hours and hours trying to do something which probably
wouldn't have solved his problem.

 

[Babul]

It's possible but highly unlikely. I think you didn't read where the
poster said that he keeps getting the source code for the login window
when he tries to access the page he wants. The problem is lack of
authentication. If he has the URL for the popup window, window 3, which
he said that he does:

$mech->get("web address from the 3rd step above");

it doesn't matter how window 2 opens window 3. Being opened with
JavaScript has nothing to do with it.


No, no, no, this is really bad advice. You didn't read the poster's
question properly.- Hide quoted text -

- Show quoted text -

Thanks everyone for the guidance and useful thought.

The reason I am using Perl, I already developed a program in Perl to
keep track of my time spend on any project. The program is working
fine and basically it is create a pipe delimited text file with the
following information:
Project name|Task name|Date|Amount of time spent

Now my intention is to add a button in that program so that when I
will click that button it will grab the information from the pipe
delimited file and will fill-out the online form.

 

[J. Gleixner]

Babul wrote:
[...]

The reason I am using Perl, I already developed a program in Perl to
keep track of my time spend on any project. The program is working
fine and basically it is create a pipe delimited text file with the
following information:
Project name|Task name|Date|Amount of time spent

Now my intention is to add a button in that program so that when I
will click that button it will grab the information from the pipe
delimited file and will fill-out the online form.

Thanks for sharing your intentions. If you have questions, then post
your code and ask, otherwise no one really needs to know your
intentions.

 

[ccc31807]

Hello everyone,

I am trying to write a program in perl (windows) that I will use to
fill out online form.

I assume that since you can write scripts on the server that you have
access to the httpd. This is how I would do it.

1. Configure your httpd for cgi. Then, write a test script to make
sure it works. You can configure your web server a couple of different
ways, so pick the one you like.

2. On the client side, make your html form the way you normally do.
<html><body>
<form action='post' method='/myurl/cgi-bin/form.cgi'>
<input type='text' name='week' />
<input type='submit' value='OK' />
</form>
</body></html>

3. On the server side use CGI.
#!/usr/bin/perl;
use CGI;
use strict;
my $week = param('week');
print "Content-type: text/html\n\n";
print "<html><body>My week is $week</body></html>
exit();

CC

 

[Babul]

Babulwrote:

[...]

The reason I am using Perl, I already developed a program in Perl to
keep track of my time spend on any project. The program is working
fine and basically it is create a pipe delimited text file with the
following information:
Project name|Task name|Date|Amount of time spent

Now my intention is to add a button in that program so that when I
will click that button it will grab the information from the pipe
delimited file and will fill-out the online form.

Thanks for sharing your intentions.  If you have questions, then post
your code and ask, otherwise no one really needs to know your
intentions.

If you think that you are really helpful, please see my code at the
beginning of the discussion. Ben Bullock suggested me to use
greasemonkey, so I explained here why I am using Perl.

 

[Peter J. Holzer]

Babulwrote:

The reason I am using Perl,

[...]

Thanks for sharing your intentions.  If you have questions, then post
your code and ask, otherwise no one really needs to know your
intentions.

If you think that you are really helpful, please see my code at the
beginning of the discussion. Ben Bullock suggested me to use
greasemonkey, so I explained here why I am using Perl.

Be more careful in quoting, then. You quoted a lot from Ben's reply but
you didn't quote the part about gresemonkey. Please quote the part you
are replying to (and only that) - otherwise nobody knows what you are
talking about.

The part which you did quote however, did probably identify your
problem. Did you try his advice?

hp