Automatically populate and submit HTML Forms

[rbt]

How can I make a python client script connect to a Web server and
automatically populate form fields and then submit the form?

For example, say I wanted to check and see if 1924 was a leap year...
how would I populate the 'year' field and then submit it for processing?
I have no control of the server... just the client script.

<form action="cgi-bin/leapyear.py">
<p>Enter a year and find out if it's a leap year:
<input type="text" name="year" size="6">
<input type="submit">
<input type="reset">
</form>

Many thanks,

rbt

 

[alex23]

Hey rbt,

You should take a look at mechanize:
http://wwwsearch.sourceforge.net/mechanize/

One of its listed features is 'Easy HTML form filling, using ClientForm
interface'.

A more recent option which came up on this group a few days ago is
twill: http://www.idyll.org/~t/www-tools/twill.html

twill combines pyparsing & mechanize to provide a way of scripting
tests for web applications. While it might not offer the functionality
you're after (I haven't had call to use either of these), it looks very
accessible, and could provide some good examples for using mechanize
with your own scripts.

-alex23

 

[erinhouston]

These two blog entries might be of help to you.

http://www.ishpeck.net/index.php?P=b1115239318ishpeck
second half is at
http://www.ishpeck.net/index.php?P=b1115225809ishpeck

 

[Grant Edwards]

How can I make a python client script connect to a Web server and
automatically populate form fields and then submit the form?

For example, say I wanted to check and see if 1924 was a leap year...
how would I populate the 'year' field and then submit it for processing?
I have no control of the server... just the client script.

<form action="cgi-bin/leapyear.py">
<p>Enter a year and find out if it's a leap year:
<input type="text" name="year" size="6">
<input type="submit">
<input type="reset">
</form>

Just use urllib() and pass the form data to the urlopen()
method. If given data, it will generate a "POST" request
instead of a "GET". Here's a snippet of code from an app of
mine that "fills in a form" and submits it:

postData = urllib.urlencode({'submit':'Remove','disp':'M','action':'change_Msgs'})

for msgid in msgIDs:
postData += "&msgid="+msgid

req2 = urllib2.Request("http://mc-s6.postini.com/exec/MsgCtr",postData)
rsp2 = ClientCookie.urlopen(req2)

In this code I've eyeballed the form and the field names are
hard-wired into the code. If your form doesn't change from one
usage to the next, that's the simplest way to do it.

In my example I'm using ClientCookie and urllib2 to create/open
the reqeust in two steps because the request seen above won't
work without some cookie values previsouly established in code
that I've snipped. Otherwise all you'd need to do is something
like this:

urllib.urlopen('http://whatever',
urllib.urlencode({'field1Name':'value1','field2Name':'value2'}))

 

[Grant Edwards]

Just use urllib() and pass the form data to the urlopen()
method. If given data, it will generate a "POST" request
instead of a "GET".

Oops, I forgot. Here's a pointer to the examples in the urllib
docs:

http://docs.python.org/lib/node478.html

 

[rbt]

Just use urllib() and pass the form data to the urlopen()
method. If given data, it will generate a "POST" request
instead of a "GET". Here's a snippet of code from an app of
mine that "fills in a form" and submits it:

postData = urllib.urlencode({'submit':'Remove','disp':'M','action':'change_Msgs'})

for msgid in msgIDs:
postData += "&msgid="+msgid

req2 = urllib2.Request("http://mc-s6.postini.com/exec/MsgCtr",postData)
rsp2 = ClientCookie.urlopen(req2)

In this code I've eyeballed the form and the field names are
hard-wired into the code. If your form doesn't change from one
usage to the next, that's the simplest way to do it.

In my example I'm using ClientCookie and urllib2 to create/open
the reqeust in two steps because the request seen above won't
work without some cookie values previsouly established in code
that I've snipped. Otherwise all you'd need to do is something
like this:

urllib.urlopen('http://whatever',
urllib.urlencode({'field1Name':'value1','field2Name':'value2'}))

Thanks Grant... I found ClientCookie and ClientForm here:

http://wwwsearch.sourceforge.net/

Two great Python modules that really simplify this!!!