Writing web bots in python

[sudharsh]

Hello,
I hav a question..How do I write a webbot that logs onto some website,
fills text into a textbox and submit that form, Sorry I am a novice in
python, apparently I have to use urllib, but I hav a few queries on
this, What happens when there are more input interfaces..does urllib
make somekind of a template when it does so..need more info on
this..links and tutorilas would be appreciated..thanx

Sudharshan S

 

[Fuzzyman]

Hello,
I hav a question..How do I write a webbot that logs onto some website,
fills text into a textbox and submit that form, Sorry I am a novice in
python, apparently I have to use urllib, but I hav a few queries on
this, What happens when there are more input interfaces..does urllib
make somekind of a template when it does so..need more info on
this..links and tutorilas would be appreciated..thanx

You should use urllib2 rather than urllib. You can see tutorials on
urllib2, and handling cookies if you need to at :

http://www.voidspace.org.uk/python/articles/urllib2.shtml
http://www.voidspace.org.uk/python/articles/cookielib.shtml

*However*, this only handles fetching the web pages and posting data
back. You need something that will understand the web 'form' and fill
it in for you.

You probably want Mechanize *or* something like FormEncode. Google for
these.

All the best,

Fuzzyman

 

[sudharsh]

Thanx for the info..i'll let you know when it works

 

[Paul McGuire]

Hello,
I hav a question..How do I write a webbot that logs onto some website,
fills text into a textbox and submit that form, Sorry I am a novice in
python, apparently I have to use urllib, but I hav a few queries on
this, What happens when there are more input interfaces..does urllib
make somekind of a template when it does so..need more info on
this..links and tutorilas would be appreciated..thanx

Sudharshan S

C.Titus Brown gave a quick but impressive demo of twill at the Pycon
lightning talks. I think this will do just what you need.
http://www.idyll.org/~t/www-tools/twill/

-- Paul

 

[Ben C]

Hello,
I hav a question..How do I write a webbot that logs onto some website,
fills text into a textbox and submit that form, Sorry I am a novice in
python, apparently I have to use urllib, but I hav a few queries on
this, What happens when there are more input interfaces..does urllib
make somekind of a template when it does so..need more info on
this..links and tutorilas would be appreciated..thanx

Not strictly Python related, but if you use Firefox, get the "Tamper
Data" extension for it. This lets you see exactly what Firefox is
submitting in the formdata, and what the urls are of the cgi scripts
etc.

I usually invoke a program called curl from python with os.popen, but
this twill someone has suggested looks interesting.