Looking for browser emulator

[Roy Smith]

I've got to write some tests in python which simulate getting a page of
HTML from an http server, finding a link, clicking on it, and then
examining the HTML on the next page to make sure it has certain features.

I can use urllib to do the basic fetching, and lxml gives me the tools
to find the link I want and extract its href attribute. What's missing
is dealing with turning the href into an absolute URL that I can give to
urlopen(). Browsers implement all sorts of stateful logic such as "if
the URL has no hostname, use the same hostname as the current page".
I'm talking about something where I can execute this sequence of calls:

urlopen("http://foo.com:9999/bar")
urlopen("/baz")

and have the second one know that it needs to get
"http://foo.com:9999/baz". Does anything like that exist?

I'm really trying to stay away from Selenium and go strictly with
something I can run under unittest.

 

[Jon Clements]

I've got to write some tests in python which simulate getting a page of
HTML from an http server, finding a link, clicking on it, and then
examining the HTML on the next page to make sure it has certain features.

I can use urllib to do the basic fetching, and lxml gives me the tools
to find the link I want and extract its href attribute.  What's missing
is dealing with turning the href into an absolute URL that I can give to
urlopen().  Browsers implement all sorts of stateful logic such as "if
the URL has no hostname, use the same hostname as the current page".  
I'm talking about something where I can execute this sequence of calls:

urlopen("http://foo.com:9999/bar")
urlopen("/baz")

and have the second one know that it needs to get
"http://foo.com:9999/baz".  Does anything like that exist?

I'm really trying to stay away from Selenium and go strictly with
something I can run under unittest.

lxml.html.make_links_absolute() ?

 

[Jon Clements]

I've got to write some tests in python which simulate getting a page of
HTML from an http server, finding a link, clicking on it, and then
examining the HTML on the next page to make sure it has certain features.

I can use urllib to do the basic fetching, and lxml gives me the tools
to find the link I want and extract its href attribute.  What's missing
is dealing with turning the href into an absolute URL that I can give to
urlopen().  Browsers implement all sorts of stateful logic such as "if
the URL has no hostname, use the same hostname as the current page".  
I'm talking about something where I can execute this sequence of calls:

urlopen("http://foo.com:9999/bar")
urlopen("/baz")

and have the second one know that it needs to get
"http://foo.com:9999/baz".  Does anything like that exist?

I'm really trying to stay away from Selenium and go strictly with
something I can run under unittest.

lxml.html.make_links_absolute() ?

 

[Roy Smith]

lxml.html.make_links_absolute() ?

Interesting. That might be exactly what I'm looking for. Thanks!

 

[Miki Tebeka]

IIRC mechanize can do that.

 

[Gary Herron]

I've got to write some tests in python which simulate getting a page of
HTML from an http server, finding a link, clicking on it, and then
examining the HTML on the next page to make sure it has certain features.

I can use urllib to do the basic fetching, and lxml gives me the tools
to find the link I want and extract its href attribute. What's missing
is dealing with turning the href into an absolute URL that I can give to
urlopen(). Browsers implement all sorts of stateful logic such as "if
the URL has no hostname, use the same hostname as the current page".
I'm talking about something where I can execute this sequence of calls:

urlopen("http://foo.com:9999/bar")
urlopen("/baz")

and have the second one know that it needs to get
"http://foo.com:9999/baz". Does anything like that exist?

I'm really trying to stay away from Selenium and go strictly with
something I can run under unittest.

Try mechanize
http://wwwsearch.sourceforge.net/mechanize/
billed as
Stateful programmatic web browsing in Python.


I handles clicking on links, cookies, logging in/out, and filling in of
forms in the same way as a "real" browser, but it's all under
programmatic control from Python.


In Ubuntu, it's the python-mechanize package.

 

[Roy Smith]

Try mechanize
http://wwwsearch.sourceforge.net/mechanize/
billed as
Stateful programmatic web browsing in Python.

Wow, this is cool, thanks! It even does cookies!