urlib - automatic cookie handling

[Alex Hunsley]

I'm using urllib to post data to a web form by issuing a command similar to this:


filename, headers =
urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.XmlFeed",
"content.txt", None, urllib.urlencode({"aParameter": "theValue"}))

Now, the problem is that the above fails, since I am not sending a session
cookie. Visitors to the web sites' html submission form are sent a session
cookie which is given back to the server when they submit a search via the
browser, as often happens.
Now, I could use urllib to get the form page and read the cookie from the
headers that are returned and then manually put that cookie in my submission to
the servlet, but my question is: is there a way to tell urllib or some other
part of HTTP handling in python that I want to remember any cookie that is
given to me, and give it back to that site if I send requests later on?

thanks
alex

 

[=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=]

Yes.

First fetch the URL which gives you the cookie. Parse the HTTP headers to
get the cookie (use the header parsing function and get the Set-Cookie
header). Then send the cookie in your headers along your next request, and
you're in. For this you must use urllib2 which allows custom headers (use
request.headers or something).

 

[Alex Hunsley]

Yes.

First fetch the URL which gives you the cookie. Parse the HTTP
headers to get the cookie (use the header parsing function and get the
Set-Cookie header). Then send the cookie in your headers along your
next request, and you're in. For this you must use urllib2 which allows
custom headers (use request.headers or something).

thanks for your response!
An even better answer, which I've just found:
http://wwwsearch.sourceforge.net/ClientCookie

Can I ask that you please don't top-post? It makes threads hard to follow...

thanks
alex

 

[Michael Geary]

I'm using urllib to post data to a web form by issuing a command similar
to this:

filename, headers =
urllib.urlretrieve("http://www.thewebsitenamehere.com/servlet/com.blah.bloo.
XmlFeed",
"content.txt", None, urllib.urlencode({"aParameter": "theValue"}))

Now, the problem is that the above fails, since I am not sending a session
cookie. Visitors to the web sites' html submission form are sent a session
cookie which is given back to the server when they submit a search via the
browser, as often happens.
Now, I could use urllib to get the form page and read the cookie from the
headers that are returned and then manually put that cookie in my
submission to the servlet, but my question is: is there a way to tell urllib
or some other part of HTTP handling in python that I want to remember
any cookie that is given to me, and give it back to that site if I send
requests later on?

You're looking for ClientCookie:

http://wwwsearch.sourceforge.net/ClientCookie/

http://wwwsearch.sourceforge.net/ClientCookie/doc.html

-Mike

 

[Alex Hunsley]

You're looking for ClientCookie:

http://wwwsearch.sourceforge.net/ClientCookie/

http://wwwsearch.sourceforge.net/ClientCookie/doc.html

-Mike

hi there!
Snap! I think we both posted at about the same time. Good to know someone was
suggesting it anyway!

thanks
alex

 

[John J. Lee]

http://wwwsearch.sourceforge.net/ClientCookie/

[...]
Snap! I think we both posted at about the same time. Good to know
someone was suggesting it anyway!

You might care to know that the cookie-handling part of ClientCookie
has found its way into Python 2.4, as module cookielib. So, the
development version of CC (which is compatible with old versions of
Python) is currently moving towards being as-compatible-as-possible
with cookielib and 2.4 urllib2.

Actually, if you're using Python 2.3 and want to be
forwards-compatible with 2.4 here, I'm 95% sure you can just grab
cookielib and urllib2 from Python CVS and use those. cookielib
doesn't import urllib2, so you can leave 2.3's urllib2 intact and use
a local copy of 2.4's urllib2, eg. by keeping it somewhere like
~/lib/python and renaming it urllib2_24.py. cookielib doesn't include
stuff like HTTPEquivProcessor that are in ClientCookie, but I'll make
those available in a form that works with 2.4 in a separate package
soon (that package will probably be ClientCookie itself, in a 1.0.x or
late 0.9.x version).

I should mention these things on the web page...


John