urllib2 : https and proxy

[cp.finances.gouv]

Hello all,

I'm facing a strange behavior of urllib2 trying to access gmail
account behind a proxy (Squid).

The following works perfectly :

wget --save-cookies cookies --keep-session-cookies --post-
data="[email protected]&Passwd=cpfinances" --no-
check-certificate -O - https://www.google.com/accounts/ServiceLoginBoxAuth


While the following urllib2 code fail with an error 501 :

In [18]: import urllib2

In [19]: r=urllib2.urlopen("https://www.google.com/accounts/
ServiceLoginBoxAuth",

"[email protected]&Passwd=cpfinances")
---------------------------------------------------------------------------
urllib2.HTTPError Traceback (most
recent call last)

/home/maric/<ipython console>

/home/maric/urllib2.py in urlopen(url, data)
128 if _opener is None:
129 _opener = build_opener()
--> 130 return _opener.open(url, data)

....

/home/maric/urllib2.py in http_error_default(self, req, fp, code, msg,
hdrs)
478 class HTTPDefaultErrorHandler(BaseHandler):
479 def http_error_default(self, req, fp, code, msg, hdrs):
--> 480 raise HTTPError(req.get_full_url(), code, msg, hdrs,
fp)
481
482 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 501: Not Implemented


I understood it's a response from the proxy about incompatible
protocols and queries but this seem rather strange in this case. Could
there be a bug in the ProxyHandler class ? Any idea about what could
cause this error on the proxy ?

 

[O.R.Senthil Kumaran]

Hello all,

I'm facing a strange behavior of urllib2 trying to access gmail
account behind a proxy (Squid).

First off, your password wont work. See if you can get a new one.
Then to setup proxy suppor for urllib2, do the steps:



proxy_url = 'http://' + PROXY_USER + ':' + PROXY_PASSWORD + \
'@' + PROXY_IP + ':' + PROXY_PORT
proxy_support = urllib2.ProxyHandler({'http': proxy_url})
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
urllib2.install_opener(opener)

Substitute PROXY_* with actual values and then do urllib2.urlopen(site) which will go through the proxy.

 

[O.R.Senthil Kumaran]

Hmmm, i've tried something similar but I get the same result.

For the code example I gave, I've put correctly the http_proxy and
https_proxy environment variables, the default ProxyHandler get them right
as this is the proxy itself which returns the error. I checked this with
wireshark, the proxy said that there is incompatibilities in some request
methods, like POST, with some protocols, Gopher is the example it gave.
That doesn't make much sense to me as the wget command I tried is intended
to do exactly the same thing (POST over https and retrieve some cookies from
the response), and go through with no problems.

Maric, in the ActiveState urllib2 HOWTO which seems to apply for Python 2.5, I
came across a point which mentions urllib2 does not support PROXY for HTTPS.
Your case seems the same, I would suggest you to look up the Python Tracker
for any already open issues, or if not kindly log one yourself.
I shall if I can help further, working on urllib2 is a project am doing as
part of Summer of Code 2007.

Thanks,

 

[cp.finances.gouv]

Maric, in the ActiveState urllib2 HOWTO which seems to apply for Python 2.5, I
came across a point which mentions urllib2 does not support PROXY for HTTPS.

Yes, I realize this too, I think this should be considered as a bug in
the documentation.

Your case seems the same, I would suggest you to look up the Python Tracker
for any already open issues, or if not kindly log one yourself.

Didn't find one when i made a quick search, but i would suggest to
submit a documentation bug.

BTW, I find this recipe on python cookbook,
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195,
works fine for me (with the two additionals comments in my case). We
could add a pointer to this in the doc before fixing urllib2.

What would you suggest ?