Urllib2 / add_password method

[mrstephengross]

I'm working on learning how to use urllib2 to use a proxy server. I've
looked through the postings on this group, and it's been helpful. I
have not, however, found complete documentation on the add_password()
functions. Here's what I've got so far:

#========================================
import urllib2
url = 'http://some.server/form.php'

proxy_url = 'http://202.62.252.3:8080' # A publicly available anonymous
proxy server

proxy_handler = urllib2.ProxyHandler( {'http': proxy_url } )

proxy_auth_handler = urllib2.HTTPBasicAuthHandler()
proxy_auth_handler.add_password('realm', 'host', 'username',
'password')

opener = urllib2.build_opener(proxy_handler, proxy_auth_handler)

urllib2.install_opener(opener)

try:
handler = urllib2.urlopen(url)
except urllib2.URLError:
print "whoops!"
else:
print handler.read()
#=======================================

It works, but I don't really know what I'm doing with the
proxy_auth_handler part. Specifying username and password make sense,
but I haven't found documentation on what 'realm' and 'host' are for.
Shouldn't username & password be sufficient?

Thanks,
--Steve ([email protected])

 

[John J. Lee]

I'm working on learning how to use urllib2 to use a proxy server. I've
looked through the postings on this group, and it's been helpful. I
have not, however, found complete documentation on the add_password()
functions. Here's what I've got so far:

Don't use a password manager with proxy auth in 2.4, it doesn't work:

http://python.org/sf/1470846

[...]

It works, but I don't really know what I'm doing with the
proxy_auth_handler part. Specifying username and password make sense,
but I haven't found documentation on what 'realm' and 'host' are for.
Shouldn't username & password be sufficient?

Both proxy and auth handling in urllib2 are buggy in 2.4, but passing
the username and password in the "userinfo" component of the URL works
(didn't read your code, hope nothing important in there):

import urllib2

proxy_handler = urllib2.ProxyHandler({
"http": "http://john:[email protected]:3128"})
opener = urllib2.build_opener(proxy_handler)

response = opener.open(url)


However, ISTR specifying a port number as I do above (the ":3128" bit)
is not supported in 2.4 (it does in current SVN, so will in 2.5).


Hmm, I notice just now that urllib2's digest auth support breaks
urllib2's handler scheme. Must file another bug...


John

 

[John J Lee]

I'm working on learning how to use urllib2 to use a proxy server. I've
looked through the postings on this group, and it's been helpful. I
have not, however, found complete documentation on the add_password()
functions. Here's what I've got so far:

Don't use a password manager with proxy auth in 2.4, it doesn't work:

http://python.org/sf/1470846

[...]

It works, but I don't really know what I'm doing with the
proxy_auth_handler part. Specifying username and password make sense,
but I haven't found documentation on what 'realm' and 'host' are for.
Shouldn't username & password be sufficient?

Oops, didn't read your question.

I'm surprised if it worked at all given that bug (maybe you're using an
older Python, and I forget all the details of the bug).

Anyway, to answer your question: yes, username & password *should* be
sufficient, but the password manager classes in urllib2 at present aren't
very friendly like that. You can use HTTPPasswordMgrWithDefaultRealm so
that at least you can forget about the realm part. (The "realm" is what
you see in the title bar of the pop-up that web browsers display. See RFC
2617 for more details.)

I'm adding friendlier proxy/auth support to package mechanize ATM (and
finding/fixing bugs as I go), so maybe some of that will find its way into
Python 2.6.


John