urllib(2) and https blues? try pytunnel for python tunnelling

[john]

The python libraries like urllib and httplib do not support ssl
through a proxy.
Urllib2 supports http through a proxy or https alone, but not https
through a proxy.

A while ago my wife complained to me that perl had trouble w/https. I
was surprised since perl has a lot of networking libraries. Preferring
python, I took a look a python and found the same problem. Eventually
I figured out what python was missing: tunnelling. Though she may
_coax_ me to port the code to perl, I'm writing it first in my
favorite language.

Pytunnel offers a tunnel that can be used for (among other things)
tunnelling the ssl through a proxy.

I'd be willing to offer suggestions or myself add the necessary code
to urllib2. Anyone know how one goes about doing either?

Your python code does not need to be changed except that the port and
ip are supplied by the tunnel. Pytunnel uses a custom recvall which is
like sendall but for recv. Recvall uses non-blocking sockets,timeouts
and sleeps to attempt to get all of the data. If you have a bad
network connection you'd want to set the timeout to be high. Recvall
will probably be slower than a standard recvall.

The code is rough and needs better error handling, comments and such.
But since I got it working I decided to post it.

You can find it at:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/213238
Example of usage:

import pytunnel,httplib

def tunnel_this(ip,port):
conn = httplib.HTTPSConnection(ip,port=port)
conn.putrequest('GET', '/')
conn.endheaders()
response = conn.getresponse()
print response.read()

tunnel=pytunnel.build(host='login.yahoo.com',proxy_host='h1',proxy_user='u',proxy_pass='p')
tunnel.run(tunnel_this)

 

[Ng Pheng Siong]

The python libraries like urllib and httplib do not support ssl
through a proxy.
Urllib2 supports http through a proxy or https alone, but not https
through a proxy.

M2Crypto's contrib/ contains an isaac.httpslib.py, with the following
blurb:

This is Isaac Salzberg's application of Mihai Ibanescu's patch
(available on SF) that allows HTTPS tunneling through an
authenticating proxy.

This one's a double whammy: it works with IIS through the
authenticating proxy, whereas the one on SF, which uses Python's
built-in SSL, doesn't.

This code is not folded into the main distribution because:

1. Apparently Mihai is still working on it.
2. Mihai uses Python's built-in SSL. Isaac patched it to use
M2Crypto.SSL. The stuff is essentially #ifdef'ed code.
3. I don't have an authenticating proxy nor an IIS server to test
against, so I can't clean up the code. Volunteers welcome. ;-)

Thanks Isaac.