Simple web proxy - help needed

[Ziv Forshtat]

Hi all,
I'm looking for a simple web proxy in python to use in order to
demonstrate a web page classifier that I have developed.
I checked around the web and tried some of the existing free servers
such as the Tiny proxy server, CTC and others. The problem is that at
least on my winxp machine all of these servers only seem to work with
some sites but consistently generate exceptions with others. I keep
getting "connection reset by peer" and "software caused connection
abort"...
Perahps someone has already encountered this problem and solved it??
Basically, what I need is to plug my web page classifier into a simple
web proxy, so that pages pass through my classifier before they are
relayed to the client.
I really need to get this - or something of the sort - done as part of
a university project. Another option might be in the form of an
internet browser plugin that will communicate with my classifier to do
the job.
Problem is - I don't have to much background in writing proxies, nor
in writing plugins.
Any help and suggestions would be welcome.

Ziv

 

[Chris Liechti]

(e-mail address removed) (Ziv Forshtat) wrote in

The problem is that at
least on my winxp machine all of these servers only seem to work with
some sites but consistently generate exceptions with others. I keep
getting "connection reset by peer" and "software caused connection
abort"...
Perahps someone has already encountered this problem and solved it??

i have a TCP forwarder program that had similar exceptions.
since i chenged it back from sock.sendall(data) to
while data:
n = sock.send(data)
data = data[n:]

it's running stable again. maybe look for sendall's in the sources.

chris

 

[John J. Lee]

Problem is - I don't have to much background in writing proxies, nor
in writing plugins.

[...]

Plugins:

For MSIE, all the docs are on MSDN. Try a Google search for "MSDN
reusing webbrowser control". Get Hammond and Robinson's book on
Python & win32.

For Konqueror, start here:

http://www.boddie.org.uk/david/Projects/Python/KDE/

http://www.boddie.org.uk/david/Projects/Python/KDE/Docs/

Warning: plugin support for PyKDE is quite new.

For Mozilla, MozPython might set you off in the right direction.
Building PyXPCOM is a pain. BTW, Mozilla doesn't seem to want to
build on Debian woody, so don't do that. Read the installation
instructions for the MozPython & PyXPCOM binaries on that page
carefully (so as not to piss off Thomas when you get it wrong ;-)

http://www.thomas-schilz.de/MozPython/README.html


(BTW, you're probably NOT interested in "Netscape plugins")


John

 

[John J. Lee]

Problem is - I don't have to much background in writing proxies, nor
in writing plugins.
Any help and suggestions would be welcome.

Back again... just remembered I posted a tiny MSIE plugin skeleton to
the ctypes list a while back. That might actually be very close to
what you need already. Use the latest version of ctypes.

I originally meant not to post it actually, because of the obvious
applications in malware, then forgot about my decision and posted it
anyway. Oh well, the black hats have plenty of evil code already, I'm
sure.


John

 

[Ziv Forshtat]

Back again... just remembered I posted a tiny MSIE plugin skeleton to
the ctypes list a while back. That might actually be very close to
what you need already. Use the latest version of ctypes.

I originally meant not to post it actually, because of the obvious
applications in malware, then forgot about my decision and posted it
anyway. Oh well, the black hats have plenty of evil code already, I'm
sure.


John

Hi John,
Thanks for your reply - it really seems like this could work for me,
but I was unable to locate the posting that you mention here.
Could you please be so kind as to send me a pointer?

Thanks,
Ziv

 

[Skip Montanaro]

Ziv> Hi all,
Ziv> I'm looking for a simple web proxy in python to use in order to
Ziv> demonstrate a web page classifier that I have developed.

Perhaps <http://theory.stanford.edu/~amitp/proxy.html> can be made to work.
You might have to tweak it a little for WinXP.

Skip

 

[John J. Lee]

(e-mail address removed) (John J. Lee) wrote in message news:<[email protected]>... [...]

Back again... just remembered I posted a tiny MSIE plugin skeleton to
the ctypes list a while back. That might actually be very close to
what you need already. Use the latest version of ctypes.

[...]
Hi John,
Thanks for your reply - it really seems like this could work for me,
but I was unable to locate the posting that you mention here.
Could you please be so kind as to send me a pointer?

Sorry for the long delay...

Here it is:

http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/1967868

(make sure to use the newest release of ctypes: there are several bugs
fixed that would otherwise break this script, including the one
described in that message)


John