Proxy in perl

[secSwami]

Hi,

I am trying to write up a proxy client in perl. I have finished part
of it where the script listens on port 7070 and when you configure
your browser to use that port, it will get the webpage just fine but I
see that if the page has calls out to other servers for images etc,
the request half heartedly fails and I just get the web page minus the
content that is hosted on some other website. Can someone shine some
light if and how I can have the proxy app fetch those urls too?

Thanks

 

[Tim Greer]

Hi,

I am trying to write up a proxy client in perl. I have finished part
of it where the script listens on port 7070 and when you configure
your browser to use that port, it will get the webpage just fine but I
see that if the page has calls out to other servers for images etc,
the request half heartedly fails and I just get the web page minus the
content that is hosted on some other website. Can someone shine some
light if and how I can have the proxy app fetch those urls too?

Thanks

It's hard to say without knowing the script you use or how it works. It
could just be displaying the content via the script on the "proxy
server" with the HTML/text and wants to just call the images normally,
as if they are also on that same server, at their expected relative or
absolute paths. You'd probably want to have the script prepend the
domain and appropriate path, as it would be seen on the server it's
fetching the content from (assuming they don't prevent that (especially
by blocking invalid refers -- not that those can't be bypassed)). You
could fetch the images, at least temporarily, too, depending on how you
want it to work.

 

[Martien Verbruggen]

Hi,

I am trying to write up a proxy client in perl. I have finished part
of it where the script listens on port 7070 and when you configure
your browser to use that port, it will get the webpage just fine but I
see that if the page has calls out to other servers for images etc,
the request half heartedly fails and I just get the web page minus the
content that is hosted on some other website. Can someone shine some
light if and how I can have the proxy app fetch those urls too?

The proxy shouldn't have to parse the HTML returned from a server to get
the references in that HTML as well (I am assuming you're talking about
a HTTP proxy here, and you're fetching HTML pages?).

The browser gets a document, lets say it's a HTML document, from a
server. Once it has the document, the original HTTP transaction is
over. It then parses that document for display. Any contained objects
that need to be displayed get fetched in a separate transaction. So, if
there's a <img src=.../> link in the HTML, the browser will create a
separate HTTP GET request for that. The proxy would see that second
request separate from the original GET request that fetched the HTML
document.

If you know this, and you really should before you write a proxy server,
I apologise, but it doesn't seem clear to me from what you wrote that
you do know. If you don't know, and you're doing this to learn about all
this, i suggest you read up on the relevant RFCs

It's a bit hard to say what you mean by "calls out to other servers" and
"the request half heartedly fails".

HTTP allows for multiple requests to be sent over the same TCP/IP
connection, but they would still be separate, serialised, and all
constructed by the user agent, not an intermediate proxy.

You do know that there are modules available to create HTTP proxy
servers?

http://search.cpan.org/search?query=proxy

Martien