Advice for Python and Web Server/Services?

[Ron Davis]

I have recently discovered Python and like it quite a bit. I would like to
use it on a new project I am starting.

The project will gather data from several web services and present the
collected data to browser users through a web server.

So basically I need a full-time web server and a separate way to poll web
services for their data. There is no database involved.

The problem is that I have tried several Python web servers (CherryPy,
Karrigell, etc) and they all have problems getting out through my Comcast
cable modem. There is a 3 second pause every 10th file. Never the 9th or
11th, but always the 10th file.

I thought maybe Comcast was limiting incomming HTTP but Apache does not do
this so I'm stumped as to why it is happening with the Python servers. All
servers work fine over the LAN.

Anyway, I wanted to get some advice on what type of setup would be best
for this using Apache. The web service polling needs to be going all the
time so a straight CGI-type thing is not an option.

I've looked in to fastcgi and mod_python but I'm not sure either is what I
need. What I want is for my Python program to be polling the web
services 24/7 and Apache to serve static pages by itself but call
a function in my already running Python program for files with a certain
extension (.xml for example).

Have any of you done something like this before?

Ideally, I'd like to figure out why the Python servers are having
problems. This is a very low volume system so straight Python would be
great if I can get it working.

Thanks for any advice you can give.

 

[Dennis Lee Bieber]

I've looked in to fastcgi and mod_python but I'm not sure either is what I
need. What I want is for my Python program to be polling the web
services 24/7 and Apache to serve static pages by itself but call
a function in my already running Python program for files with a certain
extension (.xml for example).

I think you'll first need to disconnect the two sides... The program
doing the polling has to run completely stand-alone from your own
web-server. I also think you'll need to create some independent
(file-system?) store which the polling program can write to; so your web
server page can do the selected retrieval -- but since you don't want a
database capable of concurrent access... The polling program needs to
include an asynchronous or threaded scheme with a local socket so the
web server (CGI) process can connect to it and requests the poller to
return the data.

Ideally, I'd like to figure out why the Python servers are having
problems. This is a very low volume system so straight Python would be
great if I can get it working.

Any possibility you need to specify an option on the socket
connections to reuse-sockets without the normal time-out?
--

 

[Ronald Davis]

Any possibility you need to specify an option on the socket
connections to reuse-sockets without the normal time-out?

The problem turned out to be keep-alive. Apparently, Comcast is limiting my
inbound connections. Apache keeps the connections alive so it works fine. The
Python servers don't support keep-alive and are using up the available
connections too quickly.

I'm now using Apache as a proxy to the Python server and it works well.

Thanks for the response.