mod_python confusion

[Kylotan]

I thought mod_python would be the answer to my CGI performance issue,
but I can't seem to make much sense out of it. All the examples are
too trivial to be of much use.

Given that I have an application consisting of several CGI scripts,
what is the simplest way to go about converting it to run on
mod_python, not using the CGI-handler which is allegedly slow and
buggy? Everything seems to suggest a major rewrite because I not only
need to shift from printing everything to calling req.write or
whatever, but I may also need to sort out some sort of dispatching
mechanism mapping URLs to functions if I use mod_python directly. PSP
doesn't work on my system as 3.1.2 gives me errors and the Win32
installer for 3.1.3 fails at the last step (finding the Apache
directory in the registry or something).

 

[Tuure Laurinolli]

I thought mod_python would be the answer to my CGI performance issue,
but I can't seem to make much sense out of it. All the examples are
too trivial to be of much use.

I recommend using some framework that abstracts the web server - page
code interface out of the way. Jonpy is very lightweight, can be used
with standard CGI, FastCGI or mod_python without many changes and has
pretty nice interface. It is probably relatively easy to write a new
"backend" for it if you ever need to.

Given that I have an application consisting of several CGI scripts,
what is the simplest way to go about converting it to run on
mod_python, not using the CGI-handler which is allegedly slow and
buggy? Everything seems to suggest a major rewrite because I not only

Converting some simple speed-needing CGI script into something jonpy can
handle should be trivial - and with FastCGI it really helps the speed
when the interpreter starting overhead is the biggest cause for slowness.

Then there are more comprehensive frameworks, which usually have their
own server processes to which the webserver of your choice proxies the
requests. I don't know much about those, and converting to them probably
isn't as easy.

 

[Eric S. Johansson]

Converting some simple speed-needing CGI script into something jonpy can
handle should be trivial - and with FastCGI it really helps the speed
when the interpreter starting overhead is the biggest cause for slowness.

here's a question that is of immediate interest for myself and possibly
others. How do you generalize the fast CGI type of solution to Python
programs in general? For example, I have a program that runs as a
filter off of emailrelay[1]. The long-term solution is integrating
python but that has its challenges (emailrelay is a threaded c++
application and we would want python applications to be threaded as well).

Since the integration isn't going to happen anytime soon, I am content
to run a persistent process (like fast CGI) with some glue logic handing
data over. The question is, is there a general framework for this. XML
RPC is a possibility but my experience has shown in to be a somewhat
sluggish protocol using the stock XML RPC server in Python.

it would be nice if one could make it a multithreaded server but I must
admit I haven't researched to see if the available XML RPC facilities in
Python can be threaded or at least forked.

I'm sure there are innumerable misunderstandings in my approach to this
problem so I await correction.

---eric

[1] emailrelay is a really nice mini MTA/proxy which lets you filter
messages at a variety of points.

 

[Troy Melhase]

Since the integration isn't going to happen anytime soon, I am content
to run a persistent process (like fast CGI) with some glue logic handing
data over. The question is, is there a general framework for this. XML
RPC is a possibility but my experience has shown in to be a somewhat
sluggish protocol using the stock XML RPC server in Python.

Have you tried the Fast Python XML RPC library?

http://sourceforge.net/projects/py-xmlrpc/

Also, you may want to host it on a server other than one from the standard
library, such as Medusa, Twisted, or Zope.

 

[Eric S. Johansson]

Have you tried the Fast Python XML RPC library?

http://sourceforge.net/projects/py-xmlrpc/

no I had not. it does look like it's very promising although I am
hoping that there has been no release since last year means that the
code is extremely stable and bug free and not that the project is dead.
It also means that now I need to find a pair of hands/mind to help
create the C lang side in emailrelay. writing C/C++ using speech
recognition is like gargling broken glass.

Also, you may want to host it on a server other than one from the standard
library, such as Medusa, Twisted, or Zope.

I've tried to use those packages awhile back but unfortunately, at the
time I tried them, they were still pretty much a "bag of parts" and I
was not able to successfully installed them. I am also reluctant to add
these tools to my environment because already I am counting on a fair
number of external pieces.

It's gotten to the point where I have created a toolkit which I call
"raging dormouse". The download component allows you to fetch software
directly from web sites or through the source forge "click-a-link"
download maze automatically. I then wrap the download component in a
shell script which unbundle's, runs installation procedure etc. for all
the components.

It's not quite as sophisticated as a portage script, rpm, or deb but it
is relatively platform independent if the platform is some form of UNIX
with bash and python.

---eric

 

[David Fraser]

I thought mod_python would be the answer to my CGI performance issue,
but I can't seem to make much sense out of it. All the examples are
too trivial to be of much use.

Given that I have an application consisting of several CGI scripts,
what is the simplest way to go about converting it to run on
mod_python, not using the CGI-handler which is allegedly slow and
buggy? Everything seems to suggest a major rewrite because I not only
need to shift from printing everything to calling req.write or
whatever, but I may also need to sort out some sort of dispatching
mechanism mapping URLs to functions if I use mod_python directly.

Basically the main performance gains from using mod_python are that you
are calling functions in an already loaded program, rather than running
a separate script each time (and the interpreter as well). So it
generally does require a redesign. But if you refactor your code step by
step this is usually achievable.

> PSP
doesn't work on my system as 3.1.2 gives me errors and the Win32
installer for 3.1.3 fails at the last step (finding the Apache
directory in the registry or something).

There is a new win32 installer at
http://davidf.sjsoft.com/files/mod_python-3.1.3-1.win32-py2.3.exe which
should fix the issue (ask here or email me or see the mod_python list
for more information)

David