Simple discussion of python cgi approaches?

[Kylotan]

[Apologies if anyone sees this twice - at first I attempted to post it
via my ISP's news server, but my ISP is notorious for poor news
handling and I don't see it on Google Groups as having got through
yet.]

I've been writing a CGI app with Python 2.3 and Apache. It's ok, but a
bit slow, and I assume a significant part of this is the process
start-up time, among other things. So I'm interested in alternatives
or optimisations. The problem is, there doesn't seem to be the kind of
information out there that I want. There seem to be plenty of places
that throw a list of URLs at you (eg.
http://www.python.org/cgi-bin/moinmoin/WebProgramming,
http://phaseit.net/claird/comp.lang.python/web_python.html), and a few
sites that will give the absolute basics of CGI programming itself
(which I don't need).

I'm particularly confused by a few people advocating things like
Xitami + LRWP for being very simple, yet it can't possibly be simpler
than just dropping a cgi script into a directory, as I do with Apache,
and the docs on LRWP
(http://www.imatix.com/html/xitami/index12.htm#TOC114) make it look
like I'd have to do significant rewriting of the scripts. It describes
itself as being like FastCGI which, if true, doesn't exactly endear
FastCGI to me either.

All I want is a short description on the technologies that very
closely resemble standard CGI, what they have to offer me, and what
sort of alterations I would have to make to my installation or code in
order to use these. Does such a comparison exist already? If not, is
anybody able to comment on their own experiences with Python CGI apps?

 

[Thomas Guettler]

Am Mon, 12 Apr 2004 18:27:58 -0700 schrieb Kylotan:

[cut]

All I want is a short description on the technologies that very
closely resemble standard CGI, what they have to offer me, and what
sort of alterations I would have to make to my installation or code in
order to use these. Does such a comparison exist already? If not, is
anybody able to comment on their own experiences with Python CGI apps?

Hi,

I started with Zope in 2001. This is very different
than CGI. Then I switched to quixote. It is very much
like CGI. Every HTTP-Request results in an method call
which gets an request object as argument.

In my situation it is enough to run quixote with plain cgi
(no SCGI, mod_python, ...): About three request per minute.
Loading the interpreter for every request is not the
bottleneck.

HTH,
Thomas

 

[Steve Holden]

[Apologies if anyone sees this twice - at first I attempted to post it
via my ISP's news server, but my ISP is notorious for poor news
handling and I don't see it on Google Groups as having got through
yet.]

I've been writing a CGI app with Python 2.3 and Apache. It's ok, but a
bit slow, and I assume a significant part of this is the process
start-up time, among other things. So I'm interested in alternatives
or optimisations. The problem is, there doesn't seem to be the kind of
information out there that I want. There seem to be plenty of places
that throw a list of URLs at you (eg.
http://www.python.org/cgi-bin/moinmoin/WebProgramming,
http://phaseit.net/claird/comp.lang.python/web_python.html), and a few
sites that will give the absolute basics of CGI programming itself
(which I don't need).

There are many differeing opinions on this subject, so maybe you'll just
have to accept that c.l.py responses reflect the prejudices of their
authors. They will at least usually be better-informed than random web
sites, one might hope.

I'm particularly confused by a few people advocating things like
Xitami + LRWP for being very simple, yet it can't possibly be simpler
than just dropping a cgi script into a directory, as I do with Apache,
and the docs on LRWP
(http://www.imatix.com/html/xitami/index12.htm#TOC114) make it look
like I'd have to do significant rewriting of the scripts. It describes
itself as being like FastCGI which, if true, doesn't exactly endear
FastCGI to me either.

Well, you said you are looking for "alternatives or optimaizations", so
it would appear that you feel you are approaching the limits of your
simple approach. It's a very unsual solution that will be simpler,
faster and cheaper than the "obvious" one, though such solutions aren't
unheard of, even in the web world.

All I want is a short description on the technologies that very
closely resemble standard CGI, what they have to offer me, and what
sort of alterations I would have to make to my installation or code in
order to use these. Does such a comparison exist already? If not, is
anybody able to comment on their own experiences with Python CGI apps?

Well, let me briefly summaries the advantages of Xitami/LRWP and
mod_python, two approaches with which I am familiar, as compared with
good 'ole vanilla CGI.

Both of these systems have the advantage that once your web
functionality is stable (i.e. the programs aren't changing each time you
want the run) you avoid the overhead of reloading both the interpreter
and the source code of your page generators (CGI scripts) for each page.

mod_python effectively works by integrating the interpreter into Apache
in such a way that each directory can, if necessary, have a separate
interpreter instance (to avoid namespace clashes between different
applications). Modules are loaded on-demand, and stay resident in the
Apache process unless changed. Sometimes there are problems if an
indirectly-imported module changes, since mod_python only checks those
it imports itself.

The intergration is very tight, and you can write Apache handlers of all
kinds in Python. I have experimented myself with writing request
handlers, with results briefly reported at PyCON this year. It's a good
general-purpose approach for uni-processor solutions.

Xitami is slightly different: the server is deliberately written as a
lightweight asynchronous request handler which is capable of queueing
requests until a service routine becomes available. The LRWP linkage
isn't actually as complicated as it might seem, and the only major
difference is that a service process must register with the server to
handle a class of URLs before it receives requests.

This gives you two immediate benefits: firstly, since network sockets
are used to communicate between the web server and the service process,
the web server can act as a conduit to external functionality very
easily - it's trivial to build a remote LRWP, since the only difference
between a local and a remote LRWP is the fact that the remote one will
use an external IP address for the web server rather than "localhost".
Second, since the web server is perfectly happy to accept multiple
registrations for a given class of URLs, and since each registration can
come from a different host, the asynchronous nature of the Xitami system
allows you to achieve potentially great scalability.

I described this approach in "Python Web Programming", since it seemed
to me to offer the easiest way to build web-based Python applications
which could potentially scale to hundreds of processors. The really nice
part is that you can add more LRWPs to any URL class that needs more
power, dynamically and without trauma.

I even built a system on Windows that had a URL to start up LRWPs on
demand, and to shut them down, under client control. It's very flexible.

Clearly anything that has performance benefits over CGI is likely to be
a little more difficult to get running but believe me, the extra effort
isn't that great when you consider the rewards.

HTH

regards
Steve

 

[A.M. Kuchling]

In my situation it is enough to run quixote with plain cgi
(no SCGI, mod_python, ...): About three request per minute.
Loading the interpreter for every request is not the
bottleneck.

It should be noted that Quixote's CGI publisher can also be run under
FastCGI; it should automatically detect when FastCGI is being used and
service multiple requests in a single process. Most FastCGI implementations
work like that (because the spec suggests it), so maybe just using FastCGI
with the existing CGI scripts will be sufficient for the original poster.

--amk

 

[Kylotan]

Well, you said you are looking for "alternatives or optimaizations", so
it would appear that you feel you are approaching the limits of your
simple approach. It's a very unsual solution that will be simpler,
faster and cheaper than the "obvious" one, though such solutions aren't
unheard of, even in the web world.

I know. I guess I am just spoiled by the simplicity of using things
like Psyco
that I sometimes forget that not everything comes for free!

mod_python [...]

The intergration is very tight, and you can write Apache handlers of all
kinds in Python. I have experimented myself with writing request
handlers, with results briefly reported at PyCON this year. It's a good
general-purpose approach for uni-processor solutions.

Yet I'm guessing it requires a significant amount of special-case code
to
use its own API? Looking at the manual it seems non-trivial to get it
to work with scripts that were originally written to consume stdin and
emit to stdout.

Clearly anything that has performance benefits over CGI is likely to be
a little more difficult to get running but believe me, the extra effort
isn't that great when you consider the rewards.

The main problem I face is that I don't know what platform my scripts
will
eventually run on. I may not have the ability to install extra
software so I
need to keep my scripts close to the CGI standard. That is, one file
per script,
using the cgi module for form processing and print for output. I don't
mind
having to redirect stdout or call these scripts indirectly instead of
via
their __main__ functions, but writing to specific APIs (beyond a small
wrapper
layer, of course) is not an option at the moment.

So I'm hoping for simple solutions that will involve no more than a
few changes
to each script, whereas most of the solutions that seem to be
available require
me to use their own request objects, or to treat pages as callable
functions within an overall page as mod_python seems to require.

 

[Steve Holden]

Kylotan wrote:
[...]

The main problem I face is that I don't know what platform my scripts
will
eventually run on. I may not have the ability to install extra
software so I
need to keep my scripts close to the CGI standard. That is, one file
per script,
using the cgi module for form processing and print for output. I don't
mind
having to redirect stdout or call these scripts indirectly instead of
via
their __main__ functions, but writing to specific APIs (beyond a small
wrapper
layer, of course) is not an option at the moment.

So I'm hoping for simple solutions that will involve no more than a
few changes
to each script, whereas most of the solutions that seem to be
available require
me to use their own request objects, or to treat pages as callable
functions within an overall page as mod_python seems to require.

Well, what you've just said probably rules out both the approaches I
chose to write about, but you've also defined your problem much more
closely and so someone else migth be able to offer more helpful advice.

regards
Steve