Question about long-running web scripts

[Gilles]

Hello

I'd like to check something about running Python web applications.

Generally speaking, the reason scripts run faster when called through
FastCGI or the mod_* modules, is because the interpreter is already up
and running.
But when running PHP scripts, this does nothing about fetching the
file from disk, recompiling, rerunning it, and usually reconnecting to
the database.

OTOH, Python web scripts can be written as long-running scripts: In
this case, what is the added-value of using FastCGI? Why can't the web
server simply call the Python script directly, just like CGI?

Thank you.

 

[Tim Golden]

I'd like to check something about running Python web applications.

Generally speaking, the reason scripts run faster when called
through FastCGI or the mod_* modules, is because the interpreter is
already up and running. But when running PHP scripts, this does
nothing about fetching the file from disk, recompiling, rerunning it,
and usually reconnecting to the database.

OTOH, Python web scripts can be written as long-running scripts: In
this case, what is the added-value of using FastCGI? Why can't the
web server simply call the Python script directly, just like CGI?

(Your question is a little confused at the end. I'm choosing to
understand: why can't we just run Python one-shot, like CGI? The likely
alternative meaning is: why can't the incoming request be routed to an
already-running Python program -- which is not, of course, what CGI
generally does. Hence my confusion).

The answer is: it can. CGI is a protocol rather than anything else. In
front of a CGI exchange is the browser (or some other web client).
Behind it is some program which is capable of producing a valid HTTP
response, including a Python program.

It's perfectly possible to run a usable website against Python running
one-shot. You won't get terrific performance out of it, but for a
website which doesn't expect humungous amounts of traffic, it'll work fine.

The amount of time it takes a half-decent, even shared, server to start
up a Python process, connect to a database, pull stuff together, and
send a response will likely not impact on an average user's experience.
As long as too many of them don't try to do that at the same time.
Exactly where the line is drawn will depend on your particular hosting
solution, your assumed traffic, and your users' expectations as to
responsiveness.

TJG

 

[Gilles]

(Your question is a little confused at the end. I'm choosing to
understand: why can't we just run Python one-shot, like CGI? The likely
alternative meaning is: why can't the incoming request be routed to an
already-running Python program -- which is not, of course, what CGI
generally does. Hence my confusion).

Yes indeed. Sorry about the confusion.

But actually, I didn't mean one-shot scripts, where the Python
interpreter + script must be loaded each time, but rather: If I leave
a Python running in an endless loop, why not just use either CGI or
some other basic way to call the script instead of FastCGI?

In the case of PHP, FastCGI makes sense, but I don't see the benefit
for long-running Python scripts.

 

[David Hutto]

I'd say that is the same as py, unless it's a cron job to limit script
iterations

The server should call a the script, or script.sleep()

There are also server options to setup when a script is run, other
than a cron jo for php.

 

[Tim Golden]

Yes indeed. Sorry about the confusion.

But actually, I didn't mean one-shot scripts, where the Python
interpreter + script must be loaded each time, but rather: If I leave
a Python running in an endless loop, why not just use either CGI or
some other basic way to call the script instead of FastCGI?

In essence, you're describing FastCGI. A Python program (or, indeed, any
program) which uses FastCGI runs continuously and waits for the incoming
request on a TCP socket (instead of as a sys.stdin stream + env vars
immediately after process startup).

The key description is here:

http://www.fastcgi.com/drupal/node/6?q=node/15

(The sections have no anchors; you're looking for the section titled "2.
FastCGI Interface")

TJG

 

[Gilles]

The server should call a the script, or script.sleep()

There are also server options to setup when a script is run, other
than a cron jo for php.

Thanks. Does it mean that Python scripts that rely on either fcgi.py
or WSGI really have some endless loop somewhere and will keep running
once they're launched by FastCGI?

 

[Gilles]

In essence, you're describing FastCGI. A Python program (or, indeed, any
program) which uses FastCGI runs continuously and waits for the incoming
request on a TCP socket (instead of as a sys.stdin stream + env vars
immediately after process startup).

Thanks for the clarification.

Since, unlike PHP, the Python interpreter is not available in a
FastCGI-capable version, this explains why the www server must be told
which specific Python script to run through FastCGI.

The reason I ask for all this, is that I want to understand how things
work under the hood before relying on a Python framework to handle the
nitty-gritty.

 

[Tim Golden]

Thanks for the clarification.

Since, unlike PHP, the Python interpreter is not available in a
FastCGI-capable version, this explains why the www server must be told
which specific Python script to run through FastCGI.

I think that this is the distinction you're making:

PHP: mod_php (fastcgi mode) runs myscript.php

Python: <some python fcgi frontend>.py runs myscript.py

which is, essentially, true, not least because PHP and web apps are
pretty much synonymous in many people's minds. Both ways: the only thing
PHP does is web; the simplest route to a web app is PHP.

Certainly there are Python equivalents (mod_python, mod_wsgi, etc.)
which can run in effectively the same way as mod_php, and they could be
configured to run an fcgi frontend script, I presume. There's always a
certain confusion here because you can often one mechanisms (say,
mod_wsgi) to act as another (say legacy one-shot CGI) and because some
things are both mechanism and protocol and it's not always easy to tease
the two apart.

The reason I ask for all this, is that I want to understand how things
work under the hood before relying on a Python framework to handle the
nitty-gritty.

Good scheme.

TJG

 

[Gilles]

Certainly there are Python equivalents (mod_python, mod_wsgi, etc.)
which can run in effectively the same way as mod_php, and they could be
configured to run an fcgi frontend script, I presume. There's always a
certain confusion here because you can often one mechanisms (say,
mod_wsgi) to act as another (say legacy one-shot CGI) and because some
things are both mechanism and protocol and it's not always easy to tease
the two apart.

Thanks again.