Python Web Programming - looking for examples of solid high-traffic sites

[Victor Kryukov]

Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

Today, as we get better idea of what we need, we're going to re-write
everything from scratch. Python is an obvious candidate for our team:
everybody knows it, everybody likes it, it has *real* objects, nice
clean syntax etc.

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

Our problem is - we yet have to find any example of high-traffic,
scalable web-site written entirely in Python. We know that YouTube is
a suspect, but we don't know what specific python web solution was
used there.

TurboGears, Django and Pylons are all nice, and provides rich features
- probably too many for us - but, as far as we understand, they don't
satisfy the stability requirement - Pylons and Django hasn't even
reached 1.0 version yet. And their provide too thick layer - we want
something 'closer to metal', probably similar to web.py -
unfortunately, web.py doesn't satisfy the stability requirement
either, or so it seems.

So the question is: what is a solid way to serve dynamic web pages in
python? Our initial though was something like python + mod_python +
Apache, but we're told that mod_python is 'scary and doesn't work very
well'.

And although http://www.python.org/about/quotes/ lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Your suggestions and comments are highly welcome!

Best Regards,
Victor.

 

[John Nagle]

Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago. ....
Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

You may not be happy with Python, then.

Having spent the last several months implementing a reasonably complex
web site in Python, I now have an understanding of the problems involved.
Which are non-trivial.

Some key web site components, like the SSL interface and the
MySQL interface, are written in C and maintained by third parties,
often by a single person. Getting the correct version for your
platform and making it work can be difficult. Getting the right
versions of MySQL, OpenSSL, and Python to all play together is
non-trivial. Expect to have to build from source, debug the build
process, look at source repositories, submit bug reports, fix
library bugs yourself, and maintain a local set of library patches.

Few hosting companies will have these modules available for you.
It's not like Perl or PHP, where it works out of the box. WebFaction
claims to support Python well, but few other hosting companies bother.
Most Linux distributions ship with older versions of Python, and
that's what most hosting companies will give you. Python 2.4
is par for the course.

High traffic sites are a problem. There are a number of
"frameworks", all supported by different people and with different
capabilities. See

http://www.polimetrix.com/pycon/slides/

for a discussion. I haven't tried most of them, so I won't
say anything about that issue.

Our problem is - we yet have to find any example of high-traffic,
scalable web-site written entirely in Python. We know that YouTube is
a suspect, but we don't know what specific python web solution was
used there.

Not that much of YouTube is really in Python. The video
codecs aren't; they'd take forever if they were. And since Google
took over, the old YouTube search engine (which was terrible) was
replaced by Google's search technology.


John Nagle

 

[=?iso-8859-1?q?Luis_M=2E_Gonz=E1lez?=]

Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

Today, as we get better idea of what we need, we're going to re-write
everything from scratch. Python is an obvious candidate for our team:
everybody knows it, everybody likes it, it has *real* objects, nice
clean syntax etc.

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

Our problem is - we yet have to find any example of high-traffic,
scalable web-site written entirely in Python. We know that YouTube is
a suspect, but we don't know what specific python web solution was
used there.

TurboGears, Django and Pylons are all nice, and provides rich features
- probably too many for us - but, as far as we understand, they don't
satisfy the stability requirement - Pylons and Django hasn't even
reached 1.0 version yet. And their provide too thick layer - we want
something 'closer to metal', probably similar to web.py -
unfortunately, web.py doesn't satisfy the stability requirement
either, or so it seems.

So the question is: what is a solid way to serve dynamic web pages in
python? Our initial though was something like python + mod_python +
Apache, but we're told that mod_python is 'scary and doesn't work very
well'.

And althoughhttp://www.python.org/about/quotes/lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Your suggestions and comments are highly welcome!

Best Regards,
Victor.

Take a look at reddit.com .
It's been developed with WEBPY and it receives millions of visits
every day.
There are also many sites built with Django (check their website) with
a lot of traffic and very good performance
And I'm sure other frameworks can show other success stories... just
check their websites.

Luis

 

[=?iso-8859-1?q?Luis_M=2E_Gonz=E1lez?=]

Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

Today, as we get better idea of what we need, we're going to re-write
everything from scratch. Python is an obvious candidate for our team:
everybody knows it, everybody likes it, it has *real* objects, nice
clean syntax etc.

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

Our problem is - we yet have to find any example of high-traffic,
scalable web-site written entirely in Python. We know that YouTube is
a suspect, but we don't know what specific python web solution was
used there.

TurboGears, Django and Pylons are all nice, and provides rich features
- probably too many for us - but, as far as we understand, they don't
satisfy the stability requirement - Pylons and Django hasn't even
reached 1.0 version yet. And their provide too thick layer - we want
something 'closer to metal', probably similar to web.py -
unfortunately, web.py doesn't satisfy the stability requirement
either, or so it seems.

So the question is: what is a solid way to serve dynamic web pages in
python? Our initial though was something like python + mod_python +
Apache, but we're told that mod_python is 'scary and doesn't work very
well'.

And althoughhttp://www.python.org/about/quotes/lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Your suggestions and comments are highly welcome!

Best Regards,
Victor.

Take a look at reddit.com .
It's been developed with WEBPY and it receives millions of visits
every day.
There are also many sites built with Django (check their website) with
a lot of traffic and very good performance
And I'm sure other frameworks can show other success stories... just
check their websites.

Luis

 

[=?iso-8859-1?q?Luis_M=2E_Gonz=E1lez?=]

Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

Today, as we get better idea of what we need, we're going to re-write
everything from scratch. Python is an obvious candidate for our team:
everybody knows it, everybody likes it, it has *real* objects, nice
clean syntax etc.

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

Our problem is - we yet have to find any example of high-traffic,
scalable web-site written entirely in Python. We know that YouTube is
a suspect, but we don't know what specific python web solution was
used there.

TurboGears, Django and Pylons are all nice, and provides rich features
- probably too many for us - but, as far as we understand, they don't
satisfy the stability requirement - Pylons and Django hasn't even
reached 1.0 version yet. And their provide too thick layer - we want
something 'closer to metal', probably similar to web.py -
unfortunately, web.py doesn't satisfy the stability requirement
either, or so it seems.

So the question is: what is a solid way to serve dynamic web pages in
python? Our initial though was something like python + mod_python +
Apache, but we're told that mod_python is 'scary and doesn't work very
well'.

And althoughhttp://www.python.org/about/quotes/lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Your suggestions and comments are highly welcome!

Best Regards,
Victor.

Reddit.com was built with webpy, and it's amongst the top 1000
websites in volume of traffic.
So if you liked it, go with it...
I believe one of the authors of reddit is also webpy's developer.

Luis

 

[Istvan Albert]

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

I think this is a requirement that is pretty much impossible to
satisfy. Only dead frameworks stay the same. I have yet to see a
framework that did not have incompatible versions.

Django has a very large user base, great documentation and is deployed
for several online new and media sites. It is fast, it's efficient and
is simple to use. Few modern frameworks (in any language) are
comparable, and I have yet to see one that is better,

http://code.djangoproject.com/wiki/DjangoPoweredSites

i.

 

[Jarek Zgoda]

Victor Kryukov napisa³(a):

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

I don't think you find anything even remotely resembling that idea here.
Moreover, I don't think you find it elsewhere. Maybe even such tools do
not exist in nature?

 

[Steve Holden]

Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

And has stayed around to dog the developers, as so many quick fixes do ...

Today, as we get better idea of what we need, we're going to re-write
everything from scratch. Python is an obvious candidate for our team:
everybody knows it, everybody likes it, it has *real* objects, nice
clean syntax etc.

Yes indeedy, that's our language!

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

Given that even Apache has bugs despite its venerable status I think you
are setting your sights a little high here.

Our problem is - we yet have to find any example of high-traffic,
scalable web-site written entirely in Python. We know that YouTube is
a suspect, but we don't know what specific python web solution was
used there.

Zope? Plone? Django? TurboGears? All are handling large volumes of data
on a daily basis, and I wouldn't use either of them (but that's just a
personal issue).

TurboGears, Django and Pylons are all nice, and provides rich features
- probably too many for us - but, as far as we understand, they don't
satisfy the stability requirement - Pylons and Django hasn't even
reached 1.0 version yet. And their provide too thick layer - we want
something 'closer to metal', probably similar to web.py -
unfortunately, web.py doesn't satisfy the stability requirement
either, or so it seems.

So the question is: what is a solid way to serve dynamic web pages in
python? Our initial though was something like python + mod_python +
Apache, but we're told that mod_python is 'scary and doesn't work very
well'.

Python CGI? mod_python is OK, but like all frameworks you have to be
aware of its limitations.

And although http://www.python.org/about/quotes/ lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Google use Python for all sorts of stuff.

Your suggestions and comments are highly welcome!

Best Regards,
Victor.

I think you are chasing a chimera (and I wrote a *book* called "Python
Web Programming").

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
------------------ Asciimercial ---------------------
Get on the web: Blog, lens and tag your way to fame!!
holdenweb.blogspot.com squidoo.com/pythonology
tagged items: del.icio.us/steve.holden/python
All these services currently offer free registration!
-------------- Thank You for Reading ----------------

 

[Paul Boddie]

When settling on an environment you need to have a clear strategy
about component versions and compatibility. Although Python provides a
fairly reasonable backwards compatibility story, one still needs to be
aware of changes between versions and their implications, but this is
the case for a large proportion of the software produced today.
Despite the baggage maintained by Microsoft so that ancient
applications might still run in recent versions of Windows, to choose
a fairly extreme example in certain respects, there are still many
things to be aware of when maintaining or upgrading your environment.
Saying that something needs "Python plus Windows/Linux plus some
database system" isn't going to be enough if you're emphasizing
stability.

You may not be happy with Python, then.

Having spent the last several months implementing a reasonably complex
web site in Python, I now have an understanding of the problems involved.
Which are non-trivial.

I'm sure they are, which is why I attempt to use the tools and
services available to reduce the impact of those problems on myself.

Some key web site components, like the SSL interface and the
MySQL interface, are written in C and maintained by third parties,
often by a single person. Getting the correct version for your
platform and making it work can be difficult. Getting the right
versions of MySQL, OpenSSL, and Python to all play together is
non-trivial. Expect to have to build from source, debug the build
process, look at source repositories, submit bug reports, fix
library bugs yourself, and maintain a local set of library patches.

I think we went through this before, but to what extent did you rely
on your distribution to handle many of these issues? Or was this on
Windows? If so, you're the master of your own distribution rather than
a group of helpful people who've probably already done most of that
work.

Few hosting companies will have these modules available for you.
It's not like Perl or PHP, where it works out of the box. WebFaction
claims to support Python well, but few other hosting companies bother.
Most Linux distributions ship with older versions of Python, and
that's what most hosting companies will give you. Python 2.4
is par for the course.

There's nothing wrong with Python 2.4. Perhaps you've inadvertently
highlighted an issue with the Python community: that everyone wants
the latest and greatest, and sees the CPython distribution as the
primary means of delivering it. If so, you've just included yourself
in the group causing yourself all those problems described above.

Paul

P.S. The inquirer may wish to visit sites representing the different
frameworks in order to seek testimonials, and to look for sites using
those frameworks in order to assess their popularity. Zope, Plone,
Django, TurboGears, Webware and many others have delivered high-
profile sites at various times, as far as I recall.

 

[Chris Cioffi]

I think the first question I would have is what kind of dynamic
content are you talking about? Is this a web app kind of thing, or
just a content pushing site?

While Django might not be v1.0 yet, it seems very solid and stable,
and perfect for quickly building powerful content based dynamic sites.

Other Python frameworks might be well suited to other types of dynamic
sites. (I like TurboGears for actual web apps...)

Depending on your needs you might also consider a non-python solution
(!) like Drupal. (I'm not a PHP fan, but since other people have done
all that great work....

As others have said, there might be no framework currently in
existence that meets all of your requirements. You'll need to work
with your team to decide what kinds of compromises you can all live
with.

Chris

 

[John Nagle]

Victor Kryukov napisa³(a):




I don't think you find anything even remotely resembling that idea here.
Moreover, I don't think you find it elsewhere. Maybe even such tools do
not exist in nature?

Sure they do. I have a complex web site, "http://www.downside.com",
that's implemented with Perl, Apache, and MySQL. It automatically reads SEC
filings and parses them to produce financial analyses. It's been
running for seven years, and hasn't been modified in five, except once
when the NASDAQ changed the format of their ticker symbol file.

It's not that useful at this point, because its purpose was to predict
failing dot-coms, but it's still running and doing a sizable amount of
work every day to update itself.

John Nagle

 

[Alex Martelli]

And although http://www.python.org/about/quotes/ lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Hmmm, I do, but exactly because I _do_ know a lot more details I cannot
discuss them unless you're under the proper Non-Disclosure Agreement
with Google, Inc. The best I can do otherwise is to point you to
already existing webpages -- I'm not going to reveal Google-Confidential
information, nor go to the substantial trouble to get such info cleared
by Legal and all other Google stakeholders.

For example, it HAS been published elsewhere that YouTube uses lighttpd,
not Apache: <http://trac.lighttpd.net/trac/wiki/PoweredByLighttpd>.

Fortunately, I managed to convince my YouTube colleagues to publically
present many more details about the evolution of their architecture
which had been discussed in Google-confidential talks and presentations
-- and my wife Anna, who's on the program selection committee of OSCON,
may have helped that talk get accepted (must not have been a hard
job. See:
<http://conferences.oreillynet.com/cs/os2007/view/e_sess/13435>

I hope to see you (and anybody else interested in solid technical
details about using Python for websites on YouTube's scale) in Portland,
OR on July 26 -- that will also be your first and best chance to ask
Mike Solomon specific questions that his talk might not directly
address. Once that's done, maybe somebody can convince the YouTube guys
to contribute a "Python Success Story" so that future querants about
this can be easily pointed to a URL!-)

I would also encourage anybody who's so keenly interested in Python to
visit our jobs-listing web app, e.g., if within the US, at
<http://www.google.com/support/jobs/bin/topic.py?loc_id=1100&dep_id=1173
&by_loc=1>. Of course, one should *never* have the implementation
language of a web-app show up in the URL, and I believe we've carefully
avoided that error in other external-facing web-apps, such as (one I can
reveal is indeed in Python, because that was mentioned at
<http://www.sauria.com/~twl/conferences/pycon2005/20050325/Python at%2
0Google.html>) code.google.com. Etc, etc.

Performance of web-apps (be they internal or external) depends more on
the architecture than on the implementation language (as long as highly
optimized C or C++, NOT Java or Python or whatever, is used for the few
jobs that are extremely CPU-intensive, such as codecs, of course.

So, if some Python-coded internal web-apps at Google perform badly
(which may be the case as you say, though I can't think of any
off-hand), it must be an issue of architecture. For example, a heavily
used internal web-app at Google is Mondrian,
<http://www.niallkennedy.com/blog/archives/2006/11/google-mondrian.html>
, Guido van Rossum's web-app for code review -- it's got a good, solid
architecture, and its performance is so good that many Googlers, me
included, have switched to it for all the reviews we do (and, believe
me, we do MANY -- _nothing_ is submitted to the global code repository
until it's been OK'd in a code review). I can't mention other such apps
because, AFAIK, they haven't been previously talked about in public and
so they're Google Confidential by default until otherwise determined.


Alex

 

[Daniel Nogradi]

For example, it HAS been published elsewhere that YouTube uses lighttpd,

not Apache: <http://trac.lighttpd.net/trac/wiki/PoweredByLighttpd>.

How do you explain these, then:

http://www.youtube.com/results.xxx
http://www.youtube.com/results.php
http://www.youtube.com/results.py

Just wondering,
Daniel

 

[Jarek Zgoda]

Daniel Nogradi napisa³(a):

How do you explain these, then:

http://www.youtube.com/results.xxx
http://www.youtube.com/results.php
http://www.youtube.com/results.py

Server signature is usually configurable.

 

[Bruno Desthuilliers]

John Nagle a écrit :

You may not be happy with Python, then.

John, I'm really getting tired of your systemic and totally
unconstructive criticism. If *you* are not happy with Python, by all
means use another language.

 

[Bruno Desthuilliers]

Istvan Albert a écrit :

I think this is a requirement that is pretty much impossible to
satisfy. Only dead frameworks stay the same. I have yet to see a
framework that did not have incompatible versions.

Django has a very large user base, great documentation and is deployed
for several online new and media sites. It is fast, it's efficient and
is simple to use. Few modern frameworks (in any language) are
comparable, and I have yet to see one that is better,

Then have a look at Pylons.

 

[Alex Martelli]

Daniel Nogradi napisa?(a):


Server signature is usually configurable.

Yeah, but I don't know why it's configured it that way. A good example
of a question that looks perfectly appropriate for YouTube's OSCON
session.


Alex

 

[Daniel Nogradi]

For example, it HAS been published elsewhere that YouTube uses lighttpd,

Yeah, but I don't know why it's configured it that way. A good example
of a question that looks perfectly appropriate for YouTube's OSCON
session.

Let us know what they say!

 

[John Nagle]

John Nagle a écrit :



John, I'm really getting tired of your systemic and totally
unconstructive criticism. If *you* are not happy with Python, by all
means use another language.

Denying the existence of the problem won't fix it.

Many of the basic libraries for web related functions do have
problems. Even standard modules like "urllib" and "SSL" are buggy,
and have been for years. Outside the standard modules, it gets
worse, especially for ones with C components. Version incompatibility
for extensions is a serious problem. That's reality.

It's a good language, but the library situation is poor. Python as
a language is better than Perl, but CPAN is better run than Cheese Shop.

As a direct result of this, neither the Linux distro builders like
Red Hat nor major hosting providers provide Python environments that
just work. That's reality.

John Nagle

 

[Daniel Nogradi]

For example, it HAS been published elsewhere that YouTube uses lighttpd,

Yeah, but I don't know why it's configured it that way. A good example
of a question that looks perfectly appropriate for YouTube's OSCON
session.

Actually, the fact that http://www.youtube.com/results.php returns
legitimate content might be explainable by the fact that youtube was
started as a PHP app so they might provide this URL for backward
compatibility although today there is no PHP at all. See the abstract
of Mike Solomon's OSCON talk: "YouTube began as a small PHP
application. [...]"
http://conferences.oreillynet.com/cs/os2007/view/e_sess/13435