The Industry choice

[Bulba!]

It's not just a matter of attitude or politics. Python is an
excellent choice for many projects. For some other projects, it's
clearly unsuitable. For yet other projects, it's a plausible choice
but there are sound technical reasons to be wary of it.

(being the usual me, I can't resist putting a stick into
a nest of pythons)

OK, so what projects and why would you consider
Python:

1. "clearly unsuitable"

2. "plausible but there are sound technical reasons
to be wary"

BTW, I don't think I agree with your statement
elsewhere about unit tests - if you write a prototype
(in any language), why bother with unit tests, and
if you write security-critical app, maybe the tests
should be written anyway (again, whatever is the
language chosen for a project).

 

[Paul Rubin]

OK, so what projects and why would you consider Python:

1. "clearly unsuitable"

An OS kernel, a high-performance signal processing application like a
video encoder, or maybe a compact embedded application for an 8-bit
microcontroller. Note that you could do the microcontroller project
with JavaCard. (Yeah, you could maybe write the outer shell of the
high-performance app in Python, but I mean the part doing the real work.)

2. "plausible but there are sound technical reasons to be wary"

A security-critical financial application.

BTW, I don't think I agree with your statement elsewhere about unit
tests - if you write a prototype (in any language), why bother with
unit tests, and if you write security-critical app, maybe the tests
should be written anyway (again, whatever is the language chosen for
a project).

You should write unit tests either way, but in Python you're relying
on the tests to find stuff that the compiler finds for you with Java.

I'm also not too impressed with Python's bundled debugging features,
as compared with gcc/gdb. People have said nice things about some of
the commercial Python environments but I haven't used those.

 

[Paul Rubin]

If it has to be both reliable and secure, I suggest you used more
redundant language such as Ada 95.

That's something to think about and it's come up in discussions, but
probably complicates stuff since it's not currently available on the
target platform. Also, the people on the project have significant
Java and Python experience but haven't used Ada. Do you think it has
real advantages over Java?

 

[Christopher Koppler]

An OS kernel, a high-performance signal processing application like a
video encoder, or maybe a compact embedded application for an 8-bit
microcontroller.

Of course, but those are not areas where I'd consider any
high-level language appropriate. C, Forth and Assembler rule where every
last bit counts, and they will continue to do so. Exceptions like Lisp
Machines are however something to hope for.

Note that you could do the microcontroller project

with JavaCard. (Yeah, you could maybe write the outer shell of the
high-performance app in Python, but I mean the part doing the real work.)

Hmm, didn't know about JavaCard, but I'd guess without bothering to look
that, like Java's Micro Edition, this features a restricted subset of
Java... Python definitely cannot do that now, but the PyPy future may
(hopefully) change that. And at least Python's beginning to try to
encroach in the very heavily Java-dominated mobile phone software market...

A security-critical financial application.

Why, specifically? Would you need to eval user input?

You should write unit tests either way, but in Python you're relying
on the tests to find stuff that the compiler finds for you with Java.

I'm also not too impressed with Python's bundled debugging features,
as compared with gcc/gdb. People have said nice things about some of
the commercial Python environments but I haven't used those.

I haven't used those either (well, I looked at some, but I generally feel
better at home in Emacs or even Idle than some glitzy IDE), but I find
Python's debugging facilities completely sufficient, especially since I
nearly never use them ;-) The interactive environment and unit testing are
just great for whatever I've needed so far. But then I haven't used Python
in a really *large* project yet, either.

 

[Alex Martelli]

True, but learning to *think* in Python takes a while longer. That static
straitjacket takes some time to loosen...

Yes to both: learning, particularly with Jython (so one keeps using
already-known Java libraries), is quite fast -- the full benefits take
more time to develop, although even using Python less than ideally can
already be faster. But for a company to convince itself that the
investment is small and the benefits accrue rapidly -- and keep accruing
in the long term -- can still be quite a leap of faith.


Alex

 

[Alex Martelli]

The moral is, of course, that either the Python community's alpha geeks
need to get access to controlling interest in a *major* company (or to
become successful enough with their own companies to register on the
current *major* companies radar as potential competition) or as you

Well, Google's market capitalization must be around 50 billion dollars
or more, in the range of the top-100 companies, I believe, and they've
never kept their Python use a secret. But they don't sell SW nor
consulting services like, say, MS or IBM...


Alex

 

[Paul Rubin]

Well, Google's market capitalization must be around 50 billion dollars
or more, in the range of the top-100 companies, I believe, and they've
never kept their Python use a secret.

They use Python for a lot of internal tools but their high-volume
services are written in C++.

 

[Alex Martelli]

They use Python for a lot of internal tools but their high-volume
services are written in C++.

I thought the "webcrawling" and updating of the indices/databases/caches
(high-volume, but not latency-critical) was coded in Python, the
latency-critical part (essentially that where every fraction of a second
counts, because a human user might be waiting for an answer), such as
queries to those DBs, in C++ -- based on the idea that you can "always"
(well, if you've got the right underlying data structures, of course,
and if the problems you're solving are as "embarassingly parallel" as
these would seem to be) enhance throughput by throwing more HW at the
problem, but latency doesn't yield to such treatment. But, I could be
wrong, of course; I don't recall where I may have read these details,
and the throughput vs latency issue is a well-known general
consideration, so I may simply have "bridged" these in my mind.


Alex

 

[beliavsky]

OK, so what projects and why would you consider
Python:
1. "clearly unsuitable"

Large-scale scientific computing projects, such as numerical weather
prediction, where performance is critical. Python could be used as the
"glue" but not the "guts", where Fortran 95 and C++ are more
appropriate. In my tests, some posted here, there has been a
significant speed differential between Fortran 95 and Python with
Numeric. I don't know if using Numarray instead would change the
results.

Calling an extension written in C, such as Numeric, can improve
performance greatly over using "pure" Python, but the speed of
evaluating simple expressions can still be close to 2 orders of
magnitude slower.

Here is an example that computes
(1) sum(x)
(2) sum(x**2)
(3) sum((x-0.5)**2)

for 1E7 uniformly distributed random numbers.

# Python
from RandomArray import random
from time import time
from Numeric import sum
n = 10000000
half = 0.5
x = random(n)
for iopt in range(1,4):
t1 = time()
if (iopt == 1):
xsum = sum(x)
elif (iopt == 2):
xsum = sum(x**2)
elif (iopt == 3):
xsum = sum((x-half)**2)
t2 = time()
print iopt,xsum,t2-t1

! Fortran 95
program xsum_power
implicit none
integer, parameter :: n = 10000000
real(kind=8) :: x(n),xsum,t1,t2
real , parameter :: half = 0.5
integer :: iopt
iopt = 1
call random_seed()
call random_number(x)
do iopt=1,3
call cpu_time(t1)
select case (iopt)
case (1) ; xsum = sum(x)
case (2) ; xsum = sum(x**2)
case (3) ; xsum = sum((x-half)**2)
end select
call cpu_time(t2)
write (*,"(i4,100f16.6)") iopt,xsum,t2-t1
end do
end program xsum_power

The approximate times taken in seconds, in Python and Fortran, are

task Fortran Python
sum(x) 0.05 0.1
sum(x**2) 0.05 2.4
sum((x-0.5)**2) 0.05 3.2

In the Fortran code, essentially all the time is spent accessing the
array elements. Powers and subtractions in array operations seem to be
free in Fortran but very expensive in Python with Numeric.

Not wanting to be guilty of "premature optimization", I did not try to
make either code more efficient than my first attempt. The Fortran
compiler is Compaq Visual Fortran 6.6c, and the Python used is 2.2.

 

[JanC]

Paul Rubin schreef:

Java code is larger and takes longer to write, but has a higher chance
of working properly once it compiles and passes basic tests.

That's why you can make most JSP sites useless by disabling cookies in
your browser?

 

[Alex Martelli]

array elements. Powers and subtractions in array operations seem to be
free in Fortran but very expensive in Python with Numeric.

Right, particularly raising to power: a good part of your observations
(not all) is accounted for by the fact that Python doesn't perform
strength reduction. Taking a trivial example (everything is slow since
I'm using a laptop in lowpower mode, but the ratio is meaningful):

kallisti:/tmp alex$ /usr/bin/python timeit.py -s'import Numeric;
xs=Numeric.ones(999); sum=Numeric.sum' 'sum(xs*xs)'
10000 loops, best of 3: 67.2 usec per loop

kallisti:/tmp alex$ /usr/bin/python timeit.py -s'import Numeric;
xs=Numeric.ones(999); sum=Numeric.sum' 'sum(xs**2)'
1000 loops, best of 3: 835 usec per loop


I guess this plays to the strengths of old fogeys like me -- having
grown up in a world where even good (-for-the-time) Fortran compilers
(!) often didn't do strength reduction, it just wouldn't _occur_ to me
to write x**2 where I mean x*x. ((Memo to self: point this out in the
Optimization section of the Nutshell's 2nd ed, since it IS a reasonably
frequent performance trap)). I remember once working with a good
scientist, back in the early 80's, on getting good running times for his
Fortran program coming from some (I believe) CDC machine to an IBM
mainframe, and achieving almost an order of magnitude worth of gain in
some important function by doing manual strength reduction in just a few
statements; apparently the CDC Fortran compiler _did_ do strength
reduction (I don't recall whether it was specifically x**2 -> x*x).

I believe, but I don't have one at hand to check, that scipy.weave may
be better in this regard.


Alex

 

[Cameron Laird]

That's something to think about and it's come up in discussions, but
probably complicates stuff since it's not currently available on the
target platform. Also, the people on the project have significant
Java and Python experience but haven't used Ada. Do you think it has
real advantages over Java?

You have me curious, Paul; what's the platform which lacks an Ada95
compiler <URL:
http://directory.google.com/Top/Computers/Programming/Languages/Ada/Compilers/ >?

Me, I think Ada has significant advantages over Java in safety,
especially where OO is *not* a benefit (as is sometimes the case).

For a not-too-different variety of safety, I like Eiffel. Again,
Eiffel compilers are available nearly, but not entirely, everywhere.

I'm not arguing for redundancy (see above), by the way, just precision.

 

[Cameron Laird]

.
.
.

Manager culture is still very much mired in rituals that may in one form
or another go back to hunter-gatherer days (or maybe even further); that
'the industry choice' is more often than not something backed by a *major*
company is part of a ritual complex based on relations to the alpha male.
Small companies ingratiate themselves with their perceived betters by
using their products, even when technically far superior products would be
available. When the 'market leader' produces a new toy, everyone who wants
to be in his favor must use it _and_ also damn the toys available from any
of those competing for leadership, viz. the ongoing state of cold war
between Sun and MS and their respective worshipers. Toys that have not
been sanctioned by the leader, or that are, even worse, de facto unknown
to him, are met with ignorance, scorn, or even repression.

[snip]

For Python a Big Thing would happen if some Major Vendor
embraced it as its Official Language(tm). Python language
itself could turn into a smoking crock the very next day, but
everybody who doesn't live under the rock would still be
writing in it.

The moral is, of course, that either the Python community's alpha geeks
need to get access to controlling interest in a *major* company (or to
become successful enough with their own companies to register on the
current *major* companies radar as potential competition) or as you
say, Python needs to be embraced like Linux was. That's the way to win the
hearts of software companies' managers.

.
.
.
I like repeating the description which emphasizes culture
and phenotype over the rationality of business schools.

Let me add a cautionary note, though: Big Companies,
including Oracle, Software AG, IBM, Cisco, and so on, have
adopted Tcl over and over. All of them still rely on Tcl
for crucial products. All of them also have employees who
sincerely wonder, "Tcl? Isn't that dead?"

I offer this as a counter-example to the belief that Adop-
tion by a heavyweight necessarily results in widespread
acceptance.

 

[Hans Nowak]

You should write unit tests either way, but in Python you're relying
on the tests to find stuff that the compiler finds for you with Java.

As I wrote on my weblog a while ago, I suspect that this effect is
largely psychological. You jump through hoops, declaring types all over
the place, checking exceptions, working around the language's
limitations, etc. So when your code compiles, it *feels* safer. Like
you're at least part of the way towards ensuring correctness. All that
work must be good for *something*, right? Never mind that when writing
unit tests for a dynamic language, you don't check for these things at
all. How often do you explicitly check types in Python unit tests?
IMHO, when using a dynamic language, you don't need most of the checks
that Java, C# and their ilk force upon you.

 

[Hans Nowak]

> --
> Christopher
>
> In theory, I'm in love with Lisp,
> but I hop into bed with Python every chance I get.

That reminds me of something my old math teacher used to say... "My wife
is my cathedral, but I pray in every chapel."

 

[Steve Holden]

.
.
.

Manager culture is still very much mired in rituals that may in one form
or another go back to hunter-gatherer days (or maybe even further); that
'the industry choice' is more often than not something backed by a *major*
company is part of a ritual complex based on relations to the alpha male.
Small companies ingratiate themselves with their perceived betters by
using their products, even when technically far superior products would be
available. When the 'market leader' produces a new toy, everyone who wants
to be in his favor must use it _and_ also damn the toys available from any
of those competing for leadership, viz. the ongoing state of cold war
between Sun and MS and their respective worshipers. Toys that have not
been sanctioned by the leader, or that are, even worse, de facto unknown
to him, are met with ignorance, scorn, or even repression.

[snip]

For Python a Big Thing would happen if some Major Vendor
embraced it as its Official Language(tm). Python language
itself could turn into a smoking crock the very next day, but
everybody who doesn't live under the rock would still be
writing in it.

The moral is, of course, that either the Python community's alpha geeks
need to get access to controlling interest in a *major* company (or to
become successful enough with their own companies to register on the
current *major* companies radar as potential competition) or as you
say, Python needs to be embraced like Linux was. That's the way to win the
hearts of software companies' managers.

.
.
.
I like repeating the description which emphasizes culture
and phenotype over the rationality of business schools.

Let me add a cautionary note, though: Big Companies,
including Oracle, Software AG, IBM, Cisco, and so on, have
adopted Tcl over and over. All of them still rely on Tcl
for crucial products. All of them also have employees who
sincerely wonder, "Tcl? Isn't that dead?"

I offer this as a counter-example to the belief that Adop-
tion by a heavyweight necessarily results in widespread
acceptance.

Indeed. This is. of course, because they adopt the technology to achieve
their business goals, and couldn't make money (using their traditional
business models) by promoting the technologies they themselves rely on.

Would anyone undertake to give a "Hidden Technologies" talk at PyCon,
outlining how this phenomenon operates against the adoption of
technologies that the big boys effectively keep to themselves by keeping
quiet about? Google's use of Python , while not a closely-kept a secret
as Oracle's use of Tcl, certainly isn;t as well-known as it deserves to be.

regards
Steve

 

[Mike Meyer]

For a not-too-different variety of safety, I like Eiffel. Again,
Eiffel compilers are available nearly, but not entirely, everywhere.

Eiffel compilers tend to generate C code, and hence work on anything
with a C compiler. The question then becomes how do you get to the
parts of your platform that you need to get work done.

<mike

 

[Paul Rubin]

Why, specifically? Would you need to eval user input?

Static typing, checked exceptions, etc.

I haven't used those either (well, I looked at some, but I generally
feel better at home in Emacs or even Idle than some glitzy IDE), but
I find Python's debugging facilities completely sufficient,
especially since I nearly never use them ;-) The interactive
environment and unit testing are just great for whatever I've needed
so far. But then I haven't used Python in a really *large* project
yet, either.

I think it's partly a matter of development style. I like debuggers
where some people prefer print statements. IDLE's debugging features
were very crude, and IDLE locked up all the time or left stray threads
around when I used it.

There's lots of times when I have a cool programming idea, and find
when I sit down at the computer that I can implement the main points
of the idea and get a neat demo running rather quickly. That creates
a very happy, liberating feeling, plus gives me something to show off
to friends or co-workers. But turning it into a finished product with
no rough edges is an order of magnitude more work. It seems to me
that IDLE and a lot of the rest of Python are examples of someone
having a cool idea and writing a demo, then releasing it with a lot of
missing components and rough edges, without realizing that it can't
reasonably be called complete without a lot more work.

 

[JanC]

Steve Holden schreef:

Would anyone undertake to give a "Hidden Technologies" talk at PyCon,
outlining how this phenomenon operates against the adoption of
technologies that the big boys effectively keep to themselves by
keeping quiet about? Google's use of Python , while not a closely-kept
a secret as Oracle's use of Tcl, certainly isn;t as well-known as it
deserves to be.

And since 2004-10-14, Corel uses Python too... ;-)

They bought JASC Software last year, who use Python as a scripting
language in their PaintShop Pro products.

 

[M.E.Farmer]

No one pointed these out so here goes:
Python's list
http://www.python.org/Quotes.html
Disney animation
http://python.oreilly.com/news/disney_0201.html
Disney Panda3d
http://www.python.org/pycon/dc2004/papers/29/

There are several other companies using Python for game logic and
scripting, I am drawing a blank right now.

M.E.Farmer