Python future performance and speed

[Ville Vainio]

Francesco> And VHLLs will become more and more used as computer
Francesco> power grows and becomes cheaper than developer time (it
Francesco> is already so in several application areas).

Don't forget that developer time is becoming cheaper as well. The
issue will be increasingly about code quality and the ability to
deliver on time / at all.

 

[Dieter Maurer]

...
Well, whenever I want to write a mainstream application and think of
using Python, my first reaction is that Python is too slow...

Maybe, you should drain your reactions

About 18 months ago, I proposed to implement an SGML/XML
based content management system in Zope/Python
because the company went bankrupt that
had offered a similar C++ based system we were relying upon.

Many of my colleagues objected because they thought Python
were too slow.

The Zope/Python CMS is now almost ready -- it is much faster
than its C++ counterpart. Not because Python is faster than C++
but because a Python application is far easier to tailor
and to tweak for efficiency than a C++ one (and because
we avoided known design flaws in the C++ system).

Dieter

 

[Grant Edwards]

Well, whenever I want to write a mainstream application and think of
using Python, my first reaction is that Python is too slow...

And I presume you're pleasantly surprised when you run your program and find
out it's not too slow?

 

[Paul Rubin]

That's a surprise! I didn't know about Lisp, so I did a quick search
in Google and I found this quote from CMUCL website:

"a sophisticated native-code compiler which is capable of powerful
type inferences, and generates code competitive in speed with C
compilers."

Heh, amazingly enough, the compiler subsystem of CMUCL also, by complete
coincidence, happens to be called Python.

Has anyone tried to learn from this experience? Could this technology
be used with Python?

I think Lisp compiler techniques can be used for Python, but some
parts of Python semantics are messier to compile. That's no big
surprise; Lisp semantics evolved over the years directly in response
to the needs of Lisp compilers. I think Python will probably evolve
the same way and I hope PyPy is deployable soon, since that will speed
up compiler development a lot. I think we should consider a
semi-moratorium til then on new language features that might affect
compilation.

"There's another guy I'd also like to invite, a Lisp expert, if that's
ok with you. He's been interested in writing a Python compiler for a
while. I'll ask him if he wants to join, but he might not."

What happened?

I don't remember what that was about, but I can guess which guy I must
have been thinking of. He's still around and used to read this newsgroup
sometimes.

 

[Paul Rubin]

And I presume you're pleasantly surprised when you run your program and find
out it's not too slow?

Sometimes that happens. Other times it really is too slow, and I
rewrite it in C. Other times Python is too bloated and I can't even
consider using it; for example, in anything that users need to download
without a big fuss.

 

[Tim Churches]

It should not be surprising at all that a general-purpose tool will be
beaten by a domain-specific tool in that domain. SAS, as you say, is
designed to munch huge numeric data sets. That is its only reason for
existing, and it's got 20 or 30 years of development effort behind it to
make it do that one task as fast as possible.

Sure, and as a programming environment SAS is both primitive (eg still
has no way to easily create user-defined functions in its main language,
and is forced to rely on macros instead) and confusing (since it needs
five or six distinct sub-languages to cope with both special-purpose and
general-purpose programming tasks eg uses Java called from its own PL/1
and Fortran-influenced "data step" language to do many things - yuck!)
when compared to Python. However, none of that is a reason for not to
wish that Python were faster than it is in some circumstances - mostly
when doing dumb stuff like iterating over a file. That's why Psyco and
increasingly Pyrex are so valuable, and why the fruits of the Starkiller
and Pypy projects are so eagerly anticipated.
--

Tim C

PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere
or at http://members.optushome.com.au/tchur/pubkey.asc
Key fingerprint = 8C22 BF76 33BA B3B5 1D5B EB37 7891 46A9 EAF9 93D0

 

[Reid Nichol]

After all python has been used as the scripting engine for a number of
commercial games - a domain that certainly cares about speed.

Not the graphics though and that is where the speed is really needed.

 

[Sam Holden]

Not the graphics though and that is where the speed is really needed.

True. However, when you run a script for every frame of said graphics
the script needs to run reasonably fast. And that has been done.

Obviously python is not *fast*, it is however, "fast enough" for the
vast majority of applications. I'd argue that for a significant
number of the applications python is not fast enough for, it is
possible to write a component in C (or C++ or whatever) with a
python wrapper.

Of course it's not that python is super fast, it's that hardware is
very fast these days. Games have gone from Assembler engines and
C game code to C (C++ even) engines and scripting language game code.

Of course python cod ebeing so much quicker to develop than C code
doesn't gain as much as machine's get faster during development

 

[beliavsky]

Speed is the primary limitation for mainstream acceptance of high level
(aka scripting) languages in general. I can think of no reason why so
much software is still written in languages like C++ other than
performance (or at least, the perception of it).

There are reasons other than performance why one might prefer C++ or
other languages to Python -- not to say that they are objectively
"better".

(1) Many people think that the static type checking of languages like
C++ is a virtue, and some of them avoid Python because it is dynamic.
I don't want to start another debate but just want to note that
programmers are divided on this.

(2) The Python approach to OOP is very different -- looser -- than
that of C++ and has less support for data hiding. For example, one can
add an attribute to an object anywhere, which seems like a dubious
freedom to me.

(3) The beginning of a Python function only tells you about the
arguments, not the result. I prefer not to scan the entire body of a
function to see what it returns. Compiled languages like C++, Fortran,
and Pascal put the argument and result info in one place.

(4) Python does not look like C++. Some programmers think that any
alternative to C++ must still look like it. I think this argument is
weak, but Java's cosmetic similarity to C++ probably has boosted its
popularity.

 

[Neuruss]

Statements like that serve only to demonstrate the narrowness of

thought of the writer.

Excuse me?
It seems that you're missing the point. There's nothing wrong with my
way of thinking, which is not "narrow" at all.
I love Python. It's an excellent, fun, clear, well designed language
and it is fast enough for 90% of the cases. But you can't deny that
for some problem domains, it is slow. Very slow.

The fact that some programmers can resort to c to speed up parts of
the code is not convincing to me. I don't like C, I like python, and
I'd love to see it performing at much better speed.

I didn't want to start a silly thread to discuss wether python is slow
or not. It is. Period.
But it doesn't mean that it is useles or bad. Again, I think Python
rocks, but I want more, and as far as I know, many people want more.

That's why I wanted to know more about all those exciting projects
aimed to improve python's performance.
I love to read comments from people like Armin Rigo, who says that
Psyco is the first step for world dominance of python (I love that
attitude!).
Or people like Mike Salib, who says that Python will be faster than
C/C++ in two years, thanks to research on type inference and compiler
improvements.

All the other comments on this thread, regarding the main limitation
for python's acceptance are true, but speed is critical, and this is
were all efforts should be concentrated.

So my intention when I started this thread, was to learn more about
these projects, read other oppinions, get some news, etc... I didn't
mean to start a a nonsense discussion!

 

[Jeremy Bowers]

Excuse me?
It seems that you're missing the point.

With respect, I think it is you who are not understanding the objections
people are having to your writing.

It is not this part that people are objecting:

But you can't deny that
for some problem domains, it is slow. Very slow.

It is this:

but speed is critical

And bordering on the offensive to some people is this:

and this is were all efforts should be concentrated.

as Python is an open-source and volunteer based this can be interpreted as
a command on how people should spend their freely offered time.

(In other words, all of *what* efforts? Are you helping to speed up Python
at all, or is this just rather strongly worded "suggestions" to other
people? Not a friendly approach.)

So my intention when I started this thread, was to learn more about
these projects, read other oppinions, get some news, etc... I didn't
mean to start a a nonsense discussion!

Referring to other people's opinions as "nonsense" is also not a path to a
productive discussion, especially when the prime criterion for
"nonsensicalness" seems to be "an opinion different then mine".

(For one, many people at many times, not just in this thread, have
observed that "speed" is a meaningless term without a context, and that
Python is fast enough for many, many contexts. Repeated assertion that
there exist contexts where Python isn't fast enough does nothing to
disprove that point. If you don't understand why that is logically true, I
suggest thinking about it for a bit before replying yet again that there
exist contexts where Python isn't fast enough.)

 

[Neuruss]

Jeremy, I didn't attack anyone's religion or race.
We are talking about a programing language...why are you so touchy?

as Python is an open-source and volunteer based this can be interpreted as
a command on how people should spend their freely offered time.

Granted. Perhaps I should have started the phrase: "In my humble
oppinion...".
English is not my mother tongue, as you might have already guessed, so
excuse me if I'm not sensitive enough...

(In other words, all of *what* efforts? Are you helping to speed up Python
at all, or is this just rather strongly worded "suggestions" to other
people? Not a friendly approach.)

If you read again my previous post, you'll see that I already
expressed my genuine admiration and gratitude for python's developers
and that I think that the path they are taking for improving python is
the correct one.
As for what I'm doing to help python, I'm affraid nothing, Im sorry.
I'm not a developer. Maybe I'm just helping by spreading the word that
python is great...
As far as I know, this forum is for all python users, not just
developers.
Now if you think I shouldn't participate here, please kindly let me
know.

Referring to other people's opinions as "nonsense" is also not a path to a
productive discussion, especially when the prime criterion for
"nonsensicalness" seems to be "an opinion different then mine".

I appreciate other people's oppinions, even when I disagree. I didn't
say that other people's oppinions were nonsense, what I meant is that
the whole discussion was getting nonsense, because there's no use in
discussing wether python is slow or not.
Now if you find this oppinion ofensive, I'm sorry.

If you don't understand why that is logically true, I
suggest thinking about it for a bit before replying yet again...

With duly respect, I don't need you to teach me how and when I should
express myself.
Please don't get me wrong, I really find much of your writing
reasonable, but the arrogant tenor of your educational suggestion is
out of place.

 

[Jeremy Sanders]

Sometimes that happens. Other times it really is too slow, and I rewrite
it in C. Other times Python is too bloated and I can't even consider
using it; for example, in anything that users need to download without a
big fuss.

Most Linux distributions include python as standard, so the users only
need to download your script (which will probably be smaller than the
equivalent C code).

Jeremy

 

[Michael Hudson]

Too slow for what?

I'm not being facetious here - what exactly are the bits of Python that
are too slow? I used to think this as well, but see my PyCon paper from
earlier this year.

Well, Python's pretty bad for things stuff like adaptive numerical
methods (the point of "adaptive" is that you can't do lots of problems
at once using numarray or similar, or at least I couldn't work out how
when I was an undergraduate .

Cheers,
mwh

 

[Aahz]

Excuse me?
It seems that you're missing the point. There's nothing wrong with my
way of thinking, which is not "narrow" at all.

Yes, it is.

I love Python. It's an excellent, fun, clear, well designed language
and it is fast enough for 90% of the cases. But you can't deny that
for some problem domains, it is slow. Very slow.

That's true. If that's what you'd said in the first place, nobody would
have argued with you. However, this is what you said:

It seems there are quite a few projects aimed to improve Python's
speed and, therefore, eliminate its main limitation for mainstream
acceptance.

If you're going to make a claim that speed is Python's main limitation,
you'll need to provide evidence to support your assertion.

I didn't want to start a silly thread to discuss wether python is slow
or not. It is. Period.

Nope. Speed is relative. Is a car slow? Depends whether you're trying
to go five feet, five miles, fifty miles, five hundred miles, or five
thousand miles -- and it also depends on the alternatives available for
traversing that distance. Repeatedly claiming that Python is slow as an
absolute statement only shows your ignorance and unwillingness to listen
to other people.

All the other comments on this thread, regarding the main limitation
for python's acceptance are true, but speed is critical, and this is
were all efforts should be concentrated.

That's nice. If that's your belief, what are you doing to work on this?

So my intention when I started this thread, was to learn more about
these projects, read other oppinions, get some news, etc... I didn't
mean to start a a nonsense discussion!

Believe me, I'm sympathetic to the fact that English is not a comfortable
language for you. However, at this point, the problem appears to be a
thinking problem, not a language problem. You simply hold an opinion at
odds with the experience of many expert Python programmers, and it's one
that is almost guaranteed to start an argument.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"To me vi is Zen. To use vi is to practice zen. Every command is a
koan. Profound to the user, unintelligible to the uninitiated. You
discover truth everytime you use it." (e-mail address removed)

 

[Paul Rubin]

Most Linux distributions include python as standard, so the users only
need to download your script (which will probably be smaller than the
equivalent C code).

IMO, mainstream applications are applications that are used by a big
chunk of the general public. That can include server side
applications (e.g. I'd consider Google web search to be a mainstream
application) as well as stuff like MS Office. The server side stuff
can of course run on Linux or anything else the implementers chose.
But I wouldn't consider any Linux client side application to be
mainstream. Much as I wish it were otherwise, there just aren't that
many people running Linux on their home computers.

 

[Tim Churches]

That's true. If that's what you'd said in the first place, nobody would
have argued with you. However, this is what you said:

It seems there are quite a few projects aimed to improve Python's
speed and, therefore, eliminate its main limitation for mainstream
acceptance.

Hey, cut the boy/girl some slack. I suspect what s/he meant to say was:

It seems there are quite a few projects aimed to improve Python's
speed and, therefore, eliminate its main perceived limitation for
mainstream acceptance.

I think you'll find it much harder to take issue with that statement. I
can only offer anecdotal evidence, but a typical conversation about our
Python-based projects goes like this (where the conversant typically
thinks that there are only two viable languages for any software project
these days: C# or Java):

a: "What language are you using?"
b: "Python"
a: "Um, isn't that too slow?"
b: "In general, no, and anyway there is a thing which is a bit like a
JIT compiler available, and for the really speed-critical parts, you can
use C modules very easily."
a: "Oh, OK, sound complex."
b: "No, it's not, really, and Python is really fast to write."
a: "But I have to write all those C modules."
b: "No, hardly ever, really."

And so on. The **perceived** speed of Python is indeed a barrier to its
acceptance, and indignant posts on this list won't do much to dispel
that perception. More published case studies and benchmarks of Python
versus whatever else when used for **complex**, real-life problems, not
artificially simple looping benchmarks, would help.

--

Tim C

PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere
or at http://members.optushome.com.au/tchur/pubkey.asc
Key fingerprint = 8C22 BF76 33BA B3B5 1D5B EB37 7891 46A9 EAF9 93D0

 

[Arthur Rambo]

OK guys.

Now that you've debated at length of the first question addressed by
Neuruss (Python limitations for mainstream acceptance), why not come
back to the other aspects addressed:

- dynamic languages getting closer to static languages in terms of
speed
- the future of Psyco, Pypy, Starkiller, Ironpython and other similar
projects.

What do you think? I'm sure many of us are interested in those issues
(didn't the widely respected Jim Hugunin mainly advertise IronPython
through its relative speed to C Python? Cf. the page title and first
item on http://ironpython.com).

Whatever the reasons for which making Python faster is desirable or
not, it represents a very interesting technical challenge. Do you
think those projects will succeed? Which other techniques do you see
as promising?

Arthur

 

[Baalbek]

Nope. Speed is relative. Is a car slow? Depends whether you're trying
to go five feet, five miles, fifty miles, five hundred miles, or five
thousand miles -- and it also depends on the alternatives available for
traversing that distance. Repeatedly claiming that Python is slow as an
absolute statement only shows your ignorance and unwillingness to listen
to other people.

Python on today's machines probably runs a lot faster than C programs
did on machines 20 years ago (or even 10 years ago).

Baalbek

 

[Neuruss]

I love Python. It's an excellent, fun, clear, well designed language

That's true. If that's what you'd said in the first place, nobody would
have argued with you...

I already replied a similar comment in my previous post.

If you're going to make a claim that speed is Python's main limitation,
you'll need to provide evidence to support your assertion.

Same as above.

Nope. Speed is relative. Is a car slow? Depends whether you're trying
to go five feet, five miles, fifty miles, five hundred miles, or five
thousand miles -- and it also depends on the alternatives available for
traversing that distance. Repeatedly claiming that Python is slow as an
absolute statement only shows your ignorance and unwillingness to listen
to other people.

I got your point and I used those lines myself many times.
I'm not making an absolute statement as though I'm a fundamentalist
fanatic of speed. What I mean, is that it is widely known that
scripting languages have a considerably lack of speed compared with
static typed languages (I didn't discover anything new, and this is
what I mean when I repeated the statement that python is not fast).
Again, as I already explained, I'm not saying that other people's
oppinion are wrong, when I used the word "nonsense" I wanted to say
that the whole discussion is in which we were all involved is
nonsense, because there's no use in discussing whether python is slow
or not. This is a proven fact.

Your arguments regarding the relativity of speed are correct and
nobody is denying it. But I am talking about this little percentage of
applications, that use to be the most relevant (usually) in enterprise
environments or in commercial applications where high volumes of data
are processed, as well as 3d programs, games, etc..).
My point is:
It is my humble oppinion (and I already acknowledged that I SHOULD
HAVE USED THIS TERMS BEFORE), that this is the main reason that
explains the little acceptance of Python in certain professional
circles.

Now, you can agree with me or not. In that case, your oppinion is
welcome, but I don't need a moral lesson as to how I should address
these topics.
I guess I already said what I have to say about my oppinions.

If I hurted someone I'm sorry!
I hope that we can move on with this topic and you can stop dissecting
my phrases looking for something new to disavow...

That's nice. If that's your belief, what are you doing to work on this?

Again, I already replied this.

Believe me, I'm sympathetic to the fact that English is not a comfortable
language for you. However, at this point, the problem appears to be a
thinking problem, not a language problem. You simply hold an opinion at
odds with the experience of many expert Python programmers, and it's one
that is almost guaranteed to start an argument.

I don't know how to comment on this without being recursive.