Python is slow?

[Robert Singer]

1. gfortran is not Absoft.

I find this comment absurd. What did you mean by it ?
Yes, gfortran is not Absoft, just as red is not blue (?!).

I also don't understand whether you're looking for a free or a
commercial compiler. I got the impression from your previous post that
money was an object.

2. If I program the same in C99 and Fortran 95, and compile with gcc
and gfortran, the C99 code runs a lot faster (I've only tested with
wavelet transforms).

Hmm. Unfortunatelly, i have none whatsoever experience in that field.
Completely different area of study here, so without seeing at least
some code, I cannot comment anything on that part.

And 'a lot faster' is a very relative term.

3. gfortran is not Absoft.

True.

Best regards
Bob

 

[sturlamolden]

Is this a good or bad thing? ;-)

It seems we have been implementing different algorithms. kd-trees are
not BK-trees.

http://www.scipy.org/Cookbook/KDTree

 

[Robert Kern]

Uh, actually we're talking about kd-trees, not BK-trees. kd-trees are
for searching through point sets in a k-dimensional space.

My apologies. I did not actually follow bearophile's link, and thought he was
talking about kd-trees like Sturla was.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[bearophileHUGS]

sturlamolden:

It seems we have been implementing different algorithms. kd-trees are
not BK-trees.
http://www.scipy.org/Cookbook/KDTree

Sorry for my silly mistake

Note: in your code I don't know if the collections.deque data
structure may help (it's faster than list for appending), but I
presume that's not a bottleneck.

Bye,
bearophile

 

[Eric Brunel]

[...]

Would it be possible to post this text to some "persistent" web page with
(links to) the code you wrote in both languages? This would be a very
interesting resource for people experiencing some resistence when they
suggest Python as a possible language because of the 'Python is slow'
myth...

 

[sturlamolden]

For those who are interested:

I've updated the cookbook tutorial on the kd-tree:

http://scipy.org/Cookbook/KDTree

It now also includes parallel search for multicore CPUs
(multiprocessing standard module). Even if you are not genuinely
interested in kd-trees, it shows how to do parallel processing in
Python despite of the GIL.

 

[sturlamolden]

I have updated the cookbook entry for yesterday to also include
parallel processing for large data sets. Even if you're not interested
in kd-trees, it is a good example of what the new multiprocessing
standard module can do. There are still people being scared by the
GIL, thinking it prevents Python from utilizing multicore processors.

http://www.scipy.org/Cookbook/KDTree

 

[David Cournapeau]

Not even Matlab, R, IDL, Octave, SciLab, S-PLUS or Mathematica?

I am fairly experienced in matlab (have been using it extensively for
5 years in academical context), and now with numpy, and generally,
they are comparable speed-wise. Matlab has some niceties which makes
it faster in some simple cases (JIT for loops, function calls faster,
sometimes COW semantics means it faster), but numpy (at its core at
least) is much more powerful IMHO. Also, matlab is horrible when you
want to interface some C to it (the C api is basically broken; in
particular, there is no way to gurantee you won't leak memory when you
Ctrl+C custom C extensions because the C api does not have facility to
deal with signals). I totally gave up matlab for numpy 2 years ago,
and never regretted it.

I think speed is not the issue when comparing matlab, R and co.
Availability of functionalities matter much more. R is quite hard to
beat if you need to do advanced statistics, specially since it is the
tool of choice for most academic statisticians. I hope numpy/scipy
will be there sometime, but it is honestly still quite far in that
domain.

cheers,

David

 

[Aahz]

AFAICT, _everybody_ is bad at programming C++.

One begins to suspect it's not the fault of the programmers.

http://www.netfunny.com/rhf/jokes/98/May/stroustrup.html

 

[sturlamolden]

I am fairly experienced in matlab (have been using it extensively for
5 years in academical context), and now with numpy, and generally,
they are comparable speed-wise. Matlab has some niceties which makes
it faster in some simple cases (JIT for loops, function calls faster,
sometimes COW semantics means it faster),

I've used Matlab for 10 years.

Matlab has a horrible pass-by-value semantics, which means that arrays
are copied in and copied out form function calls. Yes there is a copy-
on-write optimization. But unless you are not careful, it will kill
performance and make memory usage skyrocket. Also, slicing creates a
new array, whereas in numpy slicing gives you a view of an existing
array. This generally makes Matlab slow, and it wastes terrible
amounts of memory.

For example I once tested D4 wavelet transforms on a 64 MB array of
doubles (i.e. length 2**23). Matlab R14 Service Pack 2 did this in 27
seconds, whereas Python 2.4 with NumPy 1.0 one required 3.4 seconds.
That is an order of magnitude speed difference in favour of Python.

Matlab also has a strange habit of fragmenting the heap. This can be
so bad that you have to stop the script, run a special function called
pack() do defragement, and restart. I've never seen that with Python.

I more or less stopped using Matlab 3 years ago, and I have not
renewed my subscription for maintenance of my personal Matlab license
since. Matlab is a nice tool, but I have grown past it.

Matlab's strongest side is data visualization though. Although we have
matplotlib, mayavi and possibility of interfacing with gnuplot, it's
not anywhere near the capabilities of Matlab.

 

[Fly Away]

Matlab's strongest side is data visualization though. Although we have
matplotlib, mayavi and possibility of interfacing with gnuplot, it's
not anywhere near the capabilities of Matlab.

What particular Matlab visualization features are you referring to? I
can't think of anything that would justify using the "not anywhere
near" term.

Cheers,
Victor.

 

[Lawrence D'Oliveiro]

In message

... and possibility of interfacing with gnuplot ...

Gnuplot is non-Free software.

 

[Fly Away]

In message


Gnuplot is non-Free software.

Yes, it is.

Victor.

 

[bearophileHUGS]

Lawrence D'Oliveiro:
Fly Away:

Yes, it is.

From:
http://www.gnuplot.info/faq/faq.txt

1.7 Does gnuplot have anything to do with the FSF and the GNU project?
[...]
Gnuplot is freeware in the sense that you don't have to pay for it.
However
it is not freeware in the sense that you would be allowed to
distribute a
modified version of your gnuplot freely. [...]

Bye,
bearophile

 

[Victor Prosolin]

Lawrence D'Oliveiro:
Fly Away:

Yes, it is.

From:
http://www.gnuplot.info/faq/faq.txt

1.7 Does gnuplot have anything to do with the FSF and the GNU project?
[...]
Gnuplot is freeware in the sense that you don't have to pay for it.
However
it is not freeware in the sense that you would be allowed to
distribute a
modified version of your gnuplot freely. [...]

Yes, I did read this prior to posting.

Victor.

 

[r0g]

Lawrence D'Oliveiro:
Fly Away:

Yes, it is.

From:
http://www.gnuplot.info/faq/faq.txt

1.7 Does gnuplot have anything to do with the FSF and the GNU project?
[...]
Gnuplot is freeware in the sense that you don't have to pay for it.
However
it is not freeware in the sense that you would be allowed to
distribute a
modified version of your gnuplot freely. [...]

Bye,
bearophile

Well, ish. You can only distribute modifications to gnuplot itself as
patches, but you can distribute it freely and they publish the source
so, while it's not GPL free it's tending towards it.

Roger.

 

[Lawrence D'Oliveiro]

You can only distribute modifications to gnuplot itself as
patches, but you can distribute it freely ...

This must be some new definition of "freely" of which I'm unaware.

 

[r0g]

This must be some new definition of "freely" of which I'm unaware.

As in beer.

 

[Lawrence D'Oliveiro]

As in beer.

You get free beer?

 

[Steven D'Aprano]

This must be some new definition of "freely" of which I'm unaware.

You're free to distribute the official release of gnuplot.

You're free to distribute patches to gnuplot.

You're even free to provide people with a script or program to apply
those patches to gnuplot.


Where's the non-free bit?


Personally, I don't get the whole "only distribute patches" requirement.
It's a bit like saying "You're free to distribute this software, but only
as a tarball". It seems silly to me. But I don't see it as non-free,
except in the sense that "only licences approved by the FSF are free".