Writing code to be optimizable

[snorble]

Sometimes I want to prototype a program in Python, with the idea of
optimizing it later by rewriting parts of it in C or Cython. But I
usually find that in order to rewrite the slow parts, I end up writing
those parts very much like C or C++ anyway, and I end up wondering
what is the point of using Python in such a project.

I liked the idea of Cython, mainly the idea of being able to write in
pure Python, then create a file that modifies the pure Python file
(with type declarations and such), but I ran into the same problems.
In order to be able to modify the code, I can't make a lot of use of
the really helpful things in Python like dicts and such. I have to
dumb down Python to where I'm basically writing C code in Python, so
again I ask myself what is the point?

Is it reasonable to prototype an application in Python that will
require performance? Are there any recommendations on how to write
code in such a way that it can later be optimized or replaced with a
module written in C or Cython? Or any good examples of this being done
that I could learn from?

 

[Chris Angelico]

Sometimes I want to prototype a program in Python, with the idea of
optimizing it later by rewriting parts of it in C or Cython. But I
usually find that in order to rewrite the slow parts, I end up writing
those parts very much like C or C++ anyway, and I end up wondering
what is the point of using Python in such a project.

There's a few ways to prototype. If you use Python as the language of
your specs documents (or pseudo-Python, perhaps) - consider the entire
Python implementation as ephemeral, and use it to verify that your
specifications are correct, but not for any sort of performance - then
it doesn't matter that you've rewritten it completely, nor that you've
written C code in Python. You would have been writing C code in
English otherwise, anyway.

If you're not sure which parts you're prototyping (ie will rewrite in
C/C++) and which parts you're writing properly (ie will keep the
Python version as the production code), I'd still recommend using all
of Python's best features... but keep your function docstrings
comprehensive. When you rewrite it in C, you're able to take advantage
of what you learned first time around, but otherwise it's just
strictly reimplementing the docstring in a different environment.

Chris Angelico

 

[Stefan Behnel]

snorble, 23.11.2011 06:19:

Sometimes I want to prototype a program in Python, with the idea of
optimizing it later by rewriting parts of it in C or Cython. But I
usually find that in order to rewrite the slow parts, I end up writing
those parts very much like C or C++ anyway, and I end up wondering
what is the point of using Python in such a project.

I liked the idea of Cython, mainly the idea of being able to write in
pure Python, then create a file that modifies the pure Python file
(with type declarations and such), but I ran into the same problems.
In order to be able to modify the code, I can't make a lot of use of
the really helpful things in Python like dicts and such. I have to
dumb down Python to where I'm basically writing C code in Python, so
again I ask myself what is the point?

Is it reasonable to prototype an application in Python that will
require performance? Are there any recommendations on how to write
code in such a way that it can later be optimized or replaced with a
module written in C or Cython?

I think the usual approach is to write it in Python and make sure it works
by writing "enough" tests, then benchmark it, then decide if a non-Python
level optimisation is required.

If it's required, Cython compile the entire module and add static types to
your hot spots, either in Python notation ("pure mode") or in an external
..pxd file. If that doesn't yield enough performance, copy code into a
separate Cython module and optimise it there using C paradigms instead of
Python paradigms, i.e. apply algorithmic optimisations by dropping data
structures into C. Then use an import to use the code from your so-called
accelerator module in your original module. Try to provide both a pure
Python implementation and a Cython implementation, as that will allow you
to run your code in other Python implementations directly and to compare
your two implementations for equivalence and performance.

The advantage of stepping down into C-ish Cython code incrementally is that
you will have a working implementation at each step and can decide to stop
optimising because it is "good enough". If you started 'optimising' your
code while still writing it, it will take longer to write it and you can
never be sure that the complexity you add to the long-term maintenance by
writing C-ish code is truly worth the performance gain (or if there even is
any gain).

Stefan

 

[Roy Smith]

Is it reasonable to prototype an application in Python that will
require performance?

Yes. Several observations:

1) The classic 80/20 rule. 80% of the time is spent running 20% of the
code. Sometimes quoted as 90/10. Why waste time optimizing the 80%?

2) Surprisingly to many people, it's difficult to predict in advance
which 20% will be the slow parts. Don't sweat it until the whole thing
is up and running and you can begin to gather profiling data.

3) Often enough, when you're done writing your program, you'll discover
that it's fast enough to get value from. Be happy and move onto the
next task on your list.

Are there any recommendations on how to write code in such a way that
it can later be optimized or replaced with a module written in C or
Cython?

Sure. Make your code modular and loosely coupled. Each module or class
should do one thing, and have narrow, well-defined interfaces. Limit
the number of other modules or classes it interacts with directly. If
you've done that, it becomes easier to a) identify the slow parts and b)
plug something better in its place.