speeding things up with C++

[bullockbefriending bard]

I've done all the requisite profiling and thought fairly deeply about
the efficiency of my python code, but am still going to have to speed
up the innermost guts of what I am doing.

Essentially, I need to pass a list of 6-tuples containing only
integers to my new sadly necessary super-fast compiled language
function which i am not looking forward to writing:

input: [(1,2,3,4,5,6), (7,8,9,10,11,12),...]

and after much thrashing of processor resources, return data which
looks like this to the Python calling environment:

output: [( (1, 2), (1,), (12,), (13), (1, 7, 11), (9,) ), ( another
nested tuple like preceding one ), .... ]

Each member of the returned list is a tuple containing 6 tuples, and
each of those 6 tuples has at least one integer member. It would also
be acceptable for the return values to be entirely nested lists
instead of having the two innermost sequence types as tuples.

I probably want to be using something like C++ because i'm going to
need to use STL vectors and sets for what i'm doing in the algorithm
i'm trying to speed up. IIRC Boost tuple is a bit painful for more
than 10 elements + isn't really essential that I use a a non-mutable
data type in what will be a small encapsulated bit of a much larger
system.

Anyway, I can probably very quickly figure out some hacked way to get
the data into my function given that in the worst case I could take
advantage of the knowledge that each input tuple always has 6
elements, and simply pass in a big array of ints. Yes, I know this is
a bit retarded, but I'm talking worst case assuming on very tight
schedule and no time to delve deeply into SWIG or whatever. Similarly
it wouldn't be too difficult to return the result as the mother all of
all strings which i could then parse fairly easily.

However, I hope someone reading this will be able to tell me that I'm
being a total pessimist and that in fact it isn't very difficult to do
what I want to do using SWIG. I'm not asking for a complete solution,
more like some general pointers from someone who has actually done
something similar before.

As an added bonus, I wouldn't if this is 'easily' doable using Ocaml
as the guts instead of C++, I'd be happy to know about it.

 

[Che Guevara]

However, I hope someone reading this will be able to tell me that I'm
being a total pessimist and that in fact it isn't very difficult to do
what I want to do using SWIG. I'm not asking for a complete solution,
more like some general pointers from someone who has actually done
something similar before.

I do stuff like that with ctypes ( http://docs.python.org/lib/module-ctypes.html
)!
I'm sure you can figure out some way to pass/return data to/from your C
++ lib.
The easiest way - use strings or arrays, but if you dig deeper into
ctyles documentation,
I'm sure you will find a better solution!

Good luck,
-- misreckoning

 

[bullockbefriending bard]

thanks! i'll look into this.

I do stuff like that with ctypes (http://docs.python.org/lib/module-ctypes.html
)!
I'm sure you can figure out some way to pass/return data to/from your C
++ lib.
The easiest way - use strings or arrays, but if you dig deeper into
ctyles documentation,
I'm sure you will find a better solution!

Good luck,
-- misreckoning

 

[bullockbefriending bard]

I wonder if Jython might be the answer? Java is going to be faster
than Python for the time-critical part of my program. Does anybody
have experience getting data structures like nested lists / tuples
into a java routine from a running jython program (and then back
again)?

 

[Kay Schluehr]

I've done all the requisite profiling and thought fairly deeply about
the efficiency of my python code, but am still going to have to speed
up the innermost guts of what I am doing.

Essentially, I need to pass a list of 6-tuples containing only
integers to my new sadly necessary super-fast compiled language
function which i am not looking forward to writing:

input: [(1,2,3,4,5,6), (7,8,9,10,11,12),...]

and after much thrashing of processor resources, return data which
looks like this to the Python calling environment:

output: [( (1, 2), (1,), (12,), (13), (1, 7, 11), (9,) ), ( another
nested tuple like preceding one ), .... ]

Each member of the returned list is a tuple containing 6 tuples, and
each of those 6 tuples has at least one integer member. It would also
be acceptable for the return values to be entirely nested lists
instead of having the two innermost sequence types as tuples.

I probably want to be using something like C++ because i'm going to
need to use STL vectors and sets for what i'm doing in the algorithm
i'm trying to speed up. IIRC Boost tuple is a bit painful for more
than 10 elements + isn't really essential that I use a a non-mutable
data type in what will be a small encapsulated bit of a much larger
system.

Anyway, I can probably very quickly figure out some hacked way to get
the data into my function given that in the worst case I could take
advantage of the knowledge that each input tuple always has 6
elements, and simply pass in a big array of ints. Yes, I know this is
a bit retarded, but I'm talking worst case assuming on very tight
schedule and no time to delve deeply into SWIG or whatever. Similarly
it wouldn't be too difficult to return the result as the mother all of
all strings which i could then parse fairly easily.

However, I hope someone reading this will be able to tell me that I'm
being a total pessimist and that in fact it isn't very difficult to do
what I want to do using SWIG. I'm not asking for a complete solution,
more like some general pointers from someone who has actually done
something similar before.

As an added bonus, I wouldn't if this is 'easily' doable using Ocaml
as the guts instead of C++, I'd be happy to know about it.

A just too obvious recommendation is to drop C++ and use D together
with the Pyd/celerid bridge. Yeah, it sounds a little esoteric but D
is really the Python among the statically typed imperative languages.

http://www.digitalmars.com/d/
http://pyd.dsource.org/index.html

 

[bullockbefriending bard]

thanks. i'll definitely look into this.

I've done all the requisite profiling and thought fairly deeply about
the efficiency of my python code, but am still going to have to speed
up the innermost guts of what I am doing.

Essentially, I need to pass a list of 6-tuples containing only
integers to my new sadly necessary super-fast compiled language
function which i am not looking forward to writing:

input: [(1,2,3,4,5,6), (7,8,9,10,11,12),...]

and after much thrashing of processor resources, return data which
looks like this to the Python calling environment:

output: [( (1, 2), (1,), (12,), (13), (1, 7, 11), (9,) ), ( another
nested tuple like preceding one ), .... ]

Each member of the returned list is a tuple containing 6 tuples, and
each of those 6 tuples has at least one integer member. It would also
be acceptable for the return values to be entirely nested lists
instead of having the two innermost sequence types as tuples.

I probably want to be using something like C++ because i'm going to
need to use STL vectors and sets for what i'm doing in the algorithm
i'm trying to speed up. IIRC Boost tuple is a bit painful for more
than 10 elements + isn't really essential that I use a a non-mutable
data type in what will be a small encapsulated bit of a much larger
system.

Anyway, I can probably very quickly figure out some hacked way to get
the data into my function given that in the worst case I could take
advantage of the knowledge that each input tuple always has 6
elements, and simply pass in a big array of ints. Yes, I know this is
a bit retarded, but I'm talking worst case assuming on very tight
schedule and no time to delve deeply into SWIG or whatever. Similarly
it wouldn't be too difficult to return the result as the mother all of
all strings which i could then parse fairly easily.

However, I hope someone reading this will be able to tell me that I'm
being a total pessimist and that in fact it isn't very difficult to do
what I want to do using SWIG. I'm not asking for a complete solution,
more like some general pointers from someone who has actually done
something similar before.

As an added bonus, I wouldn't if this is 'easily' doable using Ocaml
as the guts instead of C++, I'd be happy to know about it.

A just too obvious recommendation is to drop C++ and use D together
with the Pyd/celerid bridge. Yeah, it sounds a little esoteric but D
is really the Python among the statically typed imperative languages.

http://www.digitalmars.com/d/http://pyd.dsource.org/index.html

 

[Jorgen Grahn]

Essentially, I need to pass a list of 6-tuples containing only
integers to my new sadly necessary super-fast compiled language
function which i am not looking forward to writing:

input: [(1,2,3,4,5,6), (7,8,9,10,11,12),...]

and after much thrashing of processor resources, return data which
looks like this to the Python calling environment:

output: [( (1, 2), (1,), (12,), (13), (1, 7, 11), (9,) ), ( another
nested tuple like preceding one ), .... ] ....
However, I hope someone reading this will be able to tell me that I'm
being a total pessimist and that in fact it isn't very difficult to do
what I want to do using SWIG.

You're talking about the actual conversion between Python data
structures and C or C++ data structures? That is easy to do even
manually, IMHO -- provided a decent C background.

Have a look in the Python/C API Reference Manual, and the mapping
becomes clear. The PyListObject stuff for example, where there's a C
function for every basic operation on lists, and where the elements
have the C type PyObject. And so on. Mapping to C is just a matter of
walking a nested data structure, where you have a good idea what it is
supposed to look like (a list of six-tuples of some kind of numbers).

/Jorgen

 

[bullockbefriending bard]

Thanks this is good news. I think my C/C++ background is sufficient to
manage to figure things out if I RTFM carefully.

Basically I want to pass in a Python list of integer tuples, create an
STL container full of equivalent tuples, apply some processor-
intensive algorithm to said list of tuples, and finally poke the
results back into another Python list of integer tuples and return it
to the calling Python environment. Data structures are well-defind and
simple, and the most complex case would be 3-deep nested list, so I
will seriously consider figuring out how to do it manually as you
suggest.

Essentially, I need to pass a list of 6-tuples containing only
integers to my new sadly necessary super-fast compiled language
function which i am not looking forward to writing:

input: [(1,2,3,4,5,6), (7,8,9,10,11,12),...]

and after much thrashing of processor resources, return data which
looks like this to the Python calling environment:

output: [( (1, 2), (1,), (12,), (13), (1, 7, 11), (9,) ), ( another
nested tuple like preceding one ), .... ] ...
However, I hope someone reading this will be able to tell me that I'm
being a total pessimist and that in fact it isn't very difficult to do
what I want to do using SWIG.

You're talking about the actual conversion between Python data
structures and C or C++ data structures? That is easy to do even
manually, IMHO -- provided a decent C background.

Have a look in the Python/C API Reference Manual, and the mapping
becomes clear. The PyListObject stuff for example, where there's a C
function for every basic operation on lists, and where the elements
have the C type PyObject. And so on. Mapping to C is just a matter of
walking a nested data structure, where you have a good idea what it is
supposed to look like (a list of six-tuples of some kind of numbers).

/Jorgen

 

[Chris Mellon]

Thanks this is good news. I think my C/C++ background is sufficient to
manage to figure things out if I RTFM carefully.

Basically I want to pass in a Python list of integer tuples, create an
STL container full of equivalent tuples, apply some processor-
intensive algorithm to said list of tuples, and finally poke the
results back into another Python list of integer tuples and return it
to the calling Python environment. Data structures are well-defind and
simple, and the most complex case would be 3-deep nested list, so I
will seriously consider figuring out how to do it manually as you
suggest.

Are you sure you want an STL container? Since the primary operator
here is Python, the extra benefits from the STL container over plain C
arrays isn't as evident.

Pyrex is a good way to write the interface between your C++ code and
the Python code - it handles the refcounting and boilerplate for you -
and perhaps for writing the algorithms as well, depending on how
complicated and performance sensitive they are.

Also, using numeric/Numarray can be a very big win. It can potentially
save you a fair amount of marshalling overhead.

 

[bullockbefriending bard]

Are you sure you want an STL container? Since the primary operator
here is Python, the extra benefits from the STL container over plain C
arrays isn't as evident.

Pyrex is a good way to write the interface between your C++ code and
the Python code - it handles the refcounting and boilerplate for you -
and perhaps for writing the algorithms as well, depending on how
complicated and performance sensitive they are.

good point. while i bow to the genius of the folks who invented
template metaprogramming, the compiler error messages tend to be
profoundly depressing . one way or the other, pyrex is something i
need to learn since i'm now completely enamoured with python and had
better develop an arsenal of tricks for the rare times when it's just
not fast enough.

Also, using numeric/Numarray can be a very big win. It can potentially
save you a fair amount of marshalling overhead.

as i understand it, this is so for applying the likes of matrix
operations, autocorrelations, FFTs, etc...where python essentially
provides scripting glue to some highly optimised C functions. i'm
assuming that the kind of algorithm i am looking at which involves
some set operations on list elements + copying between lists isn't
going to be helped so much by using numpy or similar.

 

[Max M]

bullockbefriending bard skrev:

good point. while i bow to the genius of the folks who invented
template metaprogramming, the compiler error messages tend to be
profoundly depressing . one way or the other, pyrex is something i
need to learn since i'm now completely enamoured with python and had
better develop an arsenal of tricks for the rare times when it's just
not fast enough.

A dash of c combined integrated via ctypes is probably the easiest solution?


--

hilsen/regards Max M, Denmark

http://www.mxm.dk/
IT's Mad Science

 

[Jorgen Grahn]

Are you sure you want an STL container? Since the primary operator
here is Python, the extra benefits from the STL container over plain C
arrays isn't as evident.

STL containers are easier to use, harder to misuse and take care of
memory allocations. Wouldn't you say that's a benefit? If I wrote an
algorithm in C++, I'd rather pass it a const std::vector<double>& than
a const double * and a length.

That said, I prefer C for simple extension modules which just wrap
something. Less dependencies for the person who builds it, more people
who understand it well enough to change it.

/Jorgen