Accessing c objects in python

G

Gary

Hi-

I've been searching the web for a while and I've been unable to find a
way to access c data objects in python without using SWIG. I can do
methods just fine but I can't access variables. It seems like
PyModule_AddObject should work but it segfaults my program. There's a
good chance I'm just using it wrong, or I should be using something
else. Could someone give me a simple, complete example pretty please?
Thanks!

Gary
 
D

Dave Kuhlman

Gary said:
Hi-

I've been searching the web for a while and I've been unable to
find a
way to access c data objects in python without using SWIG. I can
do
methods just fine but I can't access variables. It seems like
PyModule_AddObject should work but it segfaults my program.
There's a good chance I'm just using it wrong, or I should be
using something
else. Could someone give me a simple, complete example pretty
please?
Thanks!


One way to solve this problem is to implement a new type for
Python that "wraps" your C object. Then your Python code can
create and manipulate instances of this new Python type. These
Python objects in turn manipulate your C object/type.

If that sounds like what you want to do, then read:

http://www.python.org/doc/current/ext/defining-new-types.html

In particular, look at:

http://www.python.org/doc/current/ext/node22.html

It contains an example that you can start with. Beginning by
replacing "Noddy" with whatever you want to call your new data type.

Then look at the struct at the top of that example. Put (a
pointer to) you C object in that struct.

Now modify the functions Noddy_new, Noddy_dealloc, and Noddy_init
(which you have re-named, whatever).

Next, add functions to get and set things in your C type.

.... and keep on following the examples.

Is there an easier way? I'm not sure, but I'd certainly read the
following section of the Pyrex documentation. It's about how to
wrap C structures:

http://ldots.org/pyrex-guide/3-structures.html

And, the part in the following about external extension types in
Pyrex seems particularly appropriate:

http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/version/Doc/extension_types.html

Any Pyrex experts listening in? Am I on the right track here?

Dave
 
T

Thomas Heller

Alex Martelli said:
Dave Kuhlman wrote:
...
[snipped: C API usage directions]
Is there an easier way? I'm not sure, but I'd certainly read the
following section of the Pyrex documentation. It's about how to
wrap C structures:

http://ldots.org/pyrex-guide/3-structures.html ...
Any Pyrex experts listening in? Am I on the right track here?

I'm no pyrex expert, but I recently re-evaluated various options
for wrapping large existing C libraries: C API, Boost Python,
Pyste, pyrex, ctypes, and Swig. Somewhat to my surprise (I had
bad memories about it from the past...), Swig came up on top in
terms of simplicity. Pyste might easily be best for _C++_
libraries, already organized into classes and possibly using
templates &c -- it, in turn, "rests" om Boost Python. Boost
itself -- or pyrex if you'd rather not use C++ for the purpose --
might likely be best if you want to code some significant amount
of fast logic, not just wrappers. But for "just wrappers", and
C libraries as opposed to C++ ones, Swig rules.

I'd use the C API directly only if the other tools were ruled
out for some reason (e.g., licensing); it's by far the largest
amount of work -- all other tools rest on top of it and can be
seen as "code generators" for it that save you aggravation.

Would you care to write a more detailed summary of your findings? This
would probably make a good page in the main Python wiki...

Thomas
 
D

Duncan Booth

Thomas Heller said:
Would you care to write a more detailed summary of your findings? This
would probably make a good page in the main Python wiki...

I did an evaluation of the various ways to link to Python for the ACCU
conference this year, although I must say my conclusion was slightly
different than Alex's: I would use Pyrex in preference to SWIG even if I
didn't want to embed much code in the interface.

You can find a paper describing my views at:

http://dales.rmplc.co.uk/Duncan/accu/integratingpython.html

The slides and source code for the talks are also on the same site. See
http://dales.rmplc.co.uk/Duncan/accu
 
T

Thomas Heller

Duncan Booth said:
I did an evaluation of the various ways to link to Python for the ACCU
conference this year, although I must say my conclusion was slightly
different than Alex's: I would use Pyrex in preference to SWIG even if I
didn't want to embed much code in the interface.

You can find a paper describing my views at:

http://dales.rmplc.co.uk/Duncan/accu/integratingpython.html

The slides and source code for the talks are also on the same site. See
http://dales.rmplc.co.uk/Duncan/accu

Interesting. I added these links to the Python wiki.

Thomas
 
A

Alex Martelli

Duncan Booth wrote:
...
I did an evaluation of the various ways to link to Python for the ACCU
conference this year, although I must say my conclusion was slightly
different than Alex's: I would use Pyrex in preference to SWIG even if I
didn't want to embed much code in the interface.

Interesting. I do agree with your observation that, with SWIG,
responsibility for allocating and freeing tends to show up at the
Python level. However, SWIG generates such a Python level itself,
as it wraps 'foo' into a low-level _foo.so and a higher-level foo.py.

Thus, the _USER_ of the overall wrapper need not be concerned with
memory allocation and freeing -- unless he WANTS to be, for performance
reasons, in which case, with SWIG, he gets that *option*. That, in
summary, is the epiphany I had on this latest round of encounter with
SWIG, which made me chose it to complete the wrapping in preference
to Boost and pyrex: the python user of the wrapper could use it in
the simplest and safest of ways _OR_, when performance mattered a
lot (as it might in some cases for these wrappings, which dealt with
multimedia-related libraries), the user could also CHOOSE to take
more direct control -- e.g. explicitly allocate a buffer at the start
of a loop, reuse the buffer N times in the loop, and free it after
the loop, rather than rely on automatic allocation and freeing and
thus pay the resulting performance cost -- in that spot of two in a
huge program where such a strategy pays off.

"Trust the programmer" and "don't stop the programmer from doing
what needs to be done" are two of the "spirit of C" principles (from
the Rationale of the C standard) which Python respects (more so, in
fact, than the other principle from the same source that is so often
quoted, "provide only one way to perform a task"). With SWIG I get
such an arrangement "almost by default" -- and I think that's quite
different from saying that the python user "has" to deal with the
issues of memory management... it is, rather, a _choice_ he or she
gets, just about "for free" from the POV of the guy who is authoring
(and those who in the future will be maintaining) the wrapper.

I could, of course, get the same choice for the user by coding in
Boost -- but then, since I'm wrapping a C library [and not a C++
one with classes already provided and decently well designed], I'd
have a lot more boilerplate-coding to do myself. (I could probably
concoct incredibly subtle C++ templates with some macros in support,
to lighten my own boiler-plate coding task -- but I fear that would
leave far too heavy a maintenance burden on future maintainers of
those wrappers).

I suspect I could most likely get the same result in pyrex, too -- but
then, again, I'd have to code a lot of boilerplate, _without_ even the
help of C++ templates, OR even of C-preprocessor macros... that's too
much raw boilerplate for me;-) Quite possibly I haven't grasped "the
pyrex way to metaprogramming" (just as I guess I hadn't grasped the
SWIG one on earlier encounters with it -- Boost never gave me any
trouble in this regard, as I already understood C++ templates & the
like in sufficient depth before I met it).

I'd better underline yet again that this applies to _wrapping C libraries_!
I do NOT claim I have any real grasp yet about the SWIG way wrt C++
(templates in particular) -- when I next need to wrap a C++ library, I'll
reach for Boost and Pyste, without doubts; and if I ever need to provide
Python users with very speedy code, psyco can't supply it (e.g. because I
need to target non-Intel-like CPU's), and Boost is not applicable (e.g.
because I can't require future maintainers to know C++), I'm most
definitely going to give pyrex another chance -- apart from the boilerplate
issues (==lack of [or insufficient grasp of mine for] metaprogramming and
code generation facilities), I found it just delightful.


Alex
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
474,083
Messages
2,570,591
Members
47,212
Latest member
RobynWiley

Latest Threads

Top