Exposing buffer interface for non-extension types?

[Ken Watford]

Is there any way to expose the PEP 3118 buffer interface for objects
that aren't extension types?

Currently, I can expose the NumPy array interface (using either
__array_interface__ or __array_struct__) for any class, extension or
otherwise. But I can't find any reference to python-side interfacing
for PEP 3118. SWIG makes an extension module for your wrapped code,
but not extension *types*, so the classes it produces are pure-python
with methods added in from the extension module.

The NumPy array interface works fine for now (especially since NumPy
is the only thing I need to consume these objects), but the
documentation claims that it's being deprecated in favor of PEP 3118,
so I thought it might be relevant to bring this up.

 

[Carl Banks]

Is there any way to expose the PEP 3118 buffer interface for objects
that aren't extension types?

Currently, I can expose the NumPy array interface (using either
__array_interface__ or __array_struct__) for any class, extension or
otherwise. But I can't find any reference to python-side interfacing
for PEP 3118. SWIG makes an extension module for your wrapped code,
but not extension *types*, so the classes it produces are pure-python
with methods added in from the extension module.

The NumPy array interface works fine for now (especially since NumPy
is the only thing I need to consume these objects), but the
documentation claims that it's being deprecated in favor of PEP 3118,
so I thought it might be relevant to bring this up.

Can you tell let us know what you want to use it for? We could offer
better help.

Numpy is generally how I get at buffers in Python 2.x. For instance
if I have an object m that supports buffer protocol (m could a string,
mmap object, Python array, etc.), then the following will create an
array view of the same buffer:

numpy.ndarray((10,10),type=numpy.float32,buffer=m)

As far as I know this procedure won't be too different under PEP 3118;
if anything it's simplified in Python 3 since it can discover type and
shape information itself. (You'll have to check with the numpy people
on that.)


Carl Banks

 

[Ken Watford]

Can you tell let us know what you want to use it for?  We could offer
better help.

Numpy is generally how I get at buffers in Python 2.x.  For instance
if I have an object m that supports buffer protocol (m could a string,
mmap object, Python array, etc.), then the following will create an
array view of the same buffer:

numpy.ndarray((10,10),type=numpy.float32,buffer=m)

As far as I know this procedure won't be too different under PEP 3118;
if anything it's simplified in Python 3 since it can discover type and
shape information itself.  (You'll have to check with the numpy people
on that.)

I'm not having trouble using buffers, I'm having trouble providing them.

As a part of SWIG-wrapping a larger C++ project, I'm producing some
wrappings for Blitz++ arrays. I can extract the shape and stride
information from the array object to fill either NumPy's or PEP 3118's
appropriate structure. In the case of NumPy, I can easily arrange for
the necessary interface on the proxy object to be fulfilled, because
NumPy doesn't care what kind of object it's attached to. But the PEP
3118 interface can only be provided by C extension types.

One possibility I've considered is injecting a small extension type
into the wrapper that provides PEP 3118 by reading the NumPy array
interface info off of the object, and then inject it into all
appropriate SWIG-generated proxy classes as an additional base class.

This isn't a big deal for me - the array interface works just fine,
and probably will for longer than I'll be working on this project -
but it just struck me as strange that some of my existing
array-interface-enabled classes can't be trivially ported to PEP 3118
because they're defined in pure Python modules rather than extension
modules.

 

[Carl Banks]

I'm not having trouble using buffers, I'm having trouble providing them.

I see.

As a part of SWIG-wrapping a larger C++ project, I'm producing some
wrappings for Blitz++ arrays. I can extract the shape and stride
information from the array object to fill either NumPy's or PEP 3118's
appropriate structure. In the case of NumPy, I can easily arrange for
the necessary interface on the proxy object to be fulfilled, because
NumPy doesn't care what kind of object it's attached to. But the PEP
3118 interface can only be provided by C extension types.

Well, let's get the facts straight before proceeding. The buffer
protocol has always been for C only. PEP 3118 is an extension to this
buffer protocol, so it's also C only. Numpy's array interface is not
the buffer protocol, and is specific to numpy (though I guess other
types are free to use it among themselves, if they wish). So what
you've been doing was *not* to provide buffers; it was to provide some
numpy-specific methods that allowed numpy to discover your buffers.

That said, your complaint is reasonable. And since the numpy docs say
"use PEP 3118 instead of array interface" your misunderstanding is
understandable.

I assume it's not practical to address your real problem (i.e., SWIG),
so we'll instead take it for granted that you have a block of memory
in some custom C code that you can't expose as a real buffer. In the
past, you've used numpy's special Python methods to let numpy know
where your memory block was, but what do you do now?

Your suggestion to write a small base class is probably the best you
can do as things stand now. Another thing you can do is write a small
extension to create a surrogate buffer object (that implements buffer
protocol), so if you want to create an ndarray from a buffer you can
do it like this:

b = create_surrogate_buffer(pointer=_proxy.get_buffer_pointer())

numpy.ndarray(shape=_proxy.get_shape(),type=_proxy.get_type(),buffer=b)

Not as convenient to use, but might be simpler to code up than the
base-class solution.

The numpy developers might be sympathetic to your concerns since you
have a reasonable use case (they might also tell you you need to get a
real wrapper generator). Just be sure to keep the distinction clear
between "buffer protocol" (which is what PEP 3118 is, is C only, and
the buffer must be owned by a Python object) and "array
interface" (which is what you were using for 2.x, is C or Python, and
the buffer can be anywhere).

HTH


Carl Banks

 

[Stefan Behnel]

Ken Watford, 21.07.2010 03:04:

>>
>> Try using Cython instead, it has native support for the buffer protocol.

>
> I've used Cython before, and I generally like it. But its purpose is
> slightly different than SWIG's, and does not particularly meet my
> current project's needs.
[...]
As a part of SWIG-wrapping a larger C++ project, I'm producing some
wrappings for Blitz++ arrays.

Right, support for C++ templates in Cython isn't as good yet as it could
be, although there has been some work in that direction. Contributions are
welcome.

However, note that you can write the tiny bit of plain C++ glue code
necessary to access the Blitz++ arrays, and then use Cython to wrap that
and to implement the buffer interface and all the other wrapper code. C++
support by itself is pretty good in Cython 0.13. (to be released soon, get
it from here: http://hg.cython.org/cython-closures )

Stefan