Magic methods in extension types

[Jacek Generowicz]

I am writing an extension type, and wish to add some magic methods
which are not catered for by the tp_<whatever> slots (eg tp_init ->
__init__) in PyTypeObject. My methods seem to work correctly when
invoked explicitly (eg obj.__iadd__(3)) but the method seems not to be
associated with the corresponding operator (ie obj += 3 does NOT
work).

What am I likely to be doing wrong ?

 

[Michael Hudson]

I am writing an extension type, and wish to add some magic methods
which are not catered for by the tp_<whatever> slots (eg tp_init ->
__init__) in PyTypeObject. My methods seem to work correctly when
invoked explicitly (eg obj.__iadd__(3)) but the method seems not to be
associated with the corresponding operator (ie obj += 3 does NOT
work).

What am I likely to be doing wrong ?

Not finding the tp_as_number->nb_inplace_add field?

I think *all* magic methods correspond to slots in (or near) the type
object -- it's practically the definition of "magic method"!

Cheers,
mwh

 

[Jacek Generowicz]

Not finding the tp_as_number->nb_inplace_add field?

( ... or tp_as_sequence ... )

I was afraid you (someone) was going to say that.

I think *all* magic methods correspond to slots in (or near) the type
object -- it's practically the definition of "magic method"!
Hmmm

... def __iadd__(self,other):
... print "__iadd__ running"
... return self
... __iadd__ running

I'd be surprised if I've added __iadd__ to a type object here, yet it
seems to work.

Python manages to map "+=" to the method called "__iadd__" in
user-defined classes, but not for extension types. What is the
essential difference that makes that mapping work in one case but not
in the other?

 

[Michael Hudson]

( ... or tp_as_sequence ... )

I was afraid you (someone) was going to say that.
Why?

... def __iadd__(self,other):
... print "__iadd__ running"
... return self
...
__iadd__ running

I'd be surprised if I've added __iadd__ to a type object here, yet it
seems to work.

Well, the type of old-style classes has something in it's
tp_as_number->nb_inplace_add slot that looks in the class dictionary
for an __iadd__ method.

If you'd made that a new-style class, you would be surprised!

Python manages to map "+=" to the method called "__iadd__" in
user-defined classes, but not for extension types. What is the
essential difference that makes that mapping work in one case but
not in the other?

Well, for old-style classes a whole bunch of code like:

BINARY_INPLACE(instance_ior, "or", PyNumber_InPlaceOr)
BINARY_INPLACE(instance_ixor, "xor", PyNumber_InPlaceXor)
BINARY_INPLACE(instance_iand, "and", PyNumber_InPlaceAnd)
BINARY_INPLACE(instance_ilshift, "lshift", PyNumber_InPlaceLshift)
BINARY_INPLACE(instance_irshift, "rshift", PyNumber_InPlaceRshift)
BINARY_INPLACE(instance_iadd, "add", PyNumber_InPlaceAdd)
BINARY_INPLACE(instance_isub, "sub", PyNumber_InPlaceSubtract)
BINARY_INPLACE(instance_imul, "mul", PyNumber_InPlaceMultiply)
BINARY_INPLACE(instance_idiv, "div", PyNumber_InPlaceDivide)
BINARY_INPLACE(instance_imod, "mod", PyNumber_InPlaceRemainder)
BINARY_INPLACE(instance_ifloordiv, "floordiv", PyNumber_InPlaceFloorDivide)
BINARY_INPLACE(instance_itruediv, "truediv", PyNumber_InPlaceTrueDivide)

and for new-style classes much hair in typeobject.c:type_new and
therein called functions. Extension types don't go through type_new
and are expected to go the other way round, in a sense: define
something in tp_as_number->nb_inplace_add and a wrapper called
__iadd__ will be created for you.

Cheers,
mwh

 

[Jacek Generowicz]

Why?

Because I'm too lazy to pollute my tiny extension type with a whole
thingy_as_number sturucture for the sake of just one maginc method

Well, the type of old-style classes has something in it's
tp_as_number->nb_inplace_add slot that looks in the class dictionary
for an __iadd__ method.

If you'd made that a new-style class, you would be surprised!

Don't follow ... you mean if I had done this:
... def __iadd__(self,other):
... print "__iadd__ running"
... return self
... __iadd__ running

?

Well, for old-style classes a whole bunch of code like:

BINARY_INPLACE(instance_ior, "or", PyNumber_InPlaceOr)
BINARY_INPLACE(instance_ixor, "xor", PyNumber_InPlaceXor)
BINARY_INPLACE(instance_iand, "and", PyNumber_InPlaceAnd)
BINARY_INPLACE(instance_ilshift, "lshift", PyNumber_InPlaceLshift)
BINARY_INPLACE(instance_irshift, "rshift", PyNumber_InPlaceRshift)
BINARY_INPLACE(instance_iadd, "add", PyNumber_InPlaceAdd)
BINARY_INPLACE(instance_isub, "sub", PyNumber_InPlaceSubtract)
BINARY_INPLACE(instance_imul, "mul", PyNumber_InPlaceMultiply)
BINARY_INPLACE(instance_idiv, "div", PyNumber_InPlaceDivide)
BINARY_INPLACE(instance_imod, "mod", PyNumber_InPlaceRemainder)
BINARY_INPLACE(instance_ifloordiv, "floordiv", PyNumber_InPlaceFloorDivide)
BINARY_INPLACE(instance_itruediv, "truediv", PyNumber_InPlaceTrueDivide)

and for new-style classes much hair in typeobject.c:type_new and
therein called functions. Extension types don't go through type_new
and are expected to go the other way round, in a sense: define
something in tp_as_number->nb_inplace_add and a wrapper called
__iadd__ will be created for you.

Yuk.

You are, in summary, saying that by _far_ the simplest way of adding
__iadd__ to an extenion type is via tp_as_number->nb_inplace_add,
aren't you.

So be it.

Thanks.

 

[Greg Ewing]

You are, in summary, saying that by _far_ the simplest way of adding
__iadd__ to an extenion type is via tp_as_number->nb_inplace_add,
aren't you.

It's the *only* way to do it.

<plug shamelevel="0">

Unless you use Pyrex, of course:

http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/

</plug>

 

[Jacek Generowicz]

<plug shamelevel="0">

Unless you use Pyrex, of course:

http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/

</plug>

Hi Greg,

I've been wanting to pounce on Pyrex for ages. Unfortunately, it is
not politically correct for my current projects. I have vague hopes of
finding the time become sufficiently versed in Pyrex to be able to
make a convincing case for making it standard practice around here,
but, unfortunately, that time seems not to be getting any closer :-(

 

[Michael Hudson]

Because I'm too lazy to pollute my tiny extension type with a whole
thingy_as_number sturucture for the sake of just one maginc method

If __iadd__ is the *only* magic method you're adding, you deserve to
lose...

Don't follow ... you mean if I had done this:

... def __iadd__(self,other):
... print "__iadd__ running"
... return self
...
__iadd__ running

?

Yup: you've added something to a type object, something you'd said
you'd be surprised to hear.

Well, for old-style classes a whole bunch of code like:

[...]

and for new-style classes much hair in typeobject.c:type_new and
therein called functions. Extension types don't go through type_new
and are expected to go the other way round, in a sense: define
something in tp_as_number->nb_inplace_add and a wrapper called
__iadd__ will be created for you.
Yuk.

Why?

You are, in summary, saying that by _far_ the simplest way of adding
__iadd__ to an extenion type is via tp_as_number->nb_inplace_add,
aren't you.

Yes. I'm surprised at your apparent surprise, but then I've had my
head in the guts of Python's implementation for quite some time
now... implementing types in C is really quite different from
implementing them in Python, something which has only become
surprising since 2.2...

Cheers,
mwh

 

[Jacek Generowicz]

Why?

Partly because I misread (thinking that *I* had to write hairy stuff
for it to work), and partly because the dichotomy offends my sense of
symmetry, beauty, coherence, cleanlyness etc.

Yes. I'm surprised at your apparent surprise, but then I've had my
head in the guts of Python's implementation for quite some time
now...

.... while I try to keep out of there. (I appreciate Python for its
ability to hide unimportant low-level details, not for it being a C
program; if I wanted to program in a low-level language, I wouldn't
need Python ... unless it would be to play with implementation details
of Python itself: you realize that I'm eagerly waiting for PyPy to
deliver

implementing types in C is really quite different from
implementing them in Python, something which has only become
surprising since 2.2...

Yes, I was blindly hoping that type-class unification would have made
implementing extension types look more like implementing Python types.

Anyway, it's clear what I have to do. Thanks.