Subclassing in C

[Iker Arizmendi]

Hello all,

I've defined a class in a C extension module that is
(or rather, I would like to be) a subclass of another
class. I do this by simply setting the tp_base pointer
of my subclass's type object. However, this results
in a class that exposes the methods of the base class,
but ONLY those. The methods of my subclass are hidden.
I'd obviously like to have the methods of the base
class AND the new methods of the subclass available.

Here's the code snippet I used:

static PyTypeObject MyPyType =
{
PyObject_HEAD_INIT(NULL)
0, /* ob_size */
"mymod.MyPyType", /* tp_name */
sizeof(MyPyType), /* tp_basicsize */
...
(destructor)MyPyType_dealloc, /* tp_dealloc */
...
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
...
MyPyType_methods, /* tp_methods */
MyPyType_members, /* tp_members */
...
(initproc)MyPyType_init, /* tp_init */
0, /* tp_alloc */
MyPyType_new, /* tp_new */
};


PyMODINIT_FUNC initmymod(void)
{
MyPyType.tp_new = MyPyType_new;
MyPyType.tp_base = pointerToOtherPyType;
if (PyType_Ready(&MyPyType) < 0)
return;
}

When using MyPyType in python code, the methods defined in
MyPyType_methods are not visible. If I comment out the
line that sets tp_base then they become visible but the
base class methods are unavailable. What's the correct
way to do this sort of thing?

Regards,
Iker

 

[Michael Hudson]

Hello all,

I've defined a class in a C extension module that is
(or rather, I would like to be) a subclass of another
class. I do this by simply setting the tp_base pointer
of my subclass's type object. However, this results
in a class that exposes the methods of the base class,
but ONLY those. The methods of my subclass are hidden.
I'd obviously like to have the methods of the base
class AND the new methods of the subclass available.

Are you setting tp_getattro to PyObject_GenericGetAttr? (just a
guess).

Cheers,
mwh

 

[Iker Arizmendi]

I was setting tp_getattr to 0 in my subclass (*), but when
I set it to PyObject_GenericGetAttr Python crashed. What
does PyObject_GenericGetAttr do that would help in
this case?

Regards,
Iker

(*) The class I'm deriving from is defined in another
extension module and it has its own tp_getattr method.


Here's the stack trace (via Valgrind), just in case:

Process terminating with default action of signal 11 (SIGSEGV)
Access not within mapped region at address 0x65C9
at 0x1B97CAC3: PyType_IsSubtype (in /usr/lib/libpython2.3.so.1.0)
by 0x1B96C92D: PyObject_GenericGetAttr (in /usr/lib/libpython2.3.so.1.0)
by 0x1B96C5E8: PyObject_GetAttr (in /usr/lib/libpython2.3.so.1.0)
by 0x1B9A51D2: (within /usr/lib/libpython2.3.so.1.0)
by 0x1B9A65E2: PyEval_EvalCodeEx (in /usr/lib/libpython2.3.so.1.0)
by 0x1B9A31B4: PyEval_EvalCode (in /usr/lib/libpython2.3.so.1.0)
by 0x1B9D4D20: (within /usr/lib/libpython2.3.so.1.0)
by 0x1B9D41EE: PyRun_SimpleFileExFlags (in /usr/lib/libpython2.3.so.1.0)
by 0x1B9D3BEF: PyRun_AnyFileExFlags (in /usr/lib/libpython2.3.so.1.0)
by 0x1B9DC7A3: Py_Main (in /usr/lib/libpython2.3.so.1.0)
by 0x8048640: main (in /usr/bin/python2.3)

 

[Michael Hudson]

I was setting tp_getattr to 0 in my subclass (*), but when
I set it to PyObject_GenericGetAttr Python crashed. What
does PyObject_GenericGetAttr do that would help in
this case?

Well, it does "the usual" attribute access stuff. Descriptors & so
on. It's what object.__getattribute__ wraps.

Regards,
Iker

(*) The class I'm deriving from is defined in another
extension module and it has its own tp_getattr method.

Hmm. I take it *it's* tp_getattr[o] method isn't
PyObject_GenericGetAttr then?

Then your initial post makes more sense; I'm afraid I don't see
any obvious reason for PyObject_GenericGetAttr to crash.

Here's the stack trace (via Valgrind), just in case:

A stack trace from gdb (i.e. with line numbers) would have been more
use.

I think you have two options here: (a) make friends with gdb or (b)
post [a link to, maybe] complete code.

Cheers,
mwh

 

[Iker Arizmendi]

(*) The class I'm deriving from is defined in another
extension module and it has its own tp_getattr method.

Hmm. I take it *it's* tp_getattr[o] method isn't
PyObject_GenericGetAttr then?

Then your initial post makes more sense; I'm afraid I don't see
any obvious reason for PyObject_GenericGetAttr to crash.

Ah! I missed the trailing 'o' and set tp_getattr, not
tp_getattro as you suggested. However, after setting it
correctly (I think), things still don't behave as I would
expect (eg, the subclass has the union of its methods and
the base class methods). The complete code is pretty long
so I'm not sure I can post it (most of it deals with non
Python stuff) - but here's a more complete piece that
relates to setting up the classes.

The base class (defined in a separated module) looks
like so:

static PyMethodDef fsmpymethods[] =
{
{
"type", (PyCFunction) fsmpytype,
METH_NOARGS,
"type() -> string\n\nReturn FSM type."
},
...
{
NULL, NULL, 0, NULL
}
};

static PyObject*
fsmpy_getattro(PyObject *obj, PyObject* name)
{
return Py_FindMethod(fsmpymethods, obj,
PyString_AsString(name));
}

PyTypeObject FSMPyType =
{
PyObject_HEAD_INIT(NULL)
0,
"fsm.Fsm",
sizeof(FSMPyObject),
0,
fsmpy_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
fsmpy_getattro, /*tp_getattro*/
...
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
...
};

The base class doesn't have a new or init method as the author
provides an explicit "create" method.

The subclass that I've put together now looks like so (note
that I now use the tp_getattro method, and cleared both
tp_methods and tp_members):

static PyMethodDef PyGrmReplace_methods[] =
{
{
"mutate", (PyCFunction)PyGrmReplace_mutate,
METH_VARARGS|METH_KEYWORDS,
"mutate(fsm1 [, mode]) -> fsm\n"
},
...
{NULL}
};

static PyObject*
PyGrm_GetAttro(PyObject *obj, PyObject* name)
{
return Py_FindMethod(PyGrmReplace_methods, obj,
PyString_AsString(name));
}

static PyTypeObject PyGrmReplaceType =
{
PyObject_HEAD_INIT(NULL)
0, /* ob_size */
"grm.GrmReplace", /* tp_name */
sizeof(PyFsmObject), /* tp_basicsize */
0, /* tp_itemsize */
PyGrmReplace_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
...
PyGrm_GetAttro, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT, /* tp_flags */
"GrmReplace objects", /* tp_doc */
...
0, /* tp_methods */
0, /* tp_members */
...
PyGrmReplace_init, /* tp_init */
0, /* tp_alloc */
PyGrmReplace_new, /* tp_new */
};

PyMODINIT_FUNC
initgrm(void)
{
PyGrmReplaceType.tp_new = PyGrmReplace_new;
PyGrmReplaceType.tp_base = pointerToFSMPyType;
if (PyType_Ready(&PyGrmReplaceType) < 0)
return;

PyObject* m = Py_InitModule3("grm", grm_methods,
"Python binding for the AT&T GRM library");
Py_INCREF(&PyGrmReplaceType);
PyModule_AddObject(m, "GrmReplace", (PyObject*)&PyGrmReplaceType);
}

And the result of this is that an instance of grm.GrmReplace
makes the mutate method available, but doesn't have the
type() method from the fsm.Fsm class. Is there some "canonical"
form for implementing this?

Thanks again,
Iker

Here's the stack trace (via Valgrind), just in case:

A stack trace from gdb (i.e. with line numbers) would have been more
use.

I think you have two options here: (a) make friends with gdb or (b)
post [a link to, maybe] complete code.

Cheers,
mwh

 

[Iker Arizmendi]

After placing a printf statement in both of these methods

fsmpy_getattro
PyGrm_GetAttro

I notice that when calling the "type" method of my
subclass like so:

g = grm.GrmReplace(...)
print g.type()

only the PyGrm_GetAttro method is called. I expected
that the call to Py_FindMethod would eventually call
fsmpy_getattro since my subclass doesn't have a method
called "type" (here the "type" method has nothing to do
with Python types, it's just what we happen to call
one of our methods).


Regards,
Iker

(*) The class I'm deriving from is defined in another
extension module and it has its own tp_getattr method.

Hmm. I take it *it's* tp_getattr[o] method isn't
PyObject_GenericGetAttr then?

Then your initial post makes more sense; I'm afraid I don't see
any obvious reason for PyObject_GenericGetAttr to crash.

Ah! I missed the trailing 'o' and set tp_getattr, not
tp_getattro as you suggested. However, after setting it
correctly (I think), things still don't behave as I would
expect (eg, the subclass has the union of its methods and
the base class methods). The complete code is pretty long
so I'm not sure I can post it (most of it deals with non
Python stuff) - but here's a more complete piece that
relates to setting up the classes.

The base class (defined in a separated module) looks
like so:

static PyMethodDef fsmpymethods[] =
{
{
"type", (PyCFunction) fsmpytype,
METH_NOARGS,
"type() -> string\n\nReturn FSM type."
},
...
{
NULL, NULL, 0, NULL
}
};

static PyObject*
fsmpy_getattro(PyObject *obj, PyObject* name)
{
return Py_FindMethod(fsmpymethods, obj,
PyString_AsString(name));
}

PyTypeObject FSMPyType =
{
PyObject_HEAD_INIT(NULL)
0,
"fsm.Fsm",
sizeof(FSMPyObject),
0,
fsmpy_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
fsmpy_getattro, /*tp_getattro*/
...
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
...
};

The base class doesn't have a new or init method as the author
provides an explicit "create" method.

The subclass that I've put together now looks like so (note
that I now use the tp_getattro method, and cleared both
tp_methods and tp_members):

static PyMethodDef PyGrmReplace_methods[] =
{
{
"mutate", (PyCFunction)PyGrmReplace_mutate,
METH_VARARGS|METH_KEYWORDS,
"mutate(fsm1 [, mode]) -> fsm\n"
},
...
{NULL}
};

static PyObject*
PyGrm_GetAttro(PyObject *obj, PyObject* name)
{
return Py_FindMethod(PyGrmReplace_methods, obj,
PyString_AsString(name));
}

static PyTypeObject PyGrmReplaceType =
{
PyObject_HEAD_INIT(NULL)
0, /* ob_size */
"grm.GrmReplace", /* tp_name */
sizeof(PyFsmObject), /* tp_basicsize */
0, /* tp_itemsize */
PyGrmReplace_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
...
PyGrm_GetAttro, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT, /* tp_flags */
"GrmReplace objects", /* tp_doc */
...
0, /* tp_methods */
0, /* tp_members */
...
PyGrmReplace_init, /* tp_init */
0, /* tp_alloc */
PyGrmReplace_new, /* tp_new */
};

PyMODINIT_FUNC
initgrm(void)
{
PyGrmReplaceType.tp_new = PyGrmReplace_new;
PyGrmReplaceType.tp_base = pointerToFSMPyType;
if (PyType_Ready(&PyGrmReplaceType) < 0)
return;

PyObject* m = Py_InitModule3("grm", grm_methods,
"Python binding for the AT&T GRM library");
Py_INCREF(&PyGrmReplaceType);
PyModule_AddObject(m, "GrmReplace", (PyObject*)&PyGrmReplaceType);
}

And the result of this is that an instance of grm.GrmReplace
makes the mutate method available, but doesn't have the
type() method from the fsm.Fsm class. Is there some "canonical"
form for implementing this?

Thanks again,
Iker

Here's the stack trace (via Valgrind), just in case:

A stack trace from gdb (i.e. with line numbers) would have been more
use.

I think you have two options here: (a) make friends with gdb or (b)
post [a link to, maybe] complete code.

Cheers,
mwh

 

[Iker Arizmendi]

After some more poking around I think I've found a solution
to my problem. But although it works, it doesn't look like
the right way to do things.

In short, I updated the tp_getattro method of my subclass to
explicitly check its base class for the presence of any
attributes it doesn't know about. Eg.

static PyObject*
PyGrm_GetAttro(PyObject *obj, PyObject* name)
{
PyObject* r = Py_FindMethod(PyGrmReplace_methods, obj,
PyString_AsString(name));
if( PyErr_Occurred() )
{
if( PyErr_ExceptionMatches(PyExc_AttributeError) )
{
PyErr_Clear();
PyTypeObject* t = obj->ob_type;
return t->tp_base->tp_getattro(obj, name);
}
}
return r;
}

Again, in this case this works just fine as I know I'm only
interested in the one base class. But if I had a deeper
inheritance hierarchy I would have to do the lookup
across each base class myself.


Regards,
Iker

After placing a printf statement in both of these methods

fsmpy_getattro
PyGrm_GetAttro

I notice that when calling the "type" method of my
subclass like so:

g = grm.GrmReplace(...)
print g.type()

only the PyGrm_GetAttro method is called. I expected
that the call to Py_FindMethod would eventually call
fsmpy_getattro since my subclass doesn't have a method
called "type" (here the "type" method has nothing to do
with Python types, it's just what we happen to call
one of our methods).


Regards,
Iker

(*) The class I'm deriving from is defined in another
extension module and it has its own tp_getattr method.




Hmm. I take it *it's* tp_getattr[o] method isn't
PyObject_GenericGetAttr then?

Then your initial post makes more sense; I'm afraid I don't see
any obvious reason for PyObject_GenericGetAttr to crash.

Ah! I missed the trailing 'o' and set tp_getattr, not
tp_getattro as you suggested. However, after setting it
correctly (I think), things still don't behave as I would
expect (eg, the subclass has the union of its methods and
the base class methods). The complete code is pretty long
so I'm not sure I can post it (most of it deals with non
Python stuff) - but here's a more complete piece that
relates to setting up the classes.

The base class (defined in a separated module) looks
like so:

static PyMethodDef fsmpymethods[] =
{
{
"type", (PyCFunction) fsmpytype,
METH_NOARGS,
"type() -> string\n\nReturn FSM type."
},
...
{
NULL, NULL, 0, NULL
}
};

static PyObject*
fsmpy_getattro(PyObject *obj, PyObject* name)
{
return Py_FindMethod(fsmpymethods, obj,
PyString_AsString(name));
}

PyTypeObject FSMPyType =
{
PyObject_HEAD_INIT(NULL)
0,
"fsm.Fsm",
sizeof(FSMPyObject),
0,
fsmpy_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
fsmpy_getattro, /*tp_getattro*/
...
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
...
};

The base class doesn't have a new or init method as the author
provides an explicit "create" method.

The subclass that I've put together now looks like so (note
that I now use the tp_getattro method, and cleared both
tp_methods and tp_members):

static PyMethodDef PyGrmReplace_methods[] =
{
{
"mutate", (PyCFunction)PyGrmReplace_mutate,
METH_VARARGS|METH_KEYWORDS,
"mutate(fsm1 [, mode]) -> fsm\n"
},
...
{NULL}
};

static PyObject*
PyGrm_GetAttro(PyObject *obj, PyObject* name)
{
return Py_FindMethod(PyGrmReplace_methods, obj,
PyString_AsString(name));
}

static PyTypeObject PyGrmReplaceType =
{
PyObject_HEAD_INIT(NULL)
0, /* ob_size */
"grm.GrmReplace", /* tp_name */
sizeof(PyFsmObject), /* tp_basicsize */
0, /* tp_itemsize */
PyGrmReplace_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
...
PyGrm_GetAttro, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT, /* tp_flags */
"GrmReplace objects", /* tp_doc */
...
0, /* tp_methods */
0, /* tp_members */
...
PyGrmReplace_init, /* tp_init */
0, /* tp_alloc */
PyGrmReplace_new, /* tp_new */
};

PyMODINIT_FUNC
initgrm(void)
{
PyGrmReplaceType.tp_new = PyGrmReplace_new;
PyGrmReplaceType.tp_base = pointerToFSMPyType;
if (PyType_Ready(&PyGrmReplaceType) < 0)
return;

PyObject* m = Py_InitModule3("grm", grm_methods,
"Python binding for the AT&T GRM library");
Py_INCREF(&PyGrmReplaceType);
PyModule_AddObject(m, "GrmReplace", (PyObject*)&PyGrmReplaceType);
}

And the result of this is that an instance of grm.GrmReplace
makes the mutate method available, but doesn't have the
type() method from the fsm.Fsm class. Is there some "canonical"
form for implementing this?

Thanks again,
Iker

Here's the stack trace (via Valgrind), just in case:




A stack trace from gdb (i.e. with line numbers) would have been more
use.

I think you have two options here: (a) make friends with gdb or (b)
post [a link to, maybe] complete code.

Cheers,
mwh

 

[Leif K-Brooks]

(*) The class I'm deriving from is defined in another
extension module and it has its own tp_getattr method.

Hmm. I take it *it's* tp_getattr[o] method isn't
PyObject_GenericGetAttr then?

[Emphasis not mine.]

Sorry, but Iker had it right and you had it wrong. "Its", with no
apostrophe, is possessive; "it's", with an apostrophe, is a contraction
of "it is". Therefor, "It's red." and "Its color is red." are both correct.

 

[Michael Hudson]

(*) The class I'm deriving from is defined in another
extension module and it has its own tp_getattr method.

Hmm. I take it *it's* tp_getattr[o] method isn't
PyObject_GenericGetAttr then?

[Emphasis not mine.]

Sorry, but Iker had it right and you had it wrong. "Its", with no
apostrophe, is possessive; "it's", with an apostrophe, is a
contraction of "it is". Therefor, "It's red." and "Its color is red."
are both correct.

Believe me, I know. However, I type fairly fast, and very often make
this mistake. (I cringed when my post came back to me from the
newsserver).

Cheers,
mwh

 

[Michael Hudson]

(*) The class I'm deriving from is defined in another
extension module and it has its own tp_getattr method.

Hmm. I take it *it's* tp_getattr[o] method isn't
PyObject_GenericGetAttr then?
Then your initial post makes more sense; I'm afraid I don't see
any obvious reason for PyObject_GenericGetAttr to crash.

Ah! I missed the trailing 'o' and set tp_getattr, not
tp_getattro as you suggested.

[snip]

OK, some questions.

1) You must be assuming 2.2 or later, right? tp_base doesn't make
sense before then.
2) Is this a third party base type?
3) If 2) is the third party assuming 2.2 or later?

If 2) but not 3), you might be in for some hacking (as you later came
up with). You can't really subclass (cleanly) a type that's totally
unprepared for it.

Py_FindMethod() is SO 2001

Cheers,
mwh

 

[Iker Arizmendi]

OK, some questions.

1) You must be assuming 2.2 or later, right? tp_base doesn't make
sense before then.

Yup, 2.2 or later (although I'm currently using 2.3).

2) Is this a third party base type?

Not really, it's written by another fellow here. But
I have access to the source and am free to make small
changes.

3) If 2) is the third party assuming 2.2 or later?

I think it was written before 2.2, but I've made some
changes that I think make it 2.2 compliant (eg, I added
the Py_TPFLAGS_BASETYPE flag, prepared the type with
PyType_Ready, and changed it to use tp_getattro instead
of tp_getattr.

If 2) but not 3), you might be in for some hacking (as you later came
up with). You can't really subclass (cleanly) a type that's totally
unprepared for it.

So I guess I have (!2 && !3), but I think I've managed
to change that to (!2 && 3). So assuming I have two types,
both written in C, what's the clean way to do the
subclassing? In particular, it would be nice if the C API
did the search of my bases for me.

Py_FindMethod() is SO 2001

Just got started with Python and put on the first thing
I found. Didn't realize they were bell bottoms

Is there a newer method?


Regards,
Iker

 

[Michael Hudson]

Yup, 2.2 or later (although I'm currently using 2.3).


Not really, it's written by another fellow here. But
I have access to the source and am free to make small
changes.

OK, then I suggest you start by updating this code to the 2.2 era...

Have you read

http://www.python.org/doc/ext/defining-new-types.html

? If not, you definitely want to.

I think it was written before 2.2, but I've made some
changes that I think make it 2.2 compliant (eg, I added
the Py_TPFLAGS_BASETYPE flag, prepared the type with
PyType_Ready, and changed it to use tp_getattro instead
of tp_getattr.

In the 2.2 era, tp_getattro should almost invariably be
PyObject_GenericGetAttr. Fill out the tp_methods, tp_members, etc
slots of the base type.

So I guess I have (!2 && !3), but I think I've managed
to change that to (!2 && 3). So assuming I have two types,
both written in C, what's the clean way to do the
subclassing? In particular, it would be nice if the C API
did the search of my bases for me.

That's what PyObject_GenericGetAttr does.

Just got started with Python and put on the first thing
I found. Didn't realize they were bell bottoms

Fair enough

Is there a newer method?

See above.

Cheers,
mwh