ctypes inheritance issue

[Steve]

I've filed a bug in python but I wanted to see if other ctypes users/
experts viewed this issue as a bug.
Consider the following:
python code:
import ctypes
class my_array( ctypes.Array ):
_type_ = ctypes.c_uint8
_length_ = 256

class my_array2( my_array ):
pass

Output:
class my_array2( my_array ):
AttributeError: class must define a '_length_' attribute, which must
be a positive integer


This is analogous to the C code
typedef char my_array[ 256 ];
typedef my_array my_array2;

As shown above, the python code raises exceptions claiming _type_ and
_length_ have not been defined. This seems like a bug. I shouldn't
need to redefine _type_ and _length_, otherwise there was no point in
subclassing from my_array.

Thoughts?

 

[Carl Banks]

I've filed a bug in python but I wanted to see if other ctypes users/
experts viewed this issue as a bug.
Consider the following:
python code:
import ctypes
class my_array( ctypes.Array ):
_type_ = ctypes.c_uint8
_length_ = 256

class my_array2( my_array ):
pass

Output:
class my_array2( my_array ):
AttributeError: class must define a '_length_' attribute, which must
be a positive integer

I'm not sure if I'd call it a bug, but I can tell you (probably) why it
happens.

ctypes.Array uses a custom metaclass that inspects the class dictionary
directly for those attributes before passing them on to the type
constructor. In simple terms, it means that ctypes is using the
_length_ and _type_ attributes before it creates the class.

That's an unfortunate aspect of metaclass programming: sometimes you
need that information in order to create the class, but you need the
class in order to inherit. So, you are stuck with forcing subtypes to
redefine the same attributes as their parents, or trying to mimic the
inheritance by searching the bases yourself (which can be prone to
mistakes).

Point is: this behavior is probably the way it is because of a technical
difficulty as opposed to an oversight or design choice.

This is analogous to the C code
typedef char my_array[ 256 ];
typedef my_array my_array2;

Not really. Typedefs don't define new types in C; they define aliases
for existing types. This is what would be analogous to your C code:

my_array_2 = my_array

(Which is not to say this affects whether this is a bug or not.)

As shown above, the python code raises exceptions claiming _type_ and
_length_ have not been defined. This seems like a bug. I shouldn't
need to redefine _type_ and _length_, otherwise there was no point in
subclassing from my_array.

Thoughts?

I would vote "not a bug", though I'm pretty close to the fence.

I doubt this behavior is documented one way or the other (if it is
that that would change things), so for it to be considered a bug it'd
have to be very contrary to user expectations. Not inheriting is quite
unexpected, yes, but when metaclass programming is involved all bets are
off about stuff like that. A lot of what makes metaclasses useful
depends on them breaking expectations with respect to ordinary
classes, so I don't think it would be wise to label every divergence a
bug.


Carl Banks

 

[Steve]

I've filed a bug in python but I wanted to see if other ctypes users/
experts viewed this issue as a bug.
Consider the following:
python code:
import ctypes
class my_array( ctypes.Array ):
    _type_    = ctypes.c_uint8
    _length_  = 256

class my_array2( my_array ):
    pass

Output:
class my_array2( my_array ):
AttributeError: class must define a '_length_' attribute, which must
be a positive integer

I'm not sure if I'd call it a bug, but I can tell you (probably) why it
happens.

ctypes.Array uses a custom metaclass that inspects the class dictionary
directly for those attributes before passing them on to the type
constructor.  In simple terms, it means that ctypes is using the
_length_ and _type_ attributes before it creates the class.

That's an unfortunate aspect of metaclass programming: sometimes you
need that information in order to create the class, but you need the
class in order to inherit.  So, you are stuck with forcing subtypes to
redefine the same attributes as their parents, or trying to mimic the
inheritance by searching the bases yourself (which can be prone to
mistakes).

Point is: this behavior is probably the way it is because of a technical
difficulty as opposed to an oversight or design choice.

This is analogous to the C code
typedef char my_array[ 256 ];
typedef my_array my_array2;

Not really.  Typedefs don't define new types in C; they define aliases
for existing types. This is what would be analogous to your C code:

my_array_2 = my_array

(Which is not to say this affects whether this is a bug or not.)

As shown above, the python code raises exceptions claiming _type_ and
_length_ have not been defined.  This seems like a bug.  I shouldn't
need to redefine _type_ and _length_, otherwise there was no point in
subclassing from my_array.

Thoughts?

I would vote "not a bug", though I'm pretty close to the fence.

I doubt this behavior is documented one way or the other (if it is
that that would change things), so for it to be considered a bug it'd
have to be very contrary to user expectations. Not inheriting is quite
unexpected, yes, but when metaclass programming is involved all bets are
off about stuff like that.  A lot of what makes metaclasses useful
depends on them breaking expectations with respect to ordinary
classes, so I don't think it would be wise to label every divergence a
bug.

Carl Banks

After looking at some metaclass examples it appears this information
is readily available. A metaclass gets a dictionary containing
information about the parent class (or should, at least). It seems
like it must have this information in order to dynamically make
decisions about how to create classes... So, "bug" or not, shouldn't
this just work? Is there something that prevents it from being
implemented? Would this break something?

 

[Carl Banks]

After looking at some metaclass examples it appears this information
is readily available. A metaclass gets a dictionary containing
information about the parent class (or should, at least).

What examples did you look at?

 It seems
like it must have this information in order to dynamically make
decisions about how to create classes...  So, "bug" or not, shouldn't
this just work?

No. Information about parent class members is available if you dig
for it but it doesn't "just work".

A metaclass gets three pieces of information it uses when constructing
a class: the name of the class, a list of bases, and a dictionary
containing everything defined in the class's scope (and only the
class's scope, not the scope of any base classes). Some, if not most,
metaclasses inspect and modify this dictionary before passing it to
the type constructor (type.__new__); inheritance hasn't even come into
play at that point.

A metaclass can look at the list of bases and try to extract
attributes from them, but that's not just working; that's digging.
(Needless to say, a lot of implementors don't go through the effort to
dig.)

 Is there something that prevents it from being
implemented?  Would this break something?

As I said, it's inherently a chicken-and-egg problem. You have a
situation where you want to inherit the information needed to create a
class, but inheritance doesn't come into play until the class is
created.

I guess you could elimiate the paradox by breaking down type
construction into steps (set up the inheritance relationships first,
then construct the type object, giving the metaclass time to get data
from the bases).

Some other language will have to try that, though. Yes it would break
things. Not a lot of things but there cases where you don't want to
inherit. I use the following pattern fairly often:


class KeepTrackOfSubtypesMetaclass(type):

subtypes = {}

def __new__(metatype,name,bases,class_dct):
key = class_dct.get('key')
self = type.__new__(metatype,name,bases,class_dct)
if key is not None:
metatype.subtypes[key] = self
return self


Any instance of this metaclass that defines key in its scope will be
added to the dict of subtypes. But I don't want a derived class to
overwrite its parent's entry in the subtype dict--it should define its
own key.


Carl Banks