Order in metaclass

[Thomas Heller]

I just looked at ctypes, and yes, that's exactly what I'm doing. I
want the structures definitions to be modified by C++ programmers and
I find the metaclass mechanism much more readable and easy to modify
(and it's not a lot of code to make everything work). I agree
everything can be done using normal Python code, but my goal here is
really to make a pseudo-language to define our formats, and the
pseudo-language is just a simpler solution than XML in my specific
case. Probably all cases of such using of metaclasses are like
that...

FWIW, I have a parser(code generator (using gccxml as frontend). This
toolchain creates ctypes' compatible structure definitions from C header
files.

Thomas

 

[Carlos Ribeiro]

And for your original question: why do you insist on your new syntax,
why don't you simply (as ctypes also does it) define the attributes in a
list, when they have to have a certain order?

I can't speak for Nicolas, but I can for myself. There are three
reasons, in my particular case:

-- The order is already there, explicit in the way the code is
written. I see no reason to be forced to state it _again_, specially
considering the fact that this is prone to errors; for example,
missing elements, duplicated elements, or out-of-order elements (in
ordered differently in the code and in the list).

-- I make extensive use of nested structs. In this case it's _much_
easier to make mistakes as the ones mentioned above.

-- Using classes and inheritance, it's easy to add new members or
redefine existing ones. But I would need to re-state the full list for
any descendant. To make matters worse, in this case, whenever I make a
change to the base class, I would have to change _all_ descendant
classes. Not exactly good OO design. Of course, clever hacks could
possibly be used, but then, it wouldn't look any better than Nicolas
(or my own) ordered-attributes hack.


It's specially important to point out that I consider the source code
ordering to be _explicit_, and not implicit, as some people may
possibly say. In fact, is there anything more explicit than that? Only
people used to Python dicts would say otherwise.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Carlos Ribeiro]

No, it doesn't:

Oh well, I just missed that Nicolas was using co_names, instead
of line numbering magic as I did in my first tests. Seems he was a
little bit more clever than I was

... a, b, c, d = "xyzt"
... print sys._getframe().f_code.co_names
...
('__name__', '__module__', 'a', 'b', 'c', 'd', 'sys', '_getframe', 'f_code',
'co_names')

whereas
['a', '__module__', 'b', 'd', 'c', '__doc__']

Of course there are other ways to garble the co_names order:
... def __getattr__(self, name): return name
...... x = any.d
... a, b, c, d = "xyzt"
... print sys._getframe().f_code.co_names
...
('__name__', '__module__', 'any', 'd', 'x', 'a', 'b', 'c', 'sys',
'_getframe', 'f_code', 'co_names')

All very useless/interesting stuff

Useless, in a sense, yes. But we learned a lot (or at least, I did).
So it was not totally useless...

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Bengt Richter]

Or, an ugly hack that might work for a while, depending on how
co_names is really generated. It seems in order of occurrence
(including right hand sides of assignment, but still top down) ...

import sys
def getnames(): return sys._getframe(1).f_code.co_names
...
def MC(cname, cbases, cdict):
... names = cdict.get('ordic',[])
... names = [name for name in names if name in cdict]
... cdict['ordic'] = dict([(name,i) for i,name in enumerate(names)])
... return type(cname, cbases, cdict)
...
class C(object):
... __metaclass__ = MC # really a function shortcut
... x = 123
... y = sys
... z = 0
... ordic = getnames()
...
C.ordic
{'ordic': 5, '__module__': 0, '__metaclass__': 1, 'y': 3, 'x': 2, 'z': 4}

A metaclass /function/ and sys._getframe() exercised on a class definition
- I think you have raised the bar for what qualifies as a hack

Hey, I came first with that a couple of weeks ago but in truth, it
was Alex Martelli that pointed to me that a metaclass function would
work... but not only is it not recommended, it's also said to make
Guido shudder ;-) Seriously, although it works, it's not recommended
practice. Metaclasses are supposed to be classes, not functions.

IIRC it was Guido himself who let the cat out of the bag about a function, in
some early metaclass notes. But maybe I don't RC ;-)

As for the getframe, I have played with it a little bit also. But in
the end I have chosen to use a simple counter, using
itertools.count(). More pythonic IMHO. And no need for clever hacks,
when all that is needed is to order the elements in the order they are
executed (which count() can guarantee). There are two situations where
the simpler counter works, but the getframe hack doesn't:

-- if I have a loop inside my class that is used to declare bunch of
attributes, all of them will have the same line number... but a
different sequence number if the simpler method is chosen.

You mean like
for name in 'a b c'.split(): locals()[name] = ord(name[0])
? Yes, I think the names have to be visible to the compiler to show up in co_names.

-- if a function is called that returns a bunch of attributes (not
common practice, but still possible). All attributes are at the same
line in this case. Example:

class Foo:
a,b,c = myfunc(...)

That one _will_ show a,b,c in order. They're all visible to the compiler as names.

Of course, we are now getting into corner cases that show how much are
we pushing class statements in Python. The _sane_ way to make it all
work would be to have a hook to provide a user-defined dict to the
class locals() dir; aternatively, the natice dict() could provide a

interestingly, you can pass a dict subtype instance as the last arg to type, but
it seems just to grab the instance's base dict. I.e., you apparently can't
get the subtype instance back as .__dict__. Oh well ;-)

ordered interface (but then it wouldn't be a simple hash mapping, a
more complex structure such as a tree would be needed). Both are far
from happening in Python 2.x, IMHO... and I really doubt if dicts will
ever be changed to accomodate ordering, even in Python 3.0. Too much
hassle for too little gain. A hook function seems to be more sensible.

class C(object, cdict=myDict): pass ??

Regards,
Bengt Richter

 

[Carlos Ribeiro]

And for your original question: why do you insist on your new syntax,
why don't you simply (as ctypes also does it) define the attributes in a
list, when they have to have a certain order?

It's funny, but I was working with a good case for native (as in
source-code-defined) ordering right now and had missed it. I'm
learning to write test cases using unittest.py, and it just stroke me
that it's better to have tests to run in the order of complexity, in a
predictable way; it makes reading the list and analyzing the results
easier. My tests are beginning to get complex, and I just thought that
this would be a good use of our techniques (specially because defs
already have ordering information stored as function code attributes).

(in fact, I just checked unittest.py; it seems to be relatively easy
to change, because it has a hook for a cmp function that can use this
extra information)

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Bengt Richter]

Do you have an example handy? I had so far no success breaking Bengt's code.

Ok. Just as an exercise -- at this point we're far from safe Python
land anyway, and it's not recommended to push it beyond this limit...
;-) Pseudo code only:

class Foo:
# loop that creates attributes out of a list
for name, val in list:
locals()[name] = val

Sure, the compiler has to see the names in order for them to get into co_names in the frame.

or (actual code tested):

... a,b,c,d = (1,2,3,4)
...
{'a': 1, '__module__': '__main__', 'b': 2, 'd': 4, 'c': 3, '__doc__': None}

The ordering became (a,b,d,c); and the code would have no way to tell
that 'c' was supposed to be classified before 'd', because the
getframe trick would return the same line.

The "getframe trick" (if you mean my hack ;-) is not using line numbers,
so I don't know what you mean:
... __metaclass__ = MC # really a function shortcut
... a,b,c,d = (1,2,3,4)
... ordic = getnames()
... {'a': 2, 'ordic': 6, '__module__': 0, 'b': 3, '__metaclass__': 1, 'd': 5, 'c': 4}

Regards,
Bengt Richter

 

[Carlos Ribeiro]

IIRC it was Guido himself who let the cat out of the bag about a function, in
some early metaclass notes. But maybe I don't RC ;-)

Alex said that he was the one that shown it to Guido, but even he
couldn't recollect it very well. But that's not the point anyway.

That one _will_ show a,b,c in order. They're all visible to the compiler as names.

Sure. I misunderstand your trick, and assumed you did similar to what
I did. Your trick is _much_ better than my original one. But still, it
can be fooled, as Peter has shown in another message; but OTOH, it
doesn't _require_ all names to be subclasses of some OrderedAttribute
class.

I'm now thinking about how to use your trick, maybe combined with
mine, into the same framework. I'll let you know if I managed to make
it.

Interestingly, you can pass a dict subtype instance as the last arg to type, but
it seems just to grab the instance's base dict. I.e., you apparently can't
get the subtype instance back as .__dict__. Oh well ;-)

I don't get it; would you mind to explain it better? Did you test it?

class C(object, cdict=myDict): pass ??

My best bet is something like metaclasses implemented as class
decorators -- something that is read by the compiler _before_ the
class statement is executed. Being a class (not a function), it could
provide methods to customize some aspects of the class statement
execution; for example, a __getlocals__() method, or something along
these lines. and, of course, it would be able to put the __metaclass__
name right into the locals() dict, which would make it backward
compatible. The problem is, function decorators are only executed
_after_ the function def is executed; and having class decorators work
differently is a big no-no.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Bengt Richter]

I'm now thinking about how to use your trick, maybe combined with
mine, into the same framework. I'll let you know if I managed to make
it.

I doubt if it's a good idea to be dependent on such implementation details,
except to make throwaway prototypes of functionality with what's available.

I don't get it; would you mind to explain it better? Did you test it?

I just did an experiment:
<dictproxy object at 0x00901090>

.... not <CD {'some': 'content', ...}> in any form, AFAICS. I.e., it acts like
I just did T = type('T',(), {'some':'content')}

IOW, I think type must just extract the contents from the supplied dict or dict subtype.
I wonder what method one should override to customize the content extraction, or if that's possible.

So T.__dict__ seems to be a new container with its own dict methods...

>>> T.__dict__.items()

>>> T.__dict__.__class__

class C(object, cdict=myDict): pass ??

My best bet is something like metaclasses implemented as class
decorators -- something that is read by the compiler _before_ the
class statement is executed. Being a class (not a function), it could
provide methods to customize some aspects of the class statement
execution; for example, a __getlocals__() method, or something along
these lines. and, of course, it would be able to put the __metaclass__
name right into the locals() dict, which would make it backward
compatible. The problem is, function decorators are only executed
_after_ the function def is executed; and having class decorators work
differently is a big no-no.

ISTM a metaclass is a lot like a decorator for classes. I.e., it is called
just after the class definition executes and before the name is bound.

They are both somewhat like atexit hooked functions. Except that the process
is not an OS process, but just a little segment of the current thread having to
do with building a class. If you looked at it in an OO way, you might have
an object that you instantiate and set up before kicking off its run method.
Then you could have properties that acccumulated chains of decorator functions
to do between the default construct and bind operations, but you could also
have other property/hooks to add e.g., pre-processing of the arg list being
passed to the constructor, and properties to hold alternate components such
as dicts etc. IOW, a big factory object with lots of knobs that ultimately
produces some custom thing. There ought to be some unified concept behind
this process IMO, whether the thing produced is a function or class or type.
They are all factories that compile and pull together pieces and glue or bolt
them together. The tough part is to make it fast with pre-glued assemblies
and yet allow custom bolting together of custom components.
Well, metaphors can only be ridden so far ;-)

Regards,
Bengt Richter

 

[Alex Martelli]

I don't know. I was thinking that maybe an ordereddict class would be
useful, something like:

Yes, a mapping that keeps order can be useful (in either meaning of the
term: remembering order of item insertion [or last modification], or,
intrinsically ordering keys [directly or by value]). There are a bit
too many variants to warrant making them built-ins, but the collections
module, in Python 2.5, could certainly accomodate several.

myDict = {'a': 4, 'b': 5 } # normal dict
myDict = o{'a': 4, 'b': 5 } # ordered dict, keeping order ['a', 'b']

The dictionary could only be use in specific cases, like configuration
stuff or temporarily when instantiating a metaclass... I don't know,
maybe I'm just crazy...

Given the rarity of the need, I very much doubt it makes sense to come
up with special syntax, or implicitly use some choice out of these
special mappings (no doubt slower than regular dicts) in some cases. In
other words, I agree with Carlos that some way to tell Python what class
to use (in lieu of default dict) for class statements -- presumably a
sys.setsomething call -- is preferable; I do agree with you that users
shouldn't need to come up with their own "ordered mappings", the
standard library should provide them.


Alex

 

[Alex Martelli]

The code is as follows -- still with debug code included, and lots of

comments:

Nice! I must be missing something (what's the role of the '__attr__'
key you're setdefaulting into instance dicts and never using
otherwise?), and there may be enhanceable spots such as:

if (isinstance(fobj,StringType) or
isinstance(fobj,IntType) or
isinstance(fobj,FloatType) or
isinstance(fobj,ListType)):

isinstance accepts a tuple as its 2nd arg exactly to avoid this need;
set once, somewhere,

elementaryTypes = StringType, IntType, FloatType, ListType

and then you can typecheck here with just

if isinstance(fobj, elementaryTypes):

....but if this, or the recommendation to follow PEP 8 (space after
comma, etc) is the most important suggestion I have to offer about your
code, I guess that's a good sign!-)


Alex

 

[Peter Otten]

if not isinstance(value, self.mytype):
# if it's a string, tries to convert to the correct
# target type (this is needed because most things read
# from files will be strings anyway)
if isinstance(value, StringType):
value = self.mytype(value)

I haven't checked if this is relevant in the context, but implicit
conversion from string is usually a bad thing. E. g.:
(True, True)

Peter

 

[Thomas Heller]

I can't speak for Nicolas, but I can for myself. There are three
reasons, in my particular case:

-- The order is already there, explicit in the way the code is
written. I see no reason to be forced to state it _again_, specially
considering the fact that this is prone to errors; for example,
missing elements, duplicated elements, or out-of-order elements (in
ordered differently in the code and in the list).

Yes, but a Python pogrammer wouldn't expect these two samples to give
different results:

class A(...):
x = 1
y = 2

class A(...):
y = 2
x = 1

-- I make extensive use of nested structs. In this case it's _much_
easier to make mistakes as the ones mentioned above.

-- Using classes and inheritance, it's easy to add new members or
redefine existing ones. But I would need to re-state the full list for
any descendant. To make matters worse, in this case, whenever I make a
change to the base class, I would have to change _all_ descendant
classes. Not exactly good OO design. Of course, clever hacks could
possibly be used, but then, it wouldn't look any better than Nicolas
(or my own) ordered-attributes hack.

In ctyoes you write:

class POINT(Structure):
_fields_ = [("x", c_int),
("y", c_int)]

which makes it explicit that _fields_ is a list, which *has* a certain
order. And for inheritance, it's easy and explicit as well:

class COLORED_POINT(POINT):
_fields_ = POINT._fields_ + [("color", RGB)]

It's specially important to point out that I consider the source code
ordering to be _explicit_, and not implicit, as some people may
possibly say. In fact, is there anything more explicit than that? Only
people used to Python dicts would say otherwise.

The point I'm trying to make is that too much magic most of the time
catches you later again. But that's only my own experience.

Thomas

 

[Carlos Ribeiro]

comments:

Nice! I must be missing something (what's the role of the '__attr__'
key you're setdefaulting into instance dicts and never using
otherwise?), and there may be enhanceable spots such as:

The relevant part of the code is as follows (debug code removed):

def __get__(self, instance, owner):
if instance:
attrdict = instance.__dict__.setdefault('__attr__', {})
return attrdict.get(self.name, self.value)
else:
return self
def __set__(self, instance, value):
attrdict = instance.__dict__.setdefault('__attr__', {})
attrdict[self.name] = value

The __attr__ is used as part of my descriptor implementation, and it's
an attribute of the instance. Its a dictionary whose keys are the
names of the GenericAttributes (or TypedAttributes) stored in the
instance.

The __get__() code checks if its being called from a class; in this
case, it returns the attribute itself (so I can check its name, and
other properties of the attribute, when working with class
references). But if the attribute is being called from an instance,
then it retrieves the current value of the attribute, as stored in the
instance; if the attribute wasn't initialized, it retrieves the
default value (that is an attribute of the descriptor).

An example may help to clarify things a little bit:

class Person(GenericTemplate)
name = TypedAttribute('unknown')

john = Person()
john.name = "John Doe"
peter = Person()

In this case, john.__attr__['name'] stores the current value of the
name attribute at the instance. john.name will retrieve this value.
peter.__attr__ wasn't still initialized; if you try to retrieve the
name, it will automatically initialize the __attr__ dictionary, and
try to retrieve the value of the 'name' key; as it doesnt exist, it
retrieves the default value.

The class code is also useful. If you refer to Person.name, it will
return the attribute itself, which allows one to retrieve its own
name:

obj = Person.name
print obj.name

isinstance accepts a tuple as its 2nd arg exactly to avoid this need;
set once, somewhere,

elementaryTypes = StringType, IntType, FloatType, ListType

This is one of the old artifacts. I had four different tests, and for
each one of them I had a special subclass: ListAttribute,
StrAttribute, and so on. It went this way because at first I only
wrote the StrAttribute, then added another one... And I'm doing it
XP-style, "the simplest code that works". At some point I finally
managed to 'get' descriptors, and implemented the TypedAttribute
class. I simply collapsed the tests into a single one (I didn't even
retype them! just deleted the intermediate lines and "and'ed" them
together; now talk about "the simplest code that works" ).

In fact, there are several parts of the code that are artifacts of
some experiments that I did. I was trying to figure out how to
properly use metaclasses, descriptors, properties, a lot of stuff.
Some of the old code is still there. I also kept debug code, and I
intend to... this is something that still bothers me and is a good
topic for another thread... Debug code clutters the main code, but I
don't like to have to add it back if I ever need it. (what about
logging support at the language level, pretty much as a assert, that
can be removed from optimized code?)

I have finished right now a unittest for the metatemplate module,
including inheritance testing. Now I can start to remove the old cruft
step by step without fear that everything will break apart

> and then you can typecheck here with just

if isinstance(fobj, elementaryTypes):

...but if this, or the recommendation to follow PEP 8 (space after
comma, etc) is the most important suggestion I have to offer about your
code, I guess that's a good sign!-)

I re-read PEP8 a few days ago. Some things are a matter of personal
style, and a little harder to change; for instance, I prefer to align
sequences of assignments vertically, something that Guido reportedly
loathes. But I'm already adhering to some recommendations, such as
right-margin and docstrings format.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Carlos Ribeiro]

I haven't checked if this is relevant in the context, but implicit
conversion from string is usually a bad thing. E. g.:

(True, True)

I had to do it to be able to read values written in configuration
files. When I write the attributes to the text file, I lose the type
information, and all that I have when I re-read them is the string
representation. In this case, the best that I can do is to try to
force the conversion from the string to the correct type.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Carlos Ribeiro]

Yes, but a Python pogrammer wouldn't expect these two samples to give
different results:

class A(...):
x = 1
y = 2

class A(...):
y = 2
x = 1

I sincerely dont agree with this argument. A *experienced* Python
programmer wouldn't assume that these two samples would give different
results, because he *knows* that ordering doesn't matter. But many
programmers would be surprised if they tried to iterated over the
fields and had them return out of the original definition ordering.
It's not natural; it's something that you have to explain _why_.

which makes it explicit that _fields_ is a list, which *has* a certain
order. And for inheritance, it's easy and explicit as well:

class COLORED_POINT(POINT):
_fields_ = POINT._fields_ + [("color", RGB)]

I agree that it's clear, and this is not a strong argument that I
could use for my own case. My own approach has to handle similar
situations for nested classes, also.

The point I'm trying to make is that too much magic most of the time
catches you later again. But that's only my own experience.

Agreed. But the lack of ordering (which is really a side effect of the
use of the dict mapping to store class attributes, and not a clear
design decision in itself) is also something that strikes newcomers as
a 'surprise factor'; in a sense, I think this makes the field even.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Michael Hudson]

IIRC it was Guido himself who let the cat out of the bag about a
function, in some early metaclass notes. But maybe I don't RC ;-)

I don't get the fuss about a function vs a class for the value of
__metaclass__. It's a callable. It gets called. No mystery *here*


Cheers,
mwh

 

[Carlos Ribeiro]

I don't get the fuss about a function vs a class for the value of
__metaclass__. It's a callable. It gets called. No mystery *here*

I think it has something to do with the intended working of
metaclasses. After all, they're called meta_classes_, not
metafunctions But on a more serious note: in other languages, such
as Smalltalk, metaclasses are a standard part of the language, and are
_classes_. There is a _metaclass hierarchy_, superimposed to the class
hierarchy. Metaclasses are supposed to be classes because they can be
inherited from, for example. And I believe that was only by accident,
and not by a explicit design choice, that Python's implementation
accepts any callable as a metaclass.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Michele Simionato]

I think it has something to do with the intended working of
metaclasses. After all, they're called meta_classes_, not
metafunctions But on a more serious note: in other languages, such
as Smalltalk, metaclasses are a standard part of the language, and are
_classes_. There is a _metaclass hierarchy_, superimposed to the class
hierarchy. Metaclasses are supposed to be classes because they can be
inherited from, for example. And I believe that was only by accident,
and not by a explicit design choice, that Python's implementation
accepts any callable as a metaclass.

Using a "metafunction" is a kind of abuse, yes. Nevertheless, I used this trick
to resolve the metaclass conflict:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/204197c

Later on, I also found a way to avoid the usage of the meta-function,
but it was too brain-exploder to publish ;-)

Michele Simionato

 

[Ville Vainio]

Michele> Using a "metafunction" is a kind of abuse,
Michele> yes. Nevertheless, I used this trick to resolve the
Michele> metaclass conflict:

Michele> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/204197c

Broken link, I guess you meant:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/204197

I don't think the term "metafunction" is applicable, because calling
it doesn't return a function, and everything that returns a callable
object is at some level a metafunction (at least in the sense
metaclass is a metaclass).

 

[Carlos Ribeiro]

Michele> Using a "metafunction" is a kind of abuse,
Michele> yes. Nevertheless, I used this trick to resolve the
Michele> metaclass conflict:

Michele> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/204197c

Broken link, I guess you meant:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/204197

I don't think the term "metafunction" is applicable, because calling
it doesn't return a function, and everything that returns a callable
object is at some level a metafunction (at least in the sense
metaclass is a metaclass).

I used the term "metafunction" originally as a joke, followed by a
smile For the lack of a better name, it was a reasonable play of
words at that particular time... but sure, a better name should exist.
As it is now, it's a degenerate metaclass of sorts -- pretty much like
a errant virotic RNA strand, lacking even the proteine capsule that
proper virus have... It's hard to classify.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)