Decorator metaclass

[thomas.karolski]

Hi,
I would like to create a Decorator metaclass, which automatically
turns a class which inherits from the "Decorator" type into a
decorator.
A decorator in this case, is simply a class which has all of its
decorator implementation inside a decorator() method. Every other
attribute access is being proxied to decorator().getParent().

Here's my attempt:
-------------------------------------------------------
from new import classobj

class DecoratorType(type):
def __new__(cls, name, bases, dct):
dct2 = {}
for key, val in dct.iteritems():
dct2[key] = val

# create a new class which will store all of the implementation
impl = classobj('%sImpl'%name,(),dct2)

# update the old class to implement this implementation
def __init__(self, *args, **dargs):
object.__setattr__(self, '__impl', impl(*args, **dargs))
def decorator(self):
return object.__getattribute__(self,'__impl')
def __getattribute__(self, attr):
if attr=="decorator":
return object.__getattribute__(self,'decorator')
return getattr(object.__getattribute__(self, 'decorator')
().getParent(), attr)
dct = {}
dct['__init__'] = __init__
dct['decorator'] = decorator
dct['__getattribute__'] = __getattribute__

return type.__new__(cls, name, bases, dct)

class Decorator(object):
__metaclass__ = DecoratorType

class HBar(Decorator):
def __init__(self, number):
Decorator.__init__(self)
self._number = number
def inc(self):
self._number += 1
def p(self):
print self._number

hbar = HBar(10)
for each in dir(hbar.decorator()):
print each

hbar.decorator().p()
hbar.decorator().inc()
hbar.decorator().p()
-------------------------------------------------------
Unfortunately this does not work. The newly defined __init__ method
inside __new__, does a call to impl(*args, **dargs). However, since
the HBar.__init__ calls the Decorator.__init__ method, but the
HBar.__init__ method no longer resides inside HBar, but rather inside
HBarImpl (which is no longer a subtype of Decorator), the compiler
complains that Decorator.__init__ is not being called with a Decorator
instance as its first argument (which is true).
I tried changing the definition of impl inside __new__ to have
Decorator as one of its bases, but then for some reason impl(*args,
**dargs) asks for 4 arguments (just like __new__) and I have no clue
as to why that happens.

Any help on this?

Regards,
Thomas K.

 

[Maric Michaud]

Le Friday 23 May 2008 04:28:22 (e-mail address removed), vous avez
écrit :

Hi,
I would like to create a Decorator metaclass, which automatically
turns a class which inherits from the "Decorator" type into a
decorator.
A decorator in this case, is simply a class which has all of its
decorator implementation inside a decorator() method. Every other
attribute access is being proxied to decorator().getParent().

...

-------------------------------------------------------
Unfortunately this does not work. The newly defined __init__ method
inside __new__, does a call to impl(*args, **dargs). However, since
the HBar.__init__ calls the Decorator.__init__ method, but the
HBar.__init__ method no longer resides inside HBar, but rather inside
HBarImpl (which is no longer a subtype of Decorator), the compiler
complains that Decorator.__init__ is not being called with a Decorator
instance as its first argument (which is true).
I tried changing the definition of impl inside __new__ to have
Decorator as one of its bases, but then for some reason impl(*args,
**dargs) asks for 4 arguments (just like __new__) and I have no clue
as to why that happens.

Any help on this?

The problem with kind of design is that you must break the rules of class
inheritance, and it seems like a strange idea to implement decorators by
inheritance.

Of course you could do all sort of magic with python, but what is your goal ?
In your example, should the implementation types inherit from each other ?
In that case, do you want to preserve the same semantic for __init__ as in
standard python class (this could be a hard job) ?

This quick fix seems to work with your example, but add extra magic to
automatically call the super __init__ of the parent implementation, this
could be a bad idea, use with caution ! (I still think it's a bad design,
using composition and proxy classes is much more simple and clear)

class DecoratorType(type):
def __new__(cls, name, bases, dct):

# create a new class which will store all of the
implementation
parent_impl_type = bases[0] is object and object \
or bases[0]._impl_type
impl = type('%sImpl'%name,(parent_impl_type,),dict(dct))
dectype = type.__new__(cls, name, bases, {'_impl_type' :
impl })

# update the old class to implement this implementation
def __init__(self, *args, **dargs):
print args, dargs
new_impl = impl(*args, **dargs)
super(dectype._impl_type, new_impl).__init__(*args,
**dargs)
object.__setattr__(self, '_impl', new_impl)
def decorator(self):
return object.__getattribute__(self,'_impl')
def __getattribute__(self, attr):
if attr=="decorator":
return
object.__getattribute__(self,'decorator')
return getattr(object.__getattribute__(
self, 'decorator')(), attr)
dectype.__init__ = __init__
dectype.decorator = decorator
dectype.__getattribute__ = __getattribute__

return dectype

class Decorator(object):

__metaclass__ = DecoratorType

class HBar(Decorator):
def __init__(self, number):
print 'hb:', number
self._number = number
def inc(self):
self._number += 1
def p(self):
print self._number

class HBar2(HBar) :
def __init__(self, number):
print 'hb2:', number
self._hb2 = number
def inc2(self):
self._hb2 += 1
def p2(self):
print self._hb2


hbar = HBar(10)
for each in dir(hbar.decorator()):
print each

hbar.decorator().p()
hbar.decorator().inc()
hbar.decorator().p()

hb2 = HBar2(5)
hb2.p()
hb2.p2()
hb2.inc()
hb2.p()
hb2.p2()
hb2.inc2()
hb2.p()
hb2.p2()

 

[Carl Banks]

Hi,
I would like to create a Decorator metaclass, which automatically
turns a class which inherits from the "Decorator" type into a
decorator.
A decorator in this case, is simply a class which has all of its
decorator implementation inside a decorator() method. Every other
attribute access is being proxied to decorator().getParent().

Here's my attempt:

You got deep stuff going on there, chief, and some of it's wrong.
I'll try to point it out.

-------------------------------------------------------
from new import classobj

class DecoratorType(type):
def __new__(cls, name, bases, dct):
dct2 = {}
for key, val in dct.iteritems():
dct2[key] = val

First of all, you can just do dct2 = dct.copy().
Second, since you never use dct again, even copying it is unnecessary.

# create a new class which will store all of the implementation
impl = classobj('%sImpl'%name,(),dct2)

classobj creates an old-style class, and I'm pretty sure you don't
want to do that. To create a new-style class, use type:

impl = type('%sImpl'%name,(),dct)

# update the old class to implement this implementation
def __init__(self, *args, **dargs):
object.__setattr__(self, '__impl', impl(*args, **dargs))

As your code stands now, object.__setattr__ isn't necessary here; just
using

self.__impl = impl(*args,**dargs)

should work fine. I'm guessing you intend to override __setattr__
later?

If you do use object.__setattr__, I suggest that you might want to
call the superclass's __setattr__ instead of object's. I imagine in
this case the superclass will rarely want to override __setattr__
itself, but in general it's a good idea. In this particular
circumstance, we don't yet have the class object (it doesn't come till
after calling type.__new__) but we do have the parent class. So you
might consider changing the definition of __init__ to this:

basecls = bases[0] if bases else object
def __init__(self, *args, **dargs):
basecls.__setattr__(self, '__impl', impl(*args, **dargs))

def decorator(self):
return object.__getattribute__(self,'__impl')

Again, consider changing it to

def decorator(self):
return basecls.__getattribute(self,'__impl')

def __getattribute__(self, attr):
if attr=="decorator":
return object.__getattribute__(self,'decorator')
return getattr(object.__getattribute__(self, 'decorator')
().getParent(), attr)
dct = {}
dct['__init__'] = __init__
dct['decorator'] = decorator
dct['__getattribute__'] = __getattribute__

return type.__new__(cls, name, bases, dct)

class Decorator(object):
__metaclass__ = DecoratorType

Parenthetical: I don't normally recommend this style, since it
obscures the fact that you're using a custom metaclass to the user.
That is something the user probably would benefit from knowing, if for
no other reason than so they can make a mental note about where to
look first if something goes wrong. I prefer to make the user use the
__metaclass__ attribute.

However, I could see it being desirable for some cases where you're
trying to be as transparent as possible, and indeed it looks as if
that's your goal here.

class HBar(Decorator):
def __init__(self, number):
Decorator.__init__(self)

Ok, at this point you have to ask yourself what you want to do,
because the solution you choose will involve trade-offs.

You will note that Decorator does not define __init__. In fact,
object.__init__ will be called, which does nothing. If you think that
all classes with DecoratorType as their metaclass will be a direct
subclass of Decorator, you can get away with not calling
Decorator.__init__ at all.

However, this can cause problems if a user wants to define their own
base class with an __init__ that does something (either by using the
__metaclass__ attribute, or by subclassing a Decorator subclass). In
that case, you will have to make arrangements to pass the decorator
object to the superclass instead of the decorated. This can be pretty
hairy, and it beyond the scope of this reply.

To do it completely transparently, your decorated class will probably
have to maintain a reference to its decorator, and will also have to
derive from a base class that delegates any method calls to the
superclass of the decorator. (Phew.)

That won't be as easy as it sounds.

self._number = number
def inc(self):
self._number += 1
def p(self):
print self._number

hbar = HBar(10)
for each in dir(hbar.decorator()):
print each

hbar.decorator().p()
hbar.decorator().inc()
hbar.decorator().p()
-------------------------------------------------------
Unfortunately this does not work. The newly defined __init__ method
inside __new__, does a call to impl(*args, **dargs). However, since
the HBar.__init__ calls the Decorator.__init__ method, but the
HBar.__init__ method no longer resides inside HBar, but rather inside
HBarImpl (which is no longer a subtype of Decorator), the compiler
complains that Decorator.__init__ is not being called with a Decorator
instance as its first argument (which is true).
I tried changing the definition of impl inside __new__ to have
Decorator as one of its bases, but then for some reason impl(*args,
**dargs) asks for 4 arguments (just like __new__) and I have no clue
as to why that happens.

I believe it's happening because you mixed old-style and new-style
classes. But it's not the right solution anyway.

Any help on this?

Probably the best piece of advice is "Don't try to use Decorator
pattern".

Seriously, you might want to see what other people have done in
similar cases. This stuff is tricky to get right, so maybe you should
shamelessly ride the coattails of someone who already ran into all the
issues. One example I can think of is the ZODB Persistent class (it's
a proxy class, so some of the same issues are involved). Perhaps
searching Python cookbook for some proxy or decorator class recipes
will give you ideas.


Carl Banks

 

[Thomas Karolski]

Thanks for pointing out all those mistakes. I think I'm already starting
to grasp all of the python magic going on in there.

Parenthetical: I don't normally recommend this style, since it
obscures the fact that you're using a custom metaclass to the user.
That is something the user probably would benefit from knowing, if for
no other reason than so they can make a mental note about where to
look first if something goes wrong. I prefer to make the user use the
__metaclass__ attribute.

Really just personal preference I think. I'm not really a friend of
declaring variables if there is a more "intuitive" way.

Ok, at this point you have to ask yourself what you want to do,
because the solution you choose will involve trade-offs.

Yes, it was probably a bad example. I decided not to call the Decorator
's __init__ method in my new version (which I have posted as a reply to
the reply of Maric Michaud).

You will note that Decorator does not define __init__. In fact,
object.__init__ will be called, which does nothing. If you think that
all classes with DecoratorType as their metaclass will be a direct
subclass of Decorator, you can get away with not calling
Decorator.__init__ at all.

Now, inside my new version, I have a class which inherits from both
Decorator and Window, out of which the __init__ for Decorator is not
called. Does this prove to be a problem?

Probably the best piece of advice is "Don't try to use Decorator
pattern".

Well, I decided on the decorator pattern, because I want to be able to
change the behavior of classes during run-time. I did not really find
any other pattern which would let me do that.

Regards,
Thomas K.

 

[Carl Banks]

Now, inside my new version, I have a class which inherits from both
Decorator and Window, out of which the __init__ for Decorator is not
called. Does this prove to be a problem?

It sounds like it could create infinite recursion. If your decorated
class inherits from Decorator, it should also get the DecorateType
metaclass, which should recursively try to create its own decorated
class, ad infinitum. Unless I misunderstand what you meant.

Well, I decided on the decorator pattern, because I want to be able to
change the behavior of classes during run-time. I did not really find
any other pattern which would let me do that.

Ah.

Well it is possible to do that in Python. though there's probably not
an official design pattern for it (but then design patterns grew up
around less flexible languages, partially as a way to cope with their
lack of flexibility). Here are a couple things to think about:

If you'd like to change the behavior of individual instances of a
class, you can assign functions to individual instances which act just
like methods. (Warning: this does not work with operators.)

Here is an example:

class A(object):
def say_hello(self):
print "hello, world"

a = A()

def custom_hello():
print "hello, my name is Inigo Montoya"

a.say_hello = custom_hello


If you'd like to change the behavior of all instances of the class,
then you can assign a new method directly to the class after it was
created:

def new_default_hello(self):
print "hello, children"

A.say_hello = new_default_hello


Notice that you need to use self when assigning it to the class
object, and not to use self when assigning it to an instance of the
class.


Carl Banks

 

[Thomas Karolski]

Turns out the first msg I sent did not reach the list, so I'll just post
what I've achieved by now:
------------------------------------------
class DecoratorDummy(object): pass

class InheritedDecoratorType(type):
def __new__(cls, name, bases, dct):
# return if its a class which inherited from Decorator
if Decorator in bases:
return type.__new__(cls, name, bases, {})

# if its a class which did not directly inherit from Decorator,
# then it inherited from a class which has been manipulated using the
# Decorator class.
# in that case we change the bases of the inheriting class.
# We'll split the manipulated class into Decorator and its implementation
for b in bases:
if type(b) is InheritedDecoratorType:
break
newbases = [x for x in bases]
# remove the manipulated base class
newbases.remove(b)
# and add the impl of the manipulated base class
newbases.append(b._impl_type)
# and add the Decorator class
newbases.append(Decorator)
# now we'll have to make sure the dict of the new class shows the
original base class
# (which has been removed) as the implementation class
dct[b.__name__] = b._impl_type
# since we have added a Decorator class, we ought to get rid of it
# through the DecoratorType metaclass
r = DecoratorType.__new__(cls, name, tuple(newbases), dct)

return r

class DecoratorType(type):
def __new__(cls, name, bases, dct):

# if the first class is DecoratorDummy, then we're handling the
# Decorator class, which is not supposed to be modified
if bases[0] is DecoratorDummy:
return type.__new__(cls, name, bases, {})

# if one of the bases is the Decorator class
b = [x for x in bases]
if Decorator in b:
# create a new impl type which inherits from every but the decorator
class
b.remove(Decorator)

impl_type = type('%sImpl'%name, tuple(b), dict(dct))
# make the returned type no longer a DecoratorType, but rather a
normal type
# Types which inherit from a class which inherited from Decorator,
will thus
# *not* be put through this metaclass.
#dectype = type.__new__(type, name, tuple(b), {'_impl_type' : impl_type })
dectype = type.__new__(InheritedDecoratorType, name, tuple(b),
{'_impl_type' : impl_type })

# update the old class to implement this implementation
def __init__(self, *args, **dargs):
new_impl = impl_type(*args, **dargs)
super(dectype._impl_type, new_impl).__init__(*args,
**dargs)
object.__setattr__(self, '_impl', new_impl)
def decorator(self):
return object.__getattribute__(self, '_impl')
def __getattribute__(self, attr):
if attr=="decorator":
return object.__getattribute__(self, 'decorator')

# if we have a specified method inside the decorator().__decorate__ var,
# then call decorator().attr(), otherwise proxy the call
d = object.__getattribute__(self, 'decorator')()
if attr in d.__decorate__:
return getattr(d, attr)
return getattr(d.getParent(), attr)
dectype.__init__ = __init__
dectype.decorator = decorator
dectype.__getattribute__ = __getattribute__

return dectype

class Decorator(DecoratorDummy):
__metaclass__ = DecoratorType

class Window(object):
def __init__(self, parent):
print "Window::__init__(%s)"%self
self._parent = parent

def setParent(self, parent):
self._parent = parent

def getParent(self):
return self._parent

class Textarea(Window):
def draw(self):
print "Textarea::draw()"

class HBar(Decorator, Window):
__decorate__ = ["draw"]
def __init__(self, parent):
print "HBar::__init__(%s)"%self
Window.__init__(self, parent=parent)

self._progress = 0.0

def setProgress(self, p):
print "setting progress to %s"%p
self._progress= p

def getProgress(self):
return self._progress

def draw(self):
self.getParent().draw()
print "HBar::draw()"

class HBarTrue(HBar):
# HBar's bases: Window (Decorator removed by metaclass)
# HBar's methods: __init__, decorator, __getattribute__
# everything else is inside decorator()
# we thus need to make calls to self within HBarTrue,
# calls to self.decorator() - this is not possible
# Otherwise we could also let HBarTrue inherit from HBarImpl,
# however then a call to HBar.__init__ would be invalid inside
# HBarTrue.__init__ unless we specify HBar internally as being HBarImpl
# - this however is not possible
# if we inherit from HBarImpl, then we no longer have the decorator
# functionality. We'd thus have to include Decorator in the list of bases

# Inherit normally from HBar and not Decorator.
# Move all methods from HBarTrue to HBar.decorator()
# create a custom __init__ method which calls HBar.decorator().__init__

def __init__(self, parent):
print "HBarTrue::__init__(%s)"%self
for each in dir(self):
print each
HBar.__init__(self, parent)

self.setProgress(0.0)

myTextarea = Textarea("main.parent")
myTextarea.draw()
myTextarea = HBarTrue(myTextarea)
myTextarea.draw()
myTextarea.decorator().setProgress(100.0)
------------------------------------------
The code above works only if I don't inherit from HBar like I did.
As it is now, the HBar class, which inherits from Decorator and Window,
is going to be manipulated by the metaclass. During this process the
Decorator base-class is removed, the implementation of HBar moved into
_impl_type and the methods __init__, decorator and __getattribute__ are
being assigned to the new class.
Everything works so far if I don't subclass HBar.
If I do subclass HBar, then the subclass goes through the second
metaclass (InheritedDecoratorType). In there I replace the base class
HBar with HBarImpl and Decorator. This way the subclass inherits the
implementation of HBar and through the Decorator class then the subclass
goes through the same metaclass magic as HBar in the first step. Of
course however, since HBarTrue is no longer a subclass of HBar (since
that baseclass has been removed), the HBar.__init__ method won't accept
the non HBar self parameter.

Now the reason why I'm using decorators, is because I want to be ably to
add the functionality dynamicly - without the need of construction
classes for the different possibilities. Composite pattern does not help
in this case, since I lose the ability to choose in what order I call
the decorator's and the decorated's methods.

I'll just keep trying. Any input greatly appreciated.

Regards,
Thomas K.

 

[Gabriel Genellina]

Turns out the first msg I sent did not reach the list, so I'll just post
what I've achieved by now:

[snip a couple of long metaclasses]

Now the reason why I'm using decorators, is because I want to be ably to
add the functionality dynamicly - without the need of construction
classes for the different possibilities. Composite pattern does not help
in this case, since I lose the ability to choose in what order I call
the decorator's and the decorated's methods.

I haven't looked in detail to your code, but if that is what you want, you don't need a Decorator metaclass at all. Just add/replace the desired methods inside the class itself (Python is a dynamic language, remember).

py> class A(object):
.... def foo(self): print "In foo, self=", self
....
py> def bar(self): print "In bar, self=", self
....
py> a = A()
py> a.foo()
In foo, self= <__main__.A object at 0x00A3CD50>
py>
py> A.bar = bar
py> a.bar
<bound method A.bar of <__main__.A object at 0x00A3CD50>>
py> a.bar()
In bar, self= <__main__.A object at 0x00A3CD50>

You can even add methods to individual instances:

py> def baz(self): print "In baz, self=", self
....
py> import new
py> b = A()
py> b.baz = new.instancemethod(baz, b, type(b))
py> a.baz()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'A' object has no attribute 'baz'
py> b.baz
<bound method A.baz of <__main__.A object at 0x00A3CDD0>>
py> b.baz()
In baz, self= <__main__.A object at 0x00A3CDD0>