Metaclass to make all methods of a class thread-safe

[Irmen de Jong]

Hi,
I've developed the Metaclass below, because I needed a way
to make a bunch of classes thread-safe.
I didn't want to change every method of the class by adding
lock.aqcuire()..lock.release() around the existing code.
So I made a metaclass that essentially replaces every method
of a class with a 'wrapper' method, that does the locking,
invocation, unlocking.

Is this the right approach? It seems to work fine. But I have
very little experience with metaclass programming, so I'd like
to hear some feedback.

Thanks !!

--Irmen de Jong.


# ---- source below ----

from types import FunctionType
from threading import RLock

class ThreadSafeMethodsMetaClass(type):
def __new__(meta, name, bases, dict):
meta.convert_methods(dict)
return super(ThreadSafeMethodsMetaClass, meta).__new__(meta, name, bases, dict)

def makeThreadsafeMethod(func):
def threadsafemethod(self, *args, **kwargs):
self._monitor_lockObj.acquire()
print ">>got lock"
try:
return func(self, *args, **kwargs)
finally:
self._monitor_lockObj.release()
print "<<released lock"
return threadsafemethod
makeThreadsafeMethod = staticmethod(makeThreadsafeMethod)

def convert_methods(cls, dict):
methods=[ v for k,v in dict.iteritems() if isinstance(v, FunctionType) ]
for m in methods:
dict[m.__name__]=cls.makeThreadsafeMethod(m)
dict["_monitor_lockObj"] = RLock()

convert_methods = classmethod(convert_methods)


class MyClass(object):
__metaclass__=ThreadSafeMethodsMetaClass

def __init__(self):
print "init!"
def method(self, a1, a2):
print a1,a2

m=MyClass()
m.method("irmen",42)

 

[Michele Simionato]

Hi,
I've developed the Metaclass below, because I needed a way
to make a bunch of classes thread-safe.
I didn't want to change every method of the class by adding
lock.aqcuire()..lock.release() around the existing code.
So I made a metaclass that essentially replaces every method
of a class with a 'wrapper' method, that does the locking,
invocation, unlocking.

Is this the right approach? It seems to work fine. But I have
very little experience with metaclass programming, so I'd like
to hear some feedback.

Thanks !!

--Irmen de Jong.

Well, it looks okay, but consider the following:

1. you have a (metaclass) static method and a (metaclass) classmethod
which could be replaced by simple functions external to the metaclass;
this would make the code much easier to read and to understand; I don't
buy the argument that they should logically stay in the metaclass, it
is enough if they stay in the same module of the metaclass, not inside it;

2. the metaclass will automagically wrap even methods of subclasses, without
you knowing it; consider using a naming convention (es. only methods
starting with "t_" are magically wrapped); if still
you want the magic, consider defining a name convention such as methods
starting with a given prefix are NOT magically wrapped;

4. built-in methods and all the objects which are non instances of FunctionType
will be not wrapped; you may want this or not;

5. consider using decorators to wrap the methods you want to be
thread safe: they a more esplicit and easier to understand solution;
also in this way you will avoid the (possible) issue of metaclass
conflicts.

6. However, if you want to enhance a code which is already written
with a minimal change of the source code, the metaclass is the
simplest solution indeed.

Just my 0.02c,

Michele Simionato

 

[Irmen de Jong]

Well, it looks okay, but consider the following:

1. you have a (metaclass) static method and a (metaclass) classmethod
which could be replaced by simple functions external to the metaclass;
this would make the code much easier to read and to understand; I don't
buy the argument that they should logically stay in the metaclass, it
is enough if they stay in the same module of the metaclass, not inside it;

Agreed, I was just modeling it after some example metaclass code that I found
on ASPN cookbook. The metaclass itself will only be a few lines then

2. the metaclass will automagically wrap even methods of subclasses, without
you knowing it; consider using a naming convention (es. only methods
starting with "t_" are magically wrapped); if still
you want the magic, consider defining a name convention such as methods
starting with a given prefix are NOT magically wrapped;

This is good advice. I started by not wrapping methods with '__' prefix...

4. built-in methods and all the objects which are non instances of FunctionType
will be not wrapped; you may want this or not;

Built-in methods such as?
About the other objects: I only cared about wrapping class methods.
Shouldn't I have / should I use something else than FunctionType?

There is one thing though; methods that you're accessing trough the
class's __dict__ (which is what the meta class is doing, right?)
.... def meth(self): pass
....

>>> type(A.meth)

>>> type(A.__dict__['meth'])

>>>

Why is this?

5. consider using decorators to wrap the methods you want to be
thread safe: they a more esplicit and easier to understand solution;
also in this way you will avoid the (possible) issue of metaclass
conflicts.

No decorators, nooo sir, it must work on Python 2.3 too

6. However, if you want to enhance a code which is already written
with a minimal change of the source code, the metaclass is the
simplest solution indeed.

Thanks for your comments, Michele.

--Irmen

 

[Jeff Shannon]

There is one thing though; methods that you're accessing trough the
class's __dict__ (which is what the meta class is doing, right?)
... def meth(self): pass
...

type(A.meth)

type(A.__dict__['meth'])

Why is this?

I don't know for certain, but it occurs to me that the object returned
by A.meth will insert A in front of all other arguments to the method,
while the object returned by A.__dict__['meth'] does not do that. Logic
seems to suggest that the instancemethod type performs this
argument-list-mangling and then passes the results on to the
FunctionType object contained in __dict__ -- that is, instancemethod is
essentially a function wrapper that provides the standard method-call
translations for class instances.

Jeff Shannon
Technician/Programmer
Credit International

 

[Michele Simionato]

Built-in methods such as?

This is instructive:
....
__class__ <type 'type'>
__delattr__ <type 'wrapper_descriptor'>
__dict__ <type 'dictproxy'>
__doc__ <type 'NoneType'>
__getattribute__ <type 'wrapper_descriptor'>
__hash__ <type 'wrapper_descriptor'>
__init__ <type 'wrapper_descriptor'>
__module__ <type 'str'>
__new__ <type 'builtin_function_or_method'>
__reduce__ <type 'method_descriptor'>
__reduce_ex__ <type 'method_descriptor'>
__repr__ <type 'wrapper_descriptor'>
__setattr__ <type 'wrapper_descriptor'>
__str__ <type 'wrapper_descriptor'>
__weakref__ <type 'getset_descriptor'>


For instance object.__new__ is a "builtin_function_or_method"

About the other objects: I only cared about wrapping class methods.
Shouldn't I have / should I use something else than FunctionType?

Yes, if you want to wrap staticmethods/classmethods and custom descriptors.

There is one thing though; methods that you're accessing trough the
class's __dict__ (which is what the meta class is doing, right?)
... def meth(self): pass
...

type(A.meth)

type(A.__dict__['meth'])

Why is this?

Google for Raymond Hettinger essay on descriptors and you will have
the answer. There is no real difference between functions and methods,
they are all descriptors and can be converted each other. For instance,
this converts a function to a bound method:
<method-wrapper object at 0x403b7f4c>

You should really look at Raymond's essay to understand what is going on
under the cover when you write
<bound method C.f of <__main__.C object at 0x403b7f0c>>

Michele Simionato

 

[Irmen de Jong]

Michele Simionato wrote:

[...interesting stuff about descriptors and method types...]

Thanks for this information, I have something to study
if I want to make my next metaclass

--Irmen