decorating functions with generic signatures (not for the faint of heart)

[Michele Simionato]

I have realized today that defining decorators for functions
with generic signatures is pretty non-trivial.

Consider for instance this typical code:

#<traced_function.py>

def traced_function(f):
def newf(*args, **kw):
print "calling %s with args %s, %s" % (f.__name__, args, kw)
return f(*args, **kw)
newf.__name__ = f.__name__
return newf

@traced_function
def f1(x):
pass

@traced_function
def f2(x, y):
pass

#</traced_function.py>

This is simple and works:
calling f2 with args (1, 2), {}

However, there is a serious disadvantage: the decorator replaces
a function with a given signature with a function with a generic
signature. This means that the decorator is *breaking pydoc*!

$ pydoc2.4 traced_function.f1
Help on function f1 in traced_function:

traced_function.f1 = f1(*args, **kw)

You see that the original signature of f1 is lost: even if I will get
an error when I will try to call it with a wrong number of arguments,
pydoc will not tell me that :-(

The same is true for f2:

$ pydoc2.4 traced_function.f2
Help on function f2 in traced_function:

traced_function.f2 = f2(*args, **kw)

In general all functions decorated by 'traced_function' will have the
same (too much) generic signature. This is a disaster for people
like me that rely heavily on Python introspection features.

I have found a workaround, by means of a helper function that
simplifies
the creation of decorators. Let's call this function 'decorate'.
I will give the implementation later, let me show how it works first.

'decorate' expects as input two functions: the first is the function
to be decorated (say 'func'); the second is a caller function
with signature 'caller(func, *args, **kw)'.
The caller will call 'func' with argument 'args' and 'kw'.
'decorate' will return a function *with the same signature* of
the original function, but enhanced by the capabilities provided
by the caller.

In our case we may name the caller function 'tracer', since
it just traces calls to the original function. The code makes
for a better explanation:

#<traced_function2.py>

from decorate import decorate

def tracer(f, *args, **kw):
print "calling %s with args %s, %s" % (f.func_name, args, kw)
return f(*args, **kw)

def traced_function(f):
"This decorator returns a function decorated with tracer."
return decorate(f, tracer)

@traced_function
def f1(x):
pass

@traced_function
def f2(x, y):
pass

#</traced_function2.py>

Let me show that the code is working:
calling f2 with args (1, 2), {}

Also, pydoc gives the right output:

$ pydoc2.4 traced_function2.f2
Help on function f1 in traced_function2:

traced_function2.f1 = f1(x)

$ pydoc2.4 traced_function2.f2
Help on function f2 in traced_function2:

traced_function2.f2 = f2(x, y)

In general all introspection tools using inspect.getargspec will
give the right signatures (modulo bugs in my implementation of
decorate).

All the magic is performed by 'decorate'. The implementation of
'decorate' is not for the faint of heart and ultimately it resorts
to 'eval' to generate the decorated function. I guess bytecode
wizards here can find a smarter way to generate the decorated function.
But my point is not about the implementation (which is very little
tested
at the moment). My point is that I would like to see something like
'decorate' in the standard library.
I think somebody already suggested a 'decorator' module containing
facilities to simplify the usage of decorators. This post is meant as
a candidate for that module. In any case, I think 'decorate' makes a
good example of decorator pattern.

Here is my the current implementation (not very tested):

#<decorate.py>

def _signature_gen(varnames, default_args, n_args, rm_defaults=False):
n_non_default_args = n_args - len(default_args)
non_default_names = varnames[:n_non_default_args]
default_names = varnames[n_non_default_args:n_args]
other_names = varnames[n_args:]
n_other_names = len(other_names)
for name in non_default_names:
yield "%s" % name
for name, default in zip(default_names, default_args):
if rm_defaults:
yield name
else:
yield "%s = %s" % (name, default)
if n_other_names == 1:
yield "*%s" % other_names[0]
elif n_other_names == 2:
yield "*%s" % other_names[0]
yield "**%s" % other_names[1]

def decorate(func, caller):
argdefs = func.func_defaults or ()
argcount = func.func_code.co_argcount
varnames = func.func_code.co_varnames
signature = ", ".join(_signature_gen(varnames, argdefs, argcount))
variables = ", ".join(_signature_gen(varnames, argdefs, argcount,
rm_defaults=True))
lambda_src = "lambda %s: call(func, %s)" % (signature, variables)
dec_func = eval(lambda_src, dict(func=func, call=caller))
dec_func.__name__ = func.__name__
dec_func.__doc__ = func.__doc__
dec_func.__dict__ = func.__dict__.copy()
return dec_func

#</decorate.py>

 

[Michele Simionato]

I said it was very little tested!

This should work better:

#<decorate.py>

def _signature_gen(varnames, n_default_args, n_args,
rm_defaults=False):
n_non_default_args = n_args - n_default_args
non_default_names = varnames[:n_non_default_args]
default_names = varnames[n_non_default_args:n_args]
other_names = varnames[n_args:]
n_other_names = len(other_names)
for name in non_default_names:
yield "%s" % name
for i, name in enumerate(default_names):
if rm_defaults:
yield name
else:
yield "%s = arg[%s]" % (name, i)
if n_other_names == 1:
yield "*%s" % other_names[0]
elif n_other_names == 2:
yield "*%s" % other_names[0]
yield "**%s" % other_names[1]

def decorate(func, caller):
argdefs = func.func_defaults or ()
argcount = func.func_code.co_argcount
varnames = func.func_code.co_varnames
signature = ", ".join(_signature_gen(varnames, len(argdefs),
argcount))
variables = ", ".join(_signature_gen(varnames, len(argdefs),
argcount,
rm_defaults=True))
lambda_src = "lambda %s: call(func, %s)" % (signature, variables)
dec_func = eval(lambda_src, dict(func=func, call=caller,
arg=argdefs))
dec_func.__name__ = func.__name__
dec_func.__doc__ = func.__doc__
dec_func.__dict__ = func.__dict__.copy()
return dec_func

#</decorate.py>

 

[Thomas Heller]

I have realized today that defining decorators for functions
with generic signatures is pretty non-trivial.

I've not completely read your post ;-), but I assume you're trying to do
something that I've also done some time ago. Maybe the following code
snippet is useful for you - it creates a source code string which can
than be compiled.

The code prints this when run:

"""
def f(a, b=42, c='spam', d=None):
'docstring'
return f._api_(a, b, c, d)
def g(*args, **kw):
''
return g._api_(*args, **kw)
"""

Thomas

<code>
def make_codestring(func):
import inspect
args, varargs, varkw, defaults = inspect.getargspec(func)
return "def %s%s:\n %r\n return %s._api_%s" % \
(func.func_name,
inspect.formatargspec(args, varargs, varkw, defaults),
func.func_doc or "",
func.func_name,
inspect.formatargspec(args, varargs, varkw))

def f(a, b=42, c="spam", d=None):
"docstring"

def g(*args, **kw):
pass

print make_codestring(f)
print make_codestring(g)
</code>

 

[Michele Simionato]

Yes, this is essentially the same idea. You compile the codestring to
bytecode,
whereas I just evalue the codestring to a lambda function. We are
essentially implementing a runtime macro by hand. I wonder if there is
any alternative
approach to get the same result, without manipulation of the source
code.
BTW, I have fixed another small bug in my original code. 'decorator'
should read

import inspect

def decorate(func, caller):
args, varargs, varkw, defaults = inspect.getargspec(func)
argdefs = defaults or ()
argcount = func.func_code.co_argcount
varnames = args + (varargs or []) + (varkw or [])
signature = ", ".join(_signature_gen(varnames, len(argdefs),
argcount))
variables = ", ".join(_signature_gen(varnames, len(argdefs),
argcount,
rm_defaults=True))
lambda_src = "lambda %s: call(func, %s)" % (signature, variables)
print func.__name__, "->", lambda_src
dec_func = eval(lambda_src, dict(func=func, call=caller,
arg=argdefs))
dec_func.__name__ = func.__name__
dec_func.__doc__ = func.__doc__
dec_func.__dict__ = func.__dict__.copy()
return dec_func

 

[darkbeethoven]

Great post. Code still works in Python 2.6.

Whoever you are, your solution to fixing the pydoc function signature when using decorators worked perfectly.

I'm running an XML-RPC server, which supports pydoc in web format to provide a web service and having your solution made the resulting documentation so much better.

Thanks again =)