Passing function objects to timeit

[Steven D'Aprano]

The timeit.Timer class times "code snippets" -- you pass it strings
rather than function objects. That's good for what it's worth, but
sometimes the code you want to time is too big to easily pass as a
string, or maybe you only have access to a function object without the
source, or for whatever reason it's not very convenient.

In this case, a good trick is to import the function by name:

timeit.Timer('spam()', 'from __main__ import spam')


But now I find myself wanting to time a function that's not defined in
__main__. Here's a illustrative example:


def factory():
def f():
return "spam"
return f

def main():
func = factory()
return timeit.Timer('func()', 'from __main__ import func').timeit()


But it doesn't work:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in main
File "/usr/lib/python2.5/timeit.py", line 161, in timeit
timing = self.inner(it, self.timer)
File "<timeit-src>", line 3, in inner
ImportError: cannot import name func


Moving the definition of func into __main__ is not an option. What do I
do? Am I reduced to re-writing the timeit module to take functions
instead of strings? (Maybe I should do that anyway.) Or is there a way to
inject the function object into the namespace used by timeit?

 

[Gabriel Genellina]

En Sat, 29 Mar 2008 07:33:40 -0300, Steven D'Aprano

The timeit.Timer class times "code snippets" -- you pass it strings
rather than function objects. That's good for what it's worth, but
sometimes the code you want to time is too big to easily pass as a
string, or maybe you only have access to a function object without the
source, or for whatever reason it's not very convenient.

In this case, a good trick is to import the function by name:

timeit.Timer('spam()', 'from __main__ import spam')


But now I find myself wanting to time a function that's not defined in
__main__. Here's a illustrative example:


def factory():
def f():
return "spam"
return f

def main():
func = factory()
return timeit.Timer('func()', 'from __main__ import func').timeit()

Would this help (untested)?

def main():
return timeit.Timer('func()', 'from __main__ import factory; func =
factory()').timeit()

Or maybe:

def main():
global func
func = factory()
return timeit.Timer('func()', 'from __main__ import func').timeit()

 

[Steven D'Aprano]

En Sat, 29 Mar 2008 07:33:40 -0300, Steven D'Aprano


Would this help (untested)?

def main():
return timeit.Timer('func()', 'from __main__ import factory; func =
factory()').timeit()

Unfortunately no.

The above was just a simple illustration. Perhaps I misled you by showing
where func() came from, but what I intended to illustrate was that func()
could come from *anywhere*. I might not know where it came from: all I
have is a function object.

In fact, my question is actually more general than that, because my
example was a little unrealistic in that the function took no arguments.
I have to deal with the function arguments as well.

The general problem is that I wish to time an arbitrary function with
arbitrary arguments. The function and arguments are provided to me as
Python objects, but timeit requires strings. Converting the objects to
strings is not practical, and the objects might not exist in the __main__
module.

The problem is that timeit creates its own namespace to execute the code
in (which is a good thing!). Is there a way to insert arbitrary objects
into that namespace?

Or maybe:

def main():
global func
func = factory()
return timeit.Timer('func()', 'from __main__ import func').timeit()

I'll try playing around with that and see.

 

[Steven D'Aprano]

Or maybe:

def main():
global func
func = factory()
return timeit.Timer('func()', 'from __main__ import func').timeit()

Alas, this does not work, because all the Timer instances share the same
state.



import timeit, time

def test_timer(func1, func2):
global gFUNC
gFUNC = func1
T1 = timeit.Timer('gFUNC()', 'from __main__ import gFUNC')
gFUNC = func2
T2 = timeit.Timer('gFUNC()', 'from __main__ import gFUNC')
print "Calling %s" % func1.__name__
T1.repeat(3, 1)
print "Calling %s" % func2.__name__
T2.repeat(3, 1)

def functionA():
print "Function A"

def functionB():
print "Function B"



And here's the results:
Calling functionA
Function B
Function B
Function B
Calling functionB
Function B
Function B
Function B

 

[Gabriel Genellina]

En Sun, 30 Mar 2008 00:20:34 -0300, Steven D'Aprano

Alas, this does not work, because all the Timer instances share the same
state.

Second try:

<code>
# timeanyfunc.py
from timeit import Timer
from itertools import count

_counter = count()

def timeanyfunc(fn, *args, **kw):
def wrapper(fn=fn, args=args, kw=kw):
return fn(*args, **kw)
wrappername = 'wrapper%d' % _counter.next()
globals()[wrappername] = wrapper
return Timer("%s()" % wrappername, "from %s import %s" % (__name__,
wrappername))
</code>

Horrible, I know. Those wrapper1,wrapper2,wrapper3... keep growing with
each call. But it's the only way I could find, at least without changing
the code template used by timeit.

py> from timeanyfunc import timeanyfunc
py>
py> def functionA(arg):
.... print "Function A",arg
....
py> def functionB(arg):
.... print "Function B",arg
....
py> def test_timer(func1, func2):
.... T1 = timeanyfunc(func1, "arg1")
.... T2 = timeanyfunc(func2, "arg2")
.... print "Calling %s" % func1.__name__
.... T1.repeat(3, 1)
.... print "Calling %s" % func2.__name__
.... T2.repeat(3, 1)
....
py> test_timer(functionA, functionB)
Calling functionA
Function A arg1
Function A arg1
Function A arg1
Calling functionB
Function B arg2
Function B arg2
Function B arg2

 

[Gabriel Genellina]

En Sat, 29 Mar 2008 23:23:07 -0300, Steven D'Aprano

The general problem is that I wish to time an arbitrary function with
arbitrary arguments. The function and arguments are provided to me as
Python objects, but timeit requires strings. Converting the objects to
strings is not practical, and the objects might not exist in the __main__
module.

Ah, ok, I understand now. I think this is more-or-less what you want:

py> def test1(s):
.... return len(s)
....
py> def test2(s):
.... return s.__len__()
....
py> from timeanyfunc import timeanyfunc
py> timeanyfunc(test1, [1,2,3])
[1.3858088108963571, 1.3810702198184406, 1.3818543976957964]
py> timeanyfunc(test2, [1,2,3])
[1.6241321173501095, 1.6240804348038651, 1.6195021993018663]

<code>
# timeanyfunc.py

from timeit import Timer

def timeanyfunc(fn, *args, **kw):
global wrapped_fn

def wrapped_fn():
return fn(*args, **kw)

return Timer("wrapped_fn()",
"from %s import wrapped_fn" % __name__
).repeat()
</code>

 

[Steven D'Aprano]

En Sat, 29 Mar 2008 23:23:07 -0300, Steven D'Aprano


Ah, ok, I understand now. I think this is more-or-less what you want:

[snip]

No, sorry, it's still sharing state.


from timeit import Timer

# Slight modification to the function to return the Timer object.
def timeanyfunc(fn, *args, **kw):
global wrapped_fn
def wrapped_fn():
return fn(*args, **kw)
return Timer("wrapped_fn()", "from %s import wrapped_fn" % __name__)

def functionA():
print "Function A"

def functionB():
print "Function B"

T1 = timeanyfunc(functionA)
T2 = timeanyfunc(functionB)

T1.repeat(3, 1) # Should print "Function A".
T2.repeat(3, 1) # Should print "Function B".

 

[Steven D'Aprano]

Second try: ....
Horrible, I know. Those wrapper1,wrapper2,wrapper3... keep growing with
each call. But it's the only way I could find, at least without changing
the code template used by timeit.

Eeek. Talk about namespace pollution.

Thanks for the effort, but if that's the only solution, I think the
solution is worse than the problem!

Perhaps it's time for me to take a different approach. Since I can't call
timeit, and I can't inherit from it (same problem with state being shared
between instances), perhaps I should write my own timer.


import timeit
import gc
import itertools

def timeit2(func, args=[], kwargs={}, it=None,
timer=timeit.default_timer):
if it is None: it = itertools.repeat(None, timeit.default_number)
save_gc = gc.isenabled()
gc.disable()
try:
start = timer()
for counter in it:
func(*args, **kwargs)
end = timer()
finally:
if save_gc:
gc.enable()
return end - start



Now to go and test it.

 

[Steven D'Aprano]

Perhaps it's time for me to take a different approach. Since I can't
call timeit, and I can't inherit from it (same problem with state being
shared between instances), perhaps I should write my own timer.


import timeit
import gc
import itertools

def timeit2(func, args=[], kwargs={}, it=None,
timer=timeit.default_timer):
if it is None: it = itertools.repeat(None, timeit.default_number)
save_gc = gc.isenabled()
gc.disable()
try:
start = timer()
for counter in it:
func(*args, **kwargs)
end = timer()
finally:
if save_gc:
gc.enable()
return end - start

I've done some comparisons, and the times reported by this function are
consistently almost double that of times reported by timeit.

Now, I don't expect that my code and the timeit code should have the same
overhead, and I realise that the variability of timer results is large.
But I'm quite surprised that this seems to have so much more overhead.
Can anyone offer some advice?

 

[Peter Otten]

Eeek. Talk about namespace pollution.

Thanks for the effort, but if that's the only solution, I think the
solution is worse than the problem!

Perhaps it's time for me to take a different approach.

[snip]

Maybe the following enhancement of timeit would be worthwhile?

$ cat timeanyfunc.py

from mytimeit import Timer

def functionA():
print "Function A"

def functionB():
print "Function B"

T1 = Timer("f()", ns=dict(f=functionA))
T2 = Timer("f()", ns=dict(f=functionB))

T1.repeat(3, 1)
T2.repeat(3, 1)

$ python timeanyfunc.py
Function A
Function A
Function A
Function B
Function B
Function B

$ diff -u /usr/lib/python2.5/timeit.py mytimeit.py
--- /usr/lib/python2.5/timeit.py 2008-03-07 05:35:55.000000000 +0100
+++ mytimeit.py 2008-03-30 14:40:58.000000000 +0200
@@ -77,7 +77,7 @@
# in Timer.__init__() depend on setup being indented 4 spaces and stmt
# being indented 8 spaces.
template = """
-def inner(_it, _timer):
+def inner(_it, _timer, %(inject)s):
%(setup)s
_t0 = _timer()
for _i in _it:
@@ -106,15 +106,16 @@
multi-line string literals.
"""

- def __init__(self, stmt="pass", setup="pass", timer=default_timer):
+ def __init__(self, stmt="pass", setup="pass", timer=default_timer, ns=None):
"""Constructor. See class doc string."""
+ if ns is None:
+ ns = {}
self.timer = timer
stmt = reindent(stmt, 8)
setup = reindent(setup, 4)
- src = template % {'stmt': stmt, 'setup': setup}
+ src = template % {'stmt': stmt, 'setup': setup, 'inject': ','.join("%s=%s" % (s, s) for s in ns)}
self.src = src # Save for traceback display
code = compile(src, dummy_src_name, "exec")
- ns = {}
exec code in globals(), ns
self.inner = ns["inner"]

By the way, haven't we been there before, two years ago?

http://mail.python.org/pipermail/python-list/2006-February/368341.html

Peter

 

[George Sakkis]

Eeek. Talk about namespace pollution.

Thanks for the effort, but if that's the only solution, I think the
solution is worse than the problem!

Perhaps it's time for me to take a different approach.

[snip]

Maybe the following enhancement of timeit would be worthwhile?

[snip]


That would be great. I sometimes avoid timeit altogether because
setting up the environment is so cumbersome. Can you post the patch to
bugs.python.org so it doesn't get lost ?

George

 

[Steven D'Aprano]

Maybe the following enhancement of timeit would be worthwhile?

[snip]

Passing a namespace argument would be excellent.

By the way, haven't we been there before, two years ago?

http://mail.python.org/pipermail/python-list/2006-February/368341.html

Two years huh? Gosh, how time flies.

 

[Peter Otten]

Second try:
...
Horrible, I know. Those wrapper1,wrapper2,wrapper3... keep growing
with each call. But it's the only way I could find, at least without
changing the code template used by timeit.

Eeek. Talk about namespace pollution.

Thanks for the effort, but if that's the only solution, I think the
solution is worse than the problem!

Perhaps it's time for me to take a different approach.

[snip]

Maybe the following enhancement of timeit would be worthwhile?

[snip]


That would be great. I sometimes avoid timeit altogether because
setting up the environment is so cumbersome. Can you post the patch to
bugs.python.org so it doesn't get lost ?

Looking into

http://svn.python.org/view/python/trunk/Lib/timeit.py?rev=54953&view=markup

I discovered that the Python developers took a different approach and timeit
now allows callables for setup and statement:
42
[3.3855438232421875e-05]

So my patch is probably a case of instant obsolescence...

Peter

 

[George Sakkis]

Steven D'Aprano wrote:
...
Horrible, I know. Those wrapper1,wrapper2,wrapper3... keep growing
with each call. But it's the only way I could find, at least without
changing the code template used by timeit.
Eeek. Talk about namespace pollution.
Thanks for the effort, but if that's the only solution, I think the
solution is worse than the problem!
Perhaps it's time for me to take a different approach.
[snip]
Maybe the following enhancement of timeit would be worthwhile?

[snip]

That would be great. I sometimes avoid timeit altogether because
setting up the environment is so cumbersome. Can you post the patch to
bugs.python.org so it doesn't get lost ?

Looking into

http://svn.python.org/view/python/trunk/Lib/timeit.py?rev=54953&view=...

I discovered that the Python developers took a different approach and timeit
now allows callables for setup and statement:

42
[3.3855438232421875e-05]

So my patch is probably a case of instant obsolescence...

Peter

I'm afraid that the taken approach is worse than your patch. For one
thing, it allows only zero-arg functions. Of course one can work
around it by passing "lambda: f(...)" but that's adding extra overhead
which can be measurable for small fast functions. Even if passing
*args and **kwds to a Timer is allowed, that's still going to be
slower (because of tuple unpacking and whatnot) as Steven's attempt
above showed.

I think it's at least worth bringing this to the attention of the
developers.

George

 

[Peter Otten]

I'm afraid that the taken approach is worse than your patch. For one
thing, it allows only zero-arg functions. Of course one can work
around it by passing "lambda: f(...)" but that's adding extra overhead
which can be measurable for small fast functions. Even if passing
*args and **kwds to a Timer is allowed, that's still going to be
slower (because of tuple unpacking and whatnot) as Steven's attempt
above showed.

I think it's at least worth bringing this to the attention of the
developers.

I decided to give it a try:

http://bugs.python.org/issue2527

Peter