changing local namespace of a function

[Nick Coghlan]

Thank again for everyone's help. I have learned a lot from the posts,
especially the wrapdict class.

Hmmm, you do realize that wrapdict uses a lot of indirection while my
equivalent approach, just posted, is very direct, right? To reiterate
the latter, and dress it up nicely too, it's

class wrapwell(object):
def __init__(self, somedict):
self.__dict__ = somedict

and use d=wrapwell(whateverdict) to make the wrapper. Now, reading or
writing d.x is just like reading or writing whateverdict['x'], etc etc.

I was wondering if that would work, but never got around to trying it. It can be
used with the property-based auto calculation approach, too (the calculated
properties just store the results in self.__dict__ instead of self._data):

class DataManipulator(object):
def __init__(self, data):
self.__dict__ = data

class _utils(object):
@staticmethod
def make_prop(name, calculator):
def get(self, _name=name, _calculator=calculator):
try:
return self.__dict__[_name]
except KeyError:
val = _calculator(self)
self.__dict__[_name] = val
return val
def set(self, val, _name=name):
self.__dict__[_name] = val
def delete(self, _name=name):
del self.__dict__[_name]
return property(get, set, delete)

@staticmethod
def calc_z(self):
return self.x + self.y

z = _utils.make_prop('z', _utils.calc_z)

Py> d = DataManipulator(dict(x=1, y=2))
Py> d.x
1
Py> d.y
2
Py> d.z
3
Py> d.x = 10
Py> d.z
3
Py> del d.z
Py> d.z
12

Cheers,
Nick.

 

[Bo Peng]

If you want to add more calculated properties to the data manipulator,
simply define additional calculator methods, and define the attribute
with make_prop.

This has became really appealing....

You know, I have a deep root in C/C++ so performance is the king and
hacking is part of my daily life. Time to change now.

My original problem was that I need to manipulate and create dictionary
items in situ and since there are so many "d['key']"s, I would like to
make my code easier to read (and save some key strokes) by using "d.key"
or simply "key", without sacrificing performance. OOP seemed to be
irrelevant.

But this on-demand calculation (quote: make first call to z automatic)
is *really* good! I can calculate self._data.stat_a = self._data.stat_b
+ 1 without worrying about the status of stat_b and users can access any
value in the dictionary, original or calculated, in a uniform and
easier way! This makes my program instantly easier to use.

I can not say enough thank you for this.

Bo

 

[Kent Johnson]

Dear list,

I have many dictionaries with the same set of keys and I would like to
write a function to calculate something based on these values. For
example, I have

a = {'x':1, 'y':2}
b = {'x':3, 'y':3}

def fun(dict):
dict['z'] = dict['x'] + dict['y']

fun(a) and fun(b) will set z in each dictionary as the sum of x and y.

My function and dictionaries are a lot more complicated than these so I
would like to set dict as the default namespace of fun. Is this
possible? The ideal code would be:

def fun(dict):
# set dict as local namespace
# locals() = dict?
z = x + y

You can part way there using keyword arguments. You just have to use dictionary syntax for changing
values in the dictionary:
... d['z'] = x + y
...{'y': 3, 'x': 3, 'z': 6}

Kent

 

[Kent Johnson]

Yes. I thought of using exec or eval. If there are a dozen statements,

def fun(d):
exec 'z = x + y' in globals(), d

seems to be more readable than

def fun(d):
d['z'] = d['x'] + d['y']

But how severe will the performance penalty be?

You can precompile the string using compile(), you only have to do this once.
... code = compile(funcStr, name, 'exec')
... def f(d):
... exec code in d
... del d['__builtins__'] # clean up extra entry in d
... return f
...{'y': 3, 'x': 3, 'z': 6}

 

[Nick Coghlan]

I can not say enough thank you for this.

Don't thank me, thank Guido. He created the property machinery - I just let you
know it was there

But yes, Python's OO is OO the way it should be - something that helps you get
the job done quickly and cleanly, rather than making you jump through irrelevant
hoops.

Cheers,
Nick.

 

[Nick Coghlan]

Something I forgot to mention. . .

You know, I have a deep root in C/C++ so performance is the king and
hacking is part of my daily life. Time to change now.

The entire design of C++ is in many ways a regrettable monument to the idea that
premature optimisation is evil - far too many of the language's default
behaviours are 'fast in particular cases, but quite simply wrong in most cases,
and the compiler often can't tell you which case applies'. So you can write
buggy warning free code just by failing to override the unsafe default behaviours.

That said, one of the reasons I like CPython is that it lets you drop into C/C++
really easily when you need to (generally for hardware interface or performance
reasons). The builtin profiler can also give some decent hints on the
bottlenecks where you need to focus your performance improvements.

Cheers,
Nick.

 

[Michael Spencer]

Bad mistake on my part, sorry!

... a class that combined property access with the above...

In a similar vein to Nick's solution:

class AutoProperty(object):
def __init__(self, meth):
self.meth = meth
self.name = meth.__name__
self.__doc__ = meth.__doc__
def __get__(self, obj, cls):
if isinstance(obj, cls):
return obj.__dict__.setdefault(self.name, self.meth(obj))
else:
return self.__doc__
# You could define __set__ and __del__ but they don't seem
# necessary based on what you've said so far


class DataManipulator(object):
def __init__(self, data):
self.__dict__ = data

class Model(DataManipulator):
def z(self):
"""x+y"""
return self.x+self.y
z = AutoProperty(z)

def z1(self):
"""Or any other useful information"""
return self.z + self.x
z1 = AutoProperty(z1)

# You could automate these calls to AutoProperty in a metaclass

Michael

 

[Bo Peng]

You can part way there using keyword arguments. You just have to use
dictionary syntax for changing values in the dictionary:
... d['z'] = x + y
...{'y': 3, 'x': 3, 'z': 6}

This is not possible in my case since my dictionary have many more items
than just x and y. So, if there is are other items in the dictionary
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/tmp/python-10176RtT.py", line 1, in ?
f(a,**a)
TypeError: f() got an unexpected keyword argument 'else'

Bo

 

[Bo Peng]

Exec is slow since compiling the string and calls to globals() use a lot

of time. The last one is most elegant but __getattr__ and __setattr__
are costly. The 'evil hack' solution is good since accessing x and y
takes no additional time.

Previous comparison was not completely fair since I could pre-compile
fun2 and I used indirect __setattr__. Here is the new one:

.... def fun1(d):
.... for i in xrange(0,N):
.... d['z'] = d['x'] + d['y']
........ def makeFunction(funcStr, name):
.... code = compile(funcStr, name, 'exec')
.... def f(d):
.... exec code in d
.... return f
........ myfun = makeFunction('z = x + y', 'myfun')
.... for i in xrange(0,N):
.... myfun(d)
.... del d['__builtins__']
....
.... # solution three: update local dictionary.... # z = x + y
.... # does not set z in d
.... def fun3(d):
.... exec "locals().update(d)"
.... for i in xrange(0,N):
.... d['z'] = x + y
........ # this makes code easier to write and read
.... class wrapdict(object):
.... """Lazy attribute access to dictionary keys. Will not access
.... keys that are not valid attribute names!"""
.... def __init__(self, mydict):
.... self.__dict__ = mydict
....
.... # use wrapper.... wd = wrapdict(d)
.... for i in xrange(0,N):
.... wd.z = wd.x + wd.y
.... 3 function calls in 0.060 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.060 0.060 <string>:1(?)
1 0.000 0.000 0.060 0.060 profile:0(fun1(a))
0 0.000 0.000 profile:0(profiler)
1 0.060 0.060 0.060 0.060 python-10176FWs.py:2(fun1)

200004 function calls in 2.130 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 2.130 2.130 <string>:1(?)
100000 0.520 0.000 0.520 0.000 myfun:1(?)
1 0.000 0.000 2.130 2.130 profile:0(fun2(a))
0 0.000 0.000 profile:0(profiler)
1 0.590 0.590 2.130 2.130 python-10176EqB.py:1(fun2)
1 0.000 0.000 0.000 0.000
python-10176Sgy.py:2(makeFunction)
100000 1.020 0.000 1.540 0.000 python-10176Sgy.py:4(f)

4 function calls (3 primitive calls) in 0.070 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
2/1 0.000 0.000 0.070 0.070 <string>:1(?)
1 0.000 0.000 0.070 0.070 profile:0(fun3(a))
0 0.000 0.000 profile:0(profiler)
1 0.070 0.070 0.070 0.070 python-10176R0H.py:4(fun3)

4 function calls in 0.100 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.100 0.100 <string>:1(?)
1 0.000 0.000 0.100 0.100 profile:0(fun4(a))
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 0.000 0.000
python-10176e-N.py:6(__init__)
1 0.100 0.100 0.100 0.100 python-10176rIU.py:1(fun4)


Since

d['x'] is fast but cumbersome
exec "z=x+y' is still slow.
exec "locals().update(d)" is evil
d.x is elegant and only a little slower than d['x']

I am announcing the winner of the contest: dictwrap! (applause)

Bo

 

[Kent Johnson]

Exec is slow since compiling the string and calls to globals() use a
lot of time. The last one is most elegant but __getattr__ and
__setattr__ are costly. The 'evil hack' solution is good since
accessing x and y takes no additional time.

Previous comparison was not completely fair since I could pre-compile
fun2 and I used indirect __setattr__. Here is the new one:... def makeFunction(funcStr, name):
... code = compile(funcStr, name, 'exec')
... def f(d):
... exec code in d
... return f
...... myfun = makeFunction('z = x + y', 'myfun')
... for i in xrange(0,N):
... myfun(d)
... del d['__builtins__']

You are still including the compile overhead in fun2. If you want to see how fast the compiled code
is you should take the definition of myfun out of fun2:

myfun = makeFunction('z = x + y', 'myfun')
def fun2(d):
for i in xrange(0,N):
myfun(d)
del d['__builtins__']

Kent

 

[Bo Peng]

Exec is slow since compiling the string and calls to globals() use a
lot of time. The last one is most elegant but __getattr__ and
__setattr__ are costly. The 'evil hack' solution is good since
accessing x and y takes no additional time.

Previous comparison was not completely fair since I could pre-compile
fun2 and I used indirect __setattr__. Here is the new one:

# solution two: use exec

... def makeFunction(funcStr, name):
... code = compile(funcStr, name, 'exec')
... def f(d):
... exec code in d
... return f
...

def fun2(d):

... myfun = makeFunction('z = x + y', 'myfun')
... for i in xrange(0,N):
... myfun(d)
... del d['__builtins__']

You are still including the compile overhead in fun2. If you want to see
how fast the compiled code is you should take the definition of myfun
out of fun2:

myfun = makeFunction('z = x + y', 'myfun')
def fun2(d):
for i in xrange(0,N):
myfun(d)
del d['__builtins__']

Kent

I assumed that most of the time will be spent on N times execution of
myfunc.
.... # myfun = makeFunction('z = x + y', 'myfun')
.... for i in xrange(0,N):
.... myfun(d)
.... del d['__builtins__'].... myfun = makeFunction('z = x + y', 'myfun')
.... for i in xrange(0,N):
.... myfun(d)
.... del d['__builtins__']
200003 function calls in 1.930 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.930 1.930 <string>:1(?)
100000 0.460 0.000 0.460 0.000 myfun:1(?)
1 0.000 0.000 1.930 1.930 profile:0(fun2(a))
0 0.000 0.000 profile:0(profiler)
100000 0.980 0.000 1.440 0.000 python-162616Io.py:4(f)
1 0.490 0.490 1.930 1.930 python-16261Ud0.py:1(fun2)
200004 function calls in 1.920 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.920 1.920 <string>:1(?)
100000 0.390 0.000 0.390 0.000 myfun:1(?)
1 0.000 0.000 1.920 1.920 profile:0(fun21(a))
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 0.000 0.000
python-162616Io.py:2(makeFunction)
100000 0.930 0.000 1.320 0.000 python-162616Io.py:4(f)
1 0.600 0.600 1.920 1.920 python-16261huu.py:1(fun21)

 

[Kent Johnson]

I assumed that most of the time will be spent on N times execution of
myfunc.

Doh! Right.

Kent