Modify the local scope inside a function

[Sandra-24]

Is there a way in python to add the items of a dictionary to the local
function scope? i.e. var_foo = dict['var_foo']. I don't know how many
items are in this dictionary, or what they are until runtime.

exec statements are difficult for debuggers to deal with, so as a
workaround I built my code into a function and saved it in a .py file.
The I load the .py file as a module and call the function instead. This
works great, and it has the added advantage of precompiled versions of
the code being saved as .pyc and .pyo files. (faster repeated
execution)

The only trouble was I execed inside a specially created scope
dictionary containing various variables and functions that the code
requires. I can't seem to figure out how to get this same effect inside
the function. Right now I'm passing the dict as an argument to the
function, but I can't modify locals() so it doesn't help me.

Thanks,
-Sandra

 

[Crutcher]

Here you go. Unfortunate that you can't modify locals() easily, but
there are other options.

def foo(d):
for k in d:
exec '%s = %s' % (k, repr(d[k]))
print a + b

foo({'a':1, 'b':2})

 

[Jason Mobarak]

Is there a way in python to add the items of a dictionary to the local
function scope? i.e. var_foo = dict['var_foo']. I don't know how many
items are in this dictionary, or what they are until runtime.

Why do you want to do this? Exec and eval should -not- be used for
this unless you are specifically creating a system allows arbitrary
code execution. What's wrong with using a dictionary? It's much safer
than allowing arbitrary names to be injected into a namespace.

 

[Steven D'Aprano]

Is there a way in python to add the items of a dictionary to the local
function scope? i.e. var_foo = dict['var_foo']. I don't know how many
items are in this dictionary, or what they are until runtime.

Are you sure you need to do this? How do you use those variables later?

That is, I'm thinking that if you do this:

variables = {"var_foo"=0, "var_bar"=1, "var_baz"=3}
somehow_transfer_to_local_scope(variables)
print var_foo # or whatever...
print var_bar
print var_baz

it sort of looks pointless to me. In other words, if you have to deal with
the variables BY NAME in your code, then just define them by name. And if
you don't deal with them by name in your code, then they don't need names:

variables = {"var_foo"=0, "var_bar"=1, "var_baz"=3}
for item in variables.values():
print item


Okay, so they are *really* basic (and pointless) examples. But in most
cases that people say they want to turn strings into variables ("if I have
a string 'abc1', how do I turn it into a variable abc1?") it turns out
that doing so is not the best way of solving their problem.

exec statements are difficult for debuggers to deal with,

Which is one of the reasons why exec statements should be avoided whenever
possible. The rule of thumb I use is, any time I feel that I absolutely
must use exec, I drink tequila until the urge goes away.

*wink*

so as a
workaround I built my code into a function and saved it in a .py file.

That sounds like normal practice to me. Why aren't you doing that in the
first place?

The I load the .py file as a module and call the function instead. This
works great, and it has the added advantage of precompiled versions of
the code being saved as .pyc and .pyo files. (faster repeated
execution)

..pyc files don't give you faster execution on repeated calls. They give
you faster loading time on import. The benefit is on the first
import, not subsequent calls to the function. There may be other benefits
as well, but execution speed is not one of them.

Here is a test:

$ ls ftest*
ftest.py ftest.pyc
$ cat ftest.py
def f(): return sum(range(100))

(you'll just have to trust me that ftest.pyc is the compiled version of
ftest.py)

$ python /usr/lib/python2.3/timeit.py --setup="def f(): return sum(range(100))" "f()"
100000 loops, best of 3: 14 usec per loop

$ python /usr/lib/python2.3/timeit.py --setup="from ftest import f" "f()"
100000 loops, best of 3: 14.1 usec per loop

No appreciable difference in execution speed.

The only trouble was I execed inside a specially created scope
dictionary containing various variables and functions that the code
requires.

Ah, your exed'ed code is effectively using global variables.

I can't seem to figure out how to get this same effect inside
the function.

Define the names you need in the module.

var_foo = 0
var_bar = 1
var_baz = 3

def function(x):
return x + var_foo + var_bar + var_baz

Right now I'm passing the dict as an argument to the
function, but I can't modify locals() so it doesn't help me.

No, you can't modify locals. Locals returns a copy of the local
environment as a dict, but changing that dict doesn't change the local
variables.

There is an exception to that rule: if you run locals from the
interpreter, outside of a function, locals() is the same as globals():
True

This means that this will work from the interpreter:

x = 6
locals()['x'] = -1
x

-1

but not anywhere else. That's a (rare) Python gotcha.

 

[Sandra-24]

Hey Crutcher, thanks for the code, that would work. I'm now debating
using that, or using function arguments to get the variables into the
namespace. This would require knowing the variables in the dict ahead
of time, but I suppose I can do that because it's part of the same
system that creates the dict. I'm just not very fond of having code
relating to one thing in more than one place, because it puts the onus
on the programmer to remember to change it in both places. Here I might
forgive it because it would make the generated code more readable.

It seems I created a fair amount of confusion over what I'm trying to
do. I use special psp like templates in my website. The template engine
was previously execing the generated template code. It uses special
environment variables that give it access to the functionality of the
web engine. These are what are in that scope dictionary of mine, and
why I exec the code in that scope.

However, I want to integrate a debugger with the web engine now, and
debugging execed generated code is a nightmare. So I save the generated
code as a function in a module that is generated by the template
engine. Unless I'm missing something about what you're saying, this
should now be faster as well, because afaik execed code has to be
compiled on the spot, wheras a module when you load it, is compiled (or
loaded from a .pyc file) at import time. So one import and repeated
function calls would be cheaper than repeated exec.

Thanks,
-Sandra

 

[Fabio Zadrozny]

Hi Sandra,

Well, first, I'm not sure if you'd be interested, but Pydev Extensions
(http://www.fabioz.com/pydev) should be able to make remote debugging in
the way you want...Now, in order to do what you are trying to do,
debuggers (or at least the pydev debugger) go for the frame you want to
execute things (that contains the locals and globals in some scope).

In pydev extensions, in interactive debugging, the code to evaluate
expressions is something like:

frame = findFrame(thread_id, frame_id)
exec expression in frame.f_globals, frame.f_locals

So, you'd just need to get the frame... pydev does multithreaded
debugging, so, it needs to know the thread too, but if you just want to
debug the current thread, you could just go to curFrame =
sys._getframe() and then go iterating back in the frames to reach the
one you want at frame.f_back (that's basically what the findFrame
function does).

Cheers,

Fabio

Hey Crutcher, thanks for the code, that would work. I'm now debating
using that, or using function arguments to get the variables into the
namespace. This would require knowing the variables in the dict ahead
of time, but I suppose I can do that because it's part of the same
system that creates the dict. I'm just not very fond of having code
relating to one thing in more than one place, because it puts the onus
on the programmer to remember to change it in both places. Here I might
forgive it because it would make the generated code more readable.

It seems I created a fair amount of confusion over what I'm trying to
do. I use special psp like templates in my website. The template engine
was previously execing the generated template code. It uses special
environment variables that give it access to the functionality of the
web engine. These are what are in that scope dictionary of mine, and
why I exec the code in that scope.

However, I want to integrate a debugger with the web engine now, and
debugging execed generated code is a nightmare. So I save the generated
code as a function in a module that is generated by the template
engine. Unless I'm missing something about what you're saying, this
should now be faster as well, because afaik execed code has to be
compiled on the spot, wheras a module when you load it, is compiled (or
loaded from a .pyc file) at import time. So one import and repeated
function calls would be cheaper than repeated exec.

Thanks,
-Sandra

--
Fabio Zadrozny
------------------------------------------------------
Software Developer

ESSS - Engineering Simulation and Scientific Software
www.esss.com.br

Pydev Extensions
www.fabioz.com/pydev

PyDev - Python Development Enviroment for Eclipse
pydev.sf.net
pydev.blogspot.com