How do I do this? (eval() on the left hand side)

[Steven Bethard]

Note also that functions which use exec cannot use the static namespace
optimization, and thus tend to be *much* slower than normal functions

In what circumstances will this be true? I couldn't verify it:

> cat fib.py

def fib1(n):
a, b = 0, 1
while True:
a, b = b, a + b
yield a


exec """\
def fib2(n):
a, b = 0, 1
while True:
a, b = b, a + b
yield a
"""

> python -m timeit -s "import fib" "fib.fib1(100)"

1000000 loops, best of 3: 0.714 usec per loop

> python -m timeit -s "import fib" "fib.fib2(100)"

1000000 loops, best of 3: 0.705 usec per loop

 

[Jeff Shannon]

Note also that functions which use exec cannot use the static
namespace optimization, and thus tend to be *much* slower than normal
functions

In what circumstances will this be true? I couldn't verify it:

[...]
exec """\
def fib2(n):
a, b = 0, 1
while True:
a, b = b, a + b
yield a
"""

I was referring to functions which have an internal exec statement, not
functions which are created entirely within an exec -- i.e., something
like this:

def fib3(n):
a, b = 0, 1
while True:
exec "a, b = b, a + b"
yield a

In your fib2(), when the function is defined, the entire contents of the
local namespace can be determined (it's just that the function isn't
"defined" until the exec statement is executed). In fib3(), when the
function is defined, the parser can't determine what's happening inside
the exec statement (it just sees a string, and that string may depend on
other runtime circumstances which will happen latter), so it can't say
for certain whether names other than a and b are created. (Consider the
case where the string to be exec'ed is passed in as a function argument...)

Jeff Shannon
Technician/Programmer
Credit International

 

[Terry Reedy]

To respond to and summarize several posts in this discussion:

Within a function, where the local namespace is distinct from the global
(module) namespace, CPython usually implements the local namespace
internally as a fixed-length array. When this is true, locals() is a
*copy* of the local namespace and not the namespace itself. Once that dict
is created, the history of how it was created is immediately forgotten,
just as with any other ordinary Python dict.

That dict can be bound to a name or other target and modified like any
other dict, and there could be reasons to do so. However, modifying it has
no more effect on the local namespace than modifying any other local dict.

Terry J. Reedy

 

[Caleb Hattingh]

Peter, I second that.

Nick

In what way is it unreliable? I can't seem to create a situation where
the update through globals and locals doesn't work. Are you referring
perhaps to the possibility of variables being garbage collected and then
not being around later when one tries to access them through a string
name? I don't know very much about the garbage collector, so I can't say
for sure.

thx
Caleb

 

[Caleb Hattingh]

Steve,

I don't think I understand. Here is what I just tried:

'>>> def f():
x = 3
d = locals()
print x
print d['x']
d['x'] = 5
print x


'>>> f()
3
3
3
'>>>

In your example, x had not yet been initialised, maybe. What I am seeing
is that "x" does not seem to update when being assigned to - I guess this
is what you are referring to by being unreliable.

But "unreliable" sounds kinda vague and intermittent, and I assume that is
not the case here - What is the Rule(tm)?

Thanks
Caleb

Nick, could you please comment on why you say this about globals()?
I've never heard of any possibility of "unreliability" in updating
globals() and, as far as I know, a large body of code exists which
does in fact rely on this -- much of mine included. ;-)

Updating locals() is unreliable. Updating globals() is fine, AFAIK.

http://docs.python.org/lib/built-in-funcs.html

I believe that the only time that locals() is updatable is when locals()
is globals():
Traceback (most recent call last):

locals()['x'] = 3
x

3

def f():

... print locals() is globals()
... locals()['x'] = 3
... print x
...False
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "<interactive input>", line 4, in f
NameError: global name 'x' is not defined


Steve

 

[Caleb Hattingh]

Thx Peter

I verified this situation myself with a post from Steven Bethard earlier
(trying to use "locals" within a function definition).

I am convinced now that locals() doesn't work as (I) expected. Steven
says there was some or other reason why locals() as used in this context
is not writable - Do you know why this is? I really do not like
guidelines like "may not work", "is unreliable" and so on. Perhaps this
is a character flaw, but I really do like to know what works, when it
works, and when it doesn't work.

In this scenario, we can see it doesn't work. To my eyes, it doesn't work
*in the way I expect* (which is highly subjective, no argument there).
Would this be a situation where it would be nice to have an exception
thrown if locals() is assigned to in a scope where it is not writable?

It would also be nice if globals and locals behaved the same, differing
only in scope (which is what I expected originally anyway). But we can't
have everything, I guess

Caleb

In what way is it
unreliable?  I can't seem to create a situation where
the update through globals and locals doesn't
work.   Are you referring

Updates to a locals() dictionary may not be reflected by the variables in
the local scope, e. g.:
... locals()["a"] = 1
... print a
...Traceback (most recent call last):
File "<stdin>", line 1, in ?
... a = 42
... locals()["a"] = 1
... print a
...42

Updating globals() should be safe.
Peter

 

[Steven Bethard]

In what circumstances will this be true? I couldn't verify it:

[snip]

I was referring to functions which have an internal exec statement, not
functions which are created entirely within an exec -- i.e., something
like this:

Thanks for the clarification. Here's the results for some functions
with internal exec statements:

> cat fib.py

def fib1(n):
a, b = 0, 1
while True:
a, b = b, a + b
yield a


exec """\
def fib2(n):
a, b = 0, 1
while True:
a, b = b, a + b
yield a
"""

def fib3(n):
a, b = 0, 1
while True:
exec "a, b = b, a + b"
yield a

def fib4(n):
exec "a, b = 0, 1"
while True:
exec "a, b = b, a + b"
yield a

>
> python -m timeit -s "import fib" "fib.fib1(100)"

1000000 loops, best of 3: 0.71 usec per loop

> python -m timeit -s "import fib" "fib.fib2(100)"

1000000 loops, best of 3: 0.678 usec per loop

> python -m timeit -s "import fib" "fib.fib3(100)"

1000000 loops, best of 3: 0.826 usec per loop

> python -m timeit -s "import fib" "fib.fib4(100)"

1000000 loops, best of 3: 0.821 usec per loop

I'm not sure I'd say they're *much* slower, but you're right; they're
definitely slower.

Steve

 

[Peter Otten]

Those who've talked about it being "unreliable" are misstating
the Rule that you are looking for. I'll quote it below, so
that you aren't left in this unfortunate state of limbo:

The Rule of locals()
Updating locals() should not be done. Treat the
return value of locals() as read-only. Never try
to update it. End of story.

Anything that appears to suggest that locals() might sometimes
actually be writable is not really happening. Look the other
way. Pay no attention to the man behind the curtain. And
especially, whatever else you do, don't let the PSU se

I agree. But much of the confusion stems from interpreter experiments like

locals()["a"] = 42
a

42

That would go away if locals() returned an ignore-write proxy where the
global and local scope are identical. The ability to "just try it" is an
asset.

Peter

 

[Peter Otten]

I am convinced now that locals() doesn't work as (I) expected. Steven
says there was some or other reason why locals() as used in this context
is not writable - Do you know why this is? I really do not like
guidelines like "may not work", "is unreliable" and so on. Perhaps this
is a character flaw, but I really do like to know what works, when it
works, and when it doesn't work.

I think Peter Hansen has answered that. Your expectations were just wrong.

In this scenario, we can see it doesn't work. To my eyes, it doesn't work
*in the way I expect* (which is highly subjective, no argument there).
Would this be a situation where it would be nice to have an exception
thrown if locals() is assigned to in a scope where it is not writable?

If python were to throw an exception, it should always be thrown. But I'm
the wrong one to worry about that as I didn't even find a single

globals()[name] = value

assignment in my scripts. I want to know a variable's name, or I put it in a
dictionary.

It would also be nice if globals and locals behaved the same, differing
only in scope (which is what I expected originally anyway). But we can't
have everything, I guess

That would mean that both would become read-only, I guess, but I don't see
that happen.

Peter

 

[Nick Coghlan]

Nick, could you please comment on why you say this about globals()?
I've never heard of any possibility of "unreliability" in updating
globals() and, as far as I know, a large body of code exists which
does in fact rely on this -- much of mine included. ;-)

As Steve pointed out, I was, well, flat out wrong. You're quite correct - it's
only locals() that can cause a problem.

Cheers,
Nick.

 

[caleb.hattingh]

Both Peters

Sure, I must concede that the problem here was my expectation of how
things should work.

Thanks for the explanations. I still don't really know whether this
behaviour of locals() is the result of a design decision, or an
implementation artifact of CPython, but at least I have a clear idea of
how to play nice with locals().

thx
Caleb

I am convinced now that locals() doesn't work as (I) expected. Steven
says there was some or other reason why locals() as used in this context
is not writable - Do you know why this is? I really do not like
guidelines like "may not work", "is unreliable" and so on. Perhaps this
is a character flaw, but I really do like to know what works, when it
works, and when it doesn't work.

I think Peter Hansen has answered that. Your expectations were just wrong.

In this scenario, we can see it doesn't work. To my eyes, it doesn't work
*in the way I expect* (which is highly subjective, no argument there).
Would this be a situation where it would be nice to have an exception
thrown if locals() is assigned to in a scope where it is not

writable?

If python were to throw an exception, it should always be thrown. But I'm
the wrong one to worry about that as I didn't even find a single

globals()[name] = value

assignment in my scripts. I want to know a variable's name, or I put it in a
dictionary.

It would also be nice if globals and locals behaved the same, differing
only in scope (which is what I expected originally anyway). But we can't
have everything, I guess

That would mean that both would become read-only, I guess, but I don't see
that happen.

Peter

 

[Bengt Richter]

Obviously but that's not what I wish to do.


In REXX, for instance, one can do a:

interpret y' = 4'

Since y contains a, then the above statement amongs to:

a = 4

There are many situations where this is useful. For instance, you might be
getting an input which is a string representing the name of a variable and
you wish to evaluate the expression (like a calculator application, for
instance).

If you want to make a calculator, why not define a calculator class that
behaves the way you like, giving it methods for interaction according to any
syntax you like?

What would you like to be able to type for your calculator to interpret?
assignment statements and expressions? There is a difference between the
statements your calculator interprets and the statements you use to implement
your calculator, unless you are hoping just to pass input through for Python
to interpret (which will be risky if you don't control what's entered!).

But, to pursue your REXX example a bit, the question IMO is how important the
spelling is to you vs the functionality. Python lets you create custom objects
that behave pretty much any way you like. You can (ab)use the way python compiles
various operators operating on or with instances of your custom objects so you can
spell things in various ways, e.g., instead of
a = 3
y = "a"
print eval(y)

you could have a magic class instance o and write
o.a = 3
o.y = "a"
print o().y

and instead of

interpret y' = 4'

write
o().y = 4

Let's try it (untested beyond what you see here ;-)
... def __call__(self):
... return DerefName(self)
... ... def __init__(self, wrapped):
... object.__setattr__(self, 'wrapped', wrapped)
... def __getattr__(self, name):
... wrapped = object.__getattribute__(self, 'wrapped')
... return getattr(wrapped, getattr(wrapped, name))
... def __setattr__(self, name, value):
... wrapped = object.__getattribute__(self, 'wrapped')
... setattr(wrapped, getattr(wrapped, name), value)
... Traceback (most recent call last):
File "<stdin>", line 1, in ?
'y'

Anyway, getattr/setattr functionality provides the primal cauldron for python magic,
if you want to cook something up, and incantations usually involve prefixing the name of
a magic instance to your spell(ing)s ;-) Add descriptors for extra spice ...

Double, double, toil and trouble... oh, wait, that's for floating point ;-)

Regards,
Bengt Richter

 

[Jeff Shannon]

As Steve pointed out, I was, well, flat out wrong. You're quite
correct - it's only locals() that can cause a problem.

Of course, just because modifications of the dict returned by globals()
*do* reliably result in modifications to the global namespace, doesn't
mean it's a good idea.

Personally, I don't *want* to create variables by "magic" like that.
I'd rather pass a few extra parameters around, and explicitly create any
global names I need. If I really need to store objects under arbitrary
names, then I probably should be using a regular dict (and passing it
around as need be) instead of dumping stuff into globals(). (But then,
I'm a devoted follower of the Pythonic principle of 'explicit is better
than implicit' -- heck, even before I knew Python existed, I typically
used this->name to reference C++ members despite the fact that 'this'
was not required, just because I wanted to be able to *see* which
variables were members and which weren't...)

Jeff Shannon
Technician/Programmer
Credit International

 

[Peter Hansen]

Of course, just because modifications of the dict returned by globals()
*do* reliably result in modifications to the global namespace, doesn't
mean it's a good idea.

"The" global namespace misses the possibility that doing this in
"a" global namespace might be a good idea. For example, in the
namespace of a special module intended to be used as a convenient
way of accessing global information. See below.

Personally, I don't *want* to create variables by "magic" like that.
I'd rather pass a few extra parameters around, and explicitly create any
global names I need. If I really need to store objects under arbitrary
names, then I probably should be using a regular dict (and passing it
around as need be) instead of dumping stuff into globals().

The main way I use this is in some sort of a "const" module, which
provides access to a large set of constant information. In other
words, in contrast to what you had in mind when you wrote the
above, I'm dealing with neither "variables" nor information that
_would_ best be put in a dict.

The last time I did this I had a .h header file from C which I
wanted to wrap up for Python. Rather than convert once, statically,
I had a const.py module which loaded the .h file contents and
converted all lines which it recognized (mostly lines of the
form "#define XXX nnn") into Python names stuck into that module's
globals().

Other modules just do "import const" and then things like
"const.DAQmx_Val_Volts" to get access to any of the over 1700
defined values.

In general I would say that mucking with globals() like this is
probably best restricted to constants like in this case, if at all.

-Peter

 

[Bengt Richter]

To respond to and summarize several posts in this discussion:

Within a function, where the local namespace is distinct from the global
(module) namespace, CPython usually implements the local namespace
internally as a fixed-length array. When this is true, locals() is a
*copy* of the local namespace and not the namespace itself. Once that dict
is created, the history of how it was created is immediately forgotten,
just as with any other ordinary Python dict.

That dict can be bound to a name or other target and modified like any
other dict, and there could be reasons to do so. However, modifying it has
no more effect on the local namespace than modifying any other local dict.

It doesn't appear to be _quite_ ordinary though (note that print d['x'] inside
does print 3 the first time and 5 after the d['x']=5 assignment, but is not
returned in d when d returns in the results):

I think I'd rather locals() totally refuse updates than allow retrievable updates
in a way that leaves me wondering what kind of object it is and what happens to it
when it is exported from a function (some kind of closure stuff involved in converting
a proxy to a dict? (will speculation)) ;-/
... x = 3
... d = locals()
... D = dict(locals())
... print 'x', x
... print "d['x']",d['x']
... d['x'] = 5
... print 'x', x
... print "d['x']",d['x']
... print "D['x']", D['x']
... D['x'] = 7
... print "d['x']",d['x']
... print "D['x']", D['x']
... return d, D, locals()
... x 3
d['x'] 3
x 3
d['x'] 5
D['x'] 3
d['x'] 5
D['x'] 7

>>> d {'x': 3, 'd': {...}, 'D': {'x': 7, 'd': {...}}}
>>> D {'x': 7, 'd': {'x': 3, 'd': {...}, 'D': {...}}}
>>> L {'x': 3, 'd': {...}, 'D': {'x': 7, 'd': {...}}}
>>> L['d']['x'] # not 5 3
>>> L['D']['x'] # is 7 7
>>> d['x'] 3
>>> D['x'] 7
>>>

Regards,
Bengt Richter

 

[Jeff Shannon]

The main way I use this is in some sort of a "const" module, which
provides access to a large set of constant information. In other
words, in contrast to what you had in mind when you wrote the
above, I'm dealing with neither "variables" nor information that
_would_ best be put in a dict.

[...]

In general I would say that mucking with globals() like this is
probably best restricted to constants like in this case, if at all.

Indeed, for almost every rule, there are exceptions.. Legitimate cases
also exist for using eval() and exec, though in general both of those
are better avoided. The case you describe does sound like a legitimate
use of writing to globals(), with realistic expectations of the costs
and benefits

(Actually, in your provided example, I might have const.py parse the
header file into a dict named const, and then have my modules use 'from
const import const'. But I will admit that the extra complication on
import may not be worth the "cleaner" module internals... As always,
it's a tradeoff, and each programmer must decide which costs are worth
bearing in their particular situation.)

Jeff Shannon
Technician/Programmer
Credit International

 

[Carl Banks]

In general I would say that mucking with globals() like this is
probably best restricted to constants like in this case, if at all.

Modifying globals() not even necessary for this. When I want to
dynamically update the global namespace, I do it this way:

mod = __import__(__name__)
setattr(mod,symbol,value)

Works perfectly unless you're worried about someone modifying the built
in __import__.

 

[Mel Wilson]

Thanks for the clarification. Here's the results for some functions
with internal exec statements:

def fib1(n):
a, b = 0, 1
while True:
a, b = b, a + b
yield a


exec """\
def fib2(n):
a, b = 0, 1
while True:
a, b = b, a + b
yield a
"""

def fib3(n):
a, b = 0, 1
while True:
exec "a, b = b, a + b"
yield a

def fib4(n):
exec "a, b = 0, 1"
while True:
exec "a, b = b, a + b"
yield a

1000000 loops, best of 3: 0.71 usec per loop

1000000 loops, best of 3: 0.678 usec per loop

1000000 loops, best of 3: 0.826 usec per loop

1000000 loops, best of 3: 0.821 usec per loop

I'm not sure I'd say they're *much* slower, but you're right; they're
definitely slower.

The thing is, that once you drop local-namespace
optimization, the entire function gets slowed down, possibly
by 40%:



def fib5 (n):
a, b, i = 0, 1, n
while i > 0:
a, b = b, a+b
yield a
i -= 1


def fib6 (n):
exec "a, b, i = 0, 1, n"
while i > 0:
a, b = b, a+b
yield a
i -= 1



f:\home\mwilson\projects\python>python e:\bin\python23\lib\timeit.py -s "import
fib" "[i for i in fib.fib5(100)]"
1000 loops, best of 3: 1.95e+003 usec per loop

f:\home\mwilson\projects\python>python e:\bin\python23\lib\timeit.py -s "import
fib" "[i for i in fib.fib6(100)]"
100 loops, best of 3: 2.82e+003 usec per loop


Regards. Mel.

 

[Caleb Hattingh]

Jeff

I do the same thing in Delphi -> prepend "Self" before all the members in
class methods even though its not required. I do it partially for the
same reason as you - so I can grok which variables are local and which are
global (well, global within the class, anyway).

The other reason is because of the magical little menu that pops up when I
type the period after "Self", and lets me pick a class member by typing
the first few letters...

Keep well
Caleb


-- heck, even before I knew Python existed, I typically

 

[Peter Otten]

The thing is, that once you drop local-namespace
optimization, the entire function gets slowed down, possibly
by 40%:

It's not that bad as most of the extra time is spend on compiling the
string.

def fib5(n):
a, b, i = 0, 1, n
while i > 0:
a, b = b, a+b
yield a
i -= 1

def fib6(n):
exec "a, b, i = 0, 1, n"
while i > 0:
a, b = b, a+b
yield a
i -= 1

def fib7(n, c=compile("a, b, i = 0, 1, n", "<nofile>", "exec")):
exec c
while i > 0:
a, b = b, a+b
yield a
i -= 1

[Python 2.3]
$ timeit.py -s"from fib import fib5 as fib" "list(fib(100))"
10000 loops, best of 3: 143 usec per loop
$ timeit.py -s"from fib import fib6 as fib" "list(fib(100))"
10000 loops, best of 3: 208 usec per loop
$ timeit.py -s"from fib import fib7 as fib" "list(fib(100))"
10000 loops, best of 3: 151 usec per loop

Peter