Dynamic variable creation from string

[Massi]

Hi everyone,

in my script I have a dictionary whose items are couples in the form
(string, integer values), say

D = {'a':1, 'b':2, 'c':3}

This dictionary is passed to a function as a parameter, e.g. :

def Sum(D) :
return D['a']+D['b']+D['c']

Is there a way to create three variables dynamically inside Sum in
order to re write the function like this?

def Sum(D) :
# Here some magic to create a,b,c from D
return a+b+c

It is really important that the scope of a,b,c is limited to the Sum
function, they must not exisit outside it or inside any other nested
functions.
Thanks in advance for your help!

 

[John Gordon]

in my script I have a dictionary whose items are couples in the form
(string, integer values), say

D = {'a':1, 'b':2, 'c':3}

This dictionary is passed to a function as a parameter, e.g. :

def Sum(D) :
return D['a']+D['b']+D['c']

Is there a way to create three variables dynamically inside Sum in
order to re write the function like this?

Do you want to sum all the values in D? If so, that's easy:

def Sum(D):
my_sum = 0
for item in D:
my_sum += D[item]
return my_sum

 

[Waldek M.]

def Sum(D) :
return D['a']+D['b']+D['c']

Is there a way to create three variables dynamically inside Sum in
order to re write the function like this?

def Sum(D) :
# Here some magic to create a,b,c from D
return a+b+c
Hello,

It is really important that the scope of a,b,c is limited to the Sum
function, they must not exisit outside it or inside any other nested
functions.
Thanks in advance for your help!

Can you clarify a bit? I'm not sure why do you need to define any
additional variables at all. You do not return variables
from a function - you can return *a value* which may be
a simple type or complex type.

Or maybe you mean something like creating a global name
from inside of a function? Well, that doesn't sound like a good idea,
though...
global a
a = 5



Best regards,
Waldek

 

[Chris Angelico]

def Sum(D) :
# Here some magic to create a,b,c from D
return a+b+c

Welcome to TMTOWTDI land! We do magic here... several different ways.

You _may_ be able to do this, which is roughly equivalent to the
extract() function in PHP:

locals().update(D)

However, this is NOT guaranteed to work. It's more likely to work with
globals(), but that wouldn't restrict the scope the way you asked.

One handy trick that I found on the internet while researching this:
Use (or abuse!) function keyword parameters to do the dictionary
extraction. You have to explicitly name your desired keys this way,
but it may be suitable. (The function doesn't have to be defined
inside the other, incidentally.)
def inner_sum(a,b,c,**kwargs):
# Real logic goes here.
return a+b+c
return inner_sum(**D)
35

Alternatively, the exec() and eval() functions can be given
dictionaries which will serve as their variable scopes, so you could
possibly use that. Definitely looking like ugly code though.

For something as trivial as Sum(), you'd do best to simply type
D['a']+D['b']+D['c'] and be done with it. For something where you're
going to use them a lot, probably easiest to be explicit:

a,b,c = D['a'],D['b'],D['c']

However, this violates DRY principle, and risks hard-to-trace mismatch
bugs. I'd be inclined to simply be explicit all the time.

Depending on what D is, you may actually want to consider rewriting
this as a class with a member function.

class whatever:
def Sum(self):
return self.a+self.b+self.c

There should be one obvious way to do it, says the zen of Python.
Frankly, I'm not sure what that one obvious way is, here, although I'm
pretty certain that several of the options posited would be very bad
for your code!

Still, down's very nice... They ARE alternative possibilities.

ChrisA

 

[MRAB]

in my script I have a dictionary whose items are couples in the form
(string, integer values), say

D = {'a':1, 'b':2, 'c':3}

This dictionary is passed to a function as a parameter, e.g. :

def Sum(D) :
return D['a']+D['b']+D['c']

Is there a way to create three variables dynamically inside Sum in
order to re write the function like this?

Do you want to sum all the values in D? If so, that's easy:

def Sum(D):
my_sum = 0
for item in D:
my_sum += D[item]
return my_sum

Or even:

def Sum(D):
return sum(D.values())

 

[Terry Reedy]

in my script I have a dictionary whose items are couples in the form
(string, integer values), say

D = {'a':1, 'b':2, 'c':3}

This dictionary is passed to a function as a parameter, e.g. :

def Sum(D) :
return D['a']+D['b']+D['c']

Is there a way to create three variables dynamically inside Sum in
order to re write the function like this?

def Sum(D) :
# Here some magic to create a,b,c from D
return a+b+c

No. The set of local names for a function is determined when the
definition is executed and the body is compiled. Python 2 had an
exception -- 'from x import *' -- that required an alternate
complilation pathway. That possibility was eliminated in Python 3 and is
now a syntax error.

Within functions, 'locals().update(D)' is more or less guaranteed to
*not* add local names to the local namespace, even if it does add keys
to the locals() dict that shadows the local namespace.

It is really important that the scope of a,b,c is limited to the Sum
function, they must not exisit outside it or inside any other nested
functions.

Local names, by definition, are lexically scoped to the function
definition and are not visible without. Since nested definitions are
part of that lexical scope, local names are always visible within nested
definitions. You cannot stop that. The association of local names is
usually dynamically limited to one function call. The two exceptions are
enclosure by a nested function that survives the function call and
generators in a paused state.

 

[Steven D'Aprano]

Is there a way to create three variables dynamically inside Sum in order
to re write the function like this?

def Sum(D) :
# Here some magic to create a,b,c from D
return a+b+c

No magic is needed.

a, b, c = D['a'], D['b'], D['c']

However, there is no way to create variables from an *arbitrary* set of
keys. And a good thing too, because how could you use them?

def Sum(D):
# Magic happens here
return a + b + z + foo + spam + fred + ... ???

Since you don't know what variable names will be created, you don't know
which variables you need to add. Dynamic creation of variables is ALMOST
ALWAYS the wrong approach.

In this specific case, what you should do to add up an arbitrary number
of values from a dict is:

sum(D.values())

It is really important that the scope of a,b,c is limited to the Sum
function, they must not exisit outside it or inside any other nested
functions.

The first part is trivially easy; since you are assigning to local
variables, by definition they will be local to the Sum function and will
not exist outside it.

The second part is impossible, because that is not how Python works.
Nested functions in Python can always see variables in their enclosing
scope. If you don't want that behaviour, use another language, or don't
use nested functions.

 

[Terry Reedy]

I should have mentioned in my earlier response that 'variable' is a bit
vague and misleading. Python has names bound to objects.

def Sum(D) :
# Here some magic to create a,b,c from D
return a+b+c

No magic is needed.

a, b, c = D['a'], D['b'], D['c']

This is not what most people mean by 'dynamically created variables'.
The names are static, in the code, before the code is executed.
In 2.x, 'from x import *' dynamically creates local names that are not
in the code that contains the import. Dynamically creating objects is
what Python code does all the time.

 

[Steven D'Aprano]

I should have mentioned in my earlier response that 'variable' is a bit
vague and misleading. Python has names bound to objects.

def Sum(D) :
# Here some magic to create a,b,c from D return a+b+c

No magic is needed.

a, b, c = D['a'], D['b'], D['c']

This is not what most people mean by 'dynamically created variables'.

I know that. I'm just pointing out that the OP can solve his *stated*
problem of creating a *fixed* number of variables with *known* names
without any magic.

I went on to discuss the case of an unknown number of unknown names, and
suggested that the OP not do that, because it is much less useful than
people think.

The names are static, in the code, before the code is executed. In 2.x,
'from x import *' dynamically creates local names that are not in the
code that contains the import. Dynamically creating objects is what
Python code does all the time.

Of course. And even dynamically creating names: names don't exist until
they are created at runtime. Name deletion is also possible. But what
isn't normally done is "dynamic variable creation" in the sense of
creating arbitrary variables (names bound to objects) based on names
known only at runtime. That's extremely rare, and for good reason.

 

[Chris Angelico]

The second part is impossible, because that is not how Python works.
Nested functions in Python can always see variables in their enclosing
scope. If you don't want that behaviour, use another language, or don't
use nested functions.

To the OP: By "nested functions", did you mean actual nested functions
(those defined inside this function), or simply functions called from
this one?

This is a nested function, as the term is usually taken to mean:

def outer_function():
a = 1
def inner_function():
b = 2
return a+b
print(inner_function()) # Prints 3

The inner function has access to all of the outer function's
namespace. But if you meant this:

def function_1():
b = 2
return a+b

def function_2():
a = 1
print(function_1())

then it's quite the other way around - Python never shares variables
in this way (this would have function_1 assume that 'a' is a global
name, so if it's not, you'll get a run-time NameError).

As Steven says, there's no way in Python to hide variables from an
actual nested function. But if you just mean a function called from
this one, then what you want is the normal behaviour (and if you
actually want to share variables, you pass them as arguments).

ChrisA

 

[Jussi Piitulainen]

I should have mentioned in my earlier response that 'variable' is a
bit vague and misleading. Python has names bound to objects.

The language reference at python.org uses both terms - name and
variable - freely. Here is one instance.

<http://docs.python.org/py3k/reference/executionmodel.html#naming-and-binding>

# If a name is bound in a block, it is a local variable of that block,
# unless declared as nonlocal. If a name is bound at the module level,
# it is a global variable. (The variables of the module code block are
# local and global.) If a variable is used in a code block but not
# defined there, it is a free variable.

Perhaps you could consider accepting this. Or trying to change it. Or
starting to tell newcomers that the official documentation of the
language is, say, vague and misleading. (It seems good to me.)

 

[Massi]

Thank you all for your replies, first of all my Sum function was an
example simplifying what I have to do in my real funciton. In general
the D dictionary is complex, with a lot of keys, so I was searching
for a quick method to access all the variables in it without doing the
explicit creation:

a, b, c = D['a'], D['b'], D['c']

and without using directly the D dictionary (boring...).
When I talked about nested function I meant both cases Chris, but this
is not a really tighten bound.
I tried to follow the hints of Chris together with some help by google
and used the following code:

for k in D : exec "%s = D[k]" %k

That seems to do the trick, but someone speaks about "dirty code", can
anyone point me out which problems this can generate?
Again, thank you for your help!

 

[Peter Otten]

for k in D : exec "%s = D[k]" %k

That seems to do the trick, but someone speaks about "dirty code", can
anyone point me out which problems this can generate?

exec can run arbitrary code, so everybody reading the above has to go back
to the definition of D to verify that it can only contain "safe" keys.
Filling D with user-input is right out because a malicious user could do
anything he likes. Here's a harmless demo that creates a file:

d = {"x = 42\nwith open('tmp.txt', 'w') as f:\n f.write('whatever')\nx": 123}
for k in d: exec "%s = d[k]" % k ....
x 123
open("tmp.txt").read()

'whatever'

 

[Steven D'Aprano]

for k in D : exec "%s = D[k]" %k

That seems to do the trick, but someone speaks about "dirty code", can
anyone point me out which problems this can generate? Again, thank you
for your help!

Just the second-most common source of viruses, malware and security
vulnerabilities (behind buffer overflows): code injection attacks.

Code injection attacks make up at least three of the top 25 security
vulnerabilities on the CWE/SANS list:

http://cwe.mitre.org/top25/index.html

including the top 2 most dangerous threats (beating even our old friend,
the buffer overflow): SQL injection and OS command injection. Your use of
exec is vulnerable to attack if a hostile user can fool you into using a
dict like this one:

D = {'a': '42',
'import os;'\
' os.system("""echo "ha ha i ownz ur system rm-rf/" """); b': '23',
}
for k in D : exec "%s = D[k]" % k


You might think you're safe from such attacks, but (1) it is MUCH harder
to protect against them than you might think; and (2) code has a habit of
being re-used. Today your application might only be used by you; next
week your code might find itself embedded in a web-application where
hostile script kiddies can destroy your server with a single upload.

My advice is:

(1) If you need to ask why exec is dangerous, you shouldn't touch it.
(2) If you're sure you can protect against code injection, you can't.
(3) If you think you need exec, you probably don't.
(4) If you think you can make exec safe with a prohibited list of
dangerous strings, you probably can't.

 

[Steven D'Aprano]

Just the second-most common source of viruses, malware and security
vulnerabilities (behind buffer overflows): code injection attacks.

Oops, I forgot to go back and revise this sentence. Code injection
attacks are now the most common, not second-most common, source of
security vulnerabilities.

http://cwe.mitre.org/top25/index.html

 

[Chris Angelico]

(4) If you think you can make exec safe with a prohibited list of
dangerous strings, you probably can't.

If you think that it's even _possible_ to make exec safe with a
blacklist, I have a nice padded cell for you over here.

Security is NEVER achieved with blacklists, ONLY whitelists.

ChrisA

 

[Ethan Furman]

Thank you all for your replies, first of all my Sum function was an
example simplifying what I have to do in my real funciton. In general
the D dictionary is complex, with a lot of keys, so I was searching
for a quick method to access all the variables in it without doing the
explicit creation:

a, b, c = D['a'], D['b'], D['c']

and without using directly the D dictionary (boring...).
When I talked about nested function I meant both cases Chris, but this
is not a really tighten bound.
I tried to follow the hints of Chris together with some help by google
and used the following code:

for k in D : exec "%s = D[k]" %k

That seems to do the trick, but someone speaks about "dirty code", can
anyone point me out which problems this can generate?
Again, thank you for your help!

Besides the serious security issues, this method won't make the problem
any better in Python 3.

~Ethan~

 

[Nobody]

Thank you all for your replies, first of all my Sum function was an
example simplifying what I have to do in my real funciton. In general
the D dictionary is complex, with a lot of keys, so I was searching
for a quick method to access all the variables in it without doing the
explicit creation:

a, b, c = D['a'], D['b'], D['c']

and without using directly the D dictionary (boring...).

If just you're trying to avoid getting a repetitive strain injury in your
right-hand little finger from typing all the [''], you could turn
the keys into object attributes, e.g.:

class DictObject:
def __init__(self, d):
for key, value in d.iteritems():
setattr(self, key, value)
...
o = DictObject(D)
# use o.a, o.b, etc

 

[alex23]

If just you're trying to avoid getting a repetitive strain injury in your
right-hand little finger from typing all the [''], you could turn
the keys into object attributes, e.g.:

        class DictObject:
            def __init__(self, d):
                for key, value in d.iteritems():
                    setattr(self, key, value)
        ...
        o = DictObject(D)
        # use o.a, o.b, etc

I hate this kind of laziness. I'd spend at least 5 minutes trying to
work out _why_ someone felt this was necessary and then get annoyed
that it was just to avoid typing.

 

[Ethan Furman]

If just you're trying to avoid getting a repetitive strain injury in your
right-hand little finger from typing all the [''], you could turn
the keys into object attributes, e.g.:

class DictObject:
def __init__(self, d):
for key, value in d.iteritems():
setattr(self, key, value)
...
o = DictObject(D)
# use o.a, o.b, etc

I hate this kind of laziness. I'd spend at least 5 minutes trying to
work out _why_ someone felt this was necessary and then get annoyed
that it was just to avoid typing.

For me, half of it is to avoid the typing, the other half to avoid the
reading.

~Ethan~