lists of variables

[Michael Pardee]

I'm relatively new to python and I was very surprised by the following behavior:

a=1
b=2
mylist=[a,b]
print mylist [1, 2]
a=3
print mylist

[1, 2]

Whoah! Are python lists only for literals? Nope:

c={}
d={}
mydlist=[c,d]
print mydlist [{}, {}]
c['x']=1
print mydlist

[{'x': 1}, {}]

So it looks like variables in a list are stored as object references.
This seems to confirm that:

mydlist[1]['y']=4[{}, {'y': 4}]

So I figure my initial example doesn't work because if you assign a
literal to something it is changing the object. But modifying a list
or dict (as long as you don't re-construct it) does not change the
object.

I can think of some ways to work around this, including using single
element lists as "pointers":

aa=[1]
bb=[2]
myplist=[aa,bb]
print myplist [[1], [2]]
aa[0]=3
print myplist

[[3], [2]]


But what would be "the python way" to accomplish "list of variables"
functionality?

 

[Steven D'Aprano]

I'm relatively new to python and I was very surprised by the following
behavior:

[snip]

I don't see why. It's fairly unusual behaviour to want, and it would be
surprising if you did this:

def test():
x = 1
mylist = [2, 4, x]
function(mylist)
assert x == 1

and the assertion failed, even though you never passed x to the function.
Such behaviour could easily turn into a never-ending source of bugs.

So it looks like variables in a list are stored as object references.

Python doesn't store variables in lists, it stores objects, always.

Even Python variables aren't variables *grin*, although it's really
difficult to avoid using the term. Python variables are mappings between
names (strings) and objects, not memory locations.

So I figure my initial example doesn't work because if you assign a
literal to something it is changing the object. But modifying a list or
dict (as long as you don't re-construct it) does not change the object.

Yes, we talk about name binding (and rebinding) versus mutation. A simple
example:

alist = blist = [] # bind two names to the same list object
alist.append(1) # mutate the list (modify in place)
blist

[1]

alist = alist + [2] # a rebinding operation
blist [1]
alist

[1, 2]

I can think of some ways to work around this, including using single
element lists as "pointers":

Yes, that's a standard way to do it, except that by "standard" I mean
"really really really rare, honestly, hardly anyone does that".

Slightly less rare, but still uncommon, is to wrap objects in an instance:

class Record:
pass

o = Record()
o.x = 1
o.y = 2
modify(o)
print o.x, o.y

Of course you can make the class as fancy, or as simple, as you want.

But a better approach is to take advantage of Python's ability to return
multiple values:

x = 1
y = 2
x, y = modify(x, y)

rather than:

x = 1
y = 2
modify([x, y])


Speaking as an old Pascal coder, you won't miss pass-by-reference very
often.

 

[Carl Banks]

I'm relatively new to python and I was very surprised by the following behavior:

a=1
b=2
mylist=[a,b]
print mylist [1, 2]
a=3
print mylist

[1, 2]

Whoah!  Are python lists only for literals?  Nope:

c={}
d={}
mydlist=[c,d]
print mydlist [{}, {}]
c['x']=1
print mydlist

[{'x': 1}, {}]

So it looks like variables in a list are stored as object references.
This seems to confirm that:

mydlist[1]['y']=4>>> print mydlist

[{}, {'y': 4}]

So I figure my initial example doesn't work because if you assign a
literal to something it is changing the object.  But modifying a list
or dict (as long as you don't re-construct it) does not change the
object.

All correct, very observant for a Python newbie.

To be more

immutable

I can think of some ways to work around this, including using single
element lists as "pointers":

aa=[1]
bb=[2]
myplist=[aa,bb]
print myplist [[1], [2]]
aa[0]=3
print myplist

[[3], [2]]

But what would be "the python way" to accomplish "list of variables"
functionality?

Python doesn't have variable references (except in a limited way; see
below), so unless you want to use lists as pointers, I'd recommend
rethinking the problem. Occasionally I feel like I'd like to be able
to do this, but usually another way exists that is, at worst, slightly
more complex.

For some cases the easiest thing is to make all the variables you're
interested in listing attributes of an object (or values of a dict).
Then store a list of names (or keys).

class X(object): pass
x = X()
x.foo = 1
x.bar = 2
s = ["foo","bar"]
setattr(x,s[0],3)
print x.foo # prints 3

A rule of thumb in deciding whether to use attributes is whether you
will typically know their names in your code ahead of time; if so
storing the values as attributes is a good idea. If you are inputing
or calculating the names, then it's better to use a dict.


** The one place where Python does have references is when accessing
variables in an enclosing scope (not counting module-level). But
these references aren't objects, so you can't store them in a list, so
it can't help you:

def f():
s = []
a = 1
def g():
print a
s.append(a)
g() # prints 1
a = 2
g() # prints 2: g's a is a reference to f's a
print s # prints [1,2] not [2,2]


Carl Banks

 

[Steven D'Aprano]

The one place where Python does have references is when accessing
variables in an enclosing scope (not counting module-level).

What makes you say that?

But these
references aren't objects, so you can't store them in a list, so it
can't help you:

I don't even understand this. Your own example clearly shows that the are
objects and you can store them in a list, so I have no understanding of
what you mean.

def f():
s = []
a = 1
def g():
print a

a is a name bound to an object which inherits a __str__ method, hence you
can print it.

s.append(a)

a is bound to an object you can put in a list.

g() # prints 1
a = 2
g() # prints 2: g's a is a reference to f's a
print s # prints [1,2] not [2,2]

Yes, you are correct that lexical scoping doesn't allow the OP to embed
references to names in lists. I'm just confused why you think that
lexical scoping is equivalent to references that can't be put in lists,
or why you think this behaviour is any different from lexical scoping
everywhere else?

# Instead of two scopes, f and g, use two scopes, the module (global)
# and local scope g:
s = []
a = 1
def g():
print a
s.append(a)

g() # prints 1
a = 2
g() # prints 2: g's a is a reference to the global a
print s # prints [1,2] not [2,2]


There is no difference between lexical scoping between a function and a
nested function, and the lexical scoping between the global namespace and
a nested function.

 

[Carl Banks]

The one place where Python does have references is when accessing
variables in an enclosing scope (not counting module-level).  

What makes you say that?

But these
references aren't objects, so you can't store them in a list, so it
can't help you:

I don't even understand this. Your own example clearly shows that the are
objects and you can store them in a list, so I have no understanding of
what you mean.

def f():
    s = []
    a = 1
    def g():
        print a

a is a name bound to an object which inherits a __str__ method, hence you
can print it.

        s.append(a)

a is bound to an object you can put in a list.

    g() # prints 1
    a = 2
    g() # prints 2: g's a is a reference to f's a
    print s # prints [1,2] not [2,2]

Yes, you are correct that lexical scoping doesn't allow the OP to embed
references to names in lists. I'm just confused why you think that
lexical scoping is equivalent to references that can't be put in lists,
or why you think this behaviour is any different from lexical scoping
everywhere else?

# Instead of two scopes, f and g, use two scopes, the module (global)
# and local scope g:
s = []
a = 1
def g():
    print a
    s.append(a)

g() # prints 1
a = 2
g() # prints 2: g's a is a reference to the global a
print s # prints [1,2] not [2,2]

There is no difference between lexical scoping between a function and a
nested function, and the lexical scoping between the global namespace and
a nested function.

http://tinyurl.com/8e7tm



Carl Banks

 

[Lie Ryan]

Python doesn't store variables in lists, it stores objects, always.

Even Python variables aren't variables *grin*, although it's really
difficult to avoid using the term. Python variables are mappings between
names (strings) and objects, not memory locations.

Actually, no programming languages have used the term "variable"
correctly, as per its original mathematical definition.

 

[Steven D'Aprano]

On Feb 20, 10:50 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:

What makes you say that? [...]
I don't even understand this. [...]
I'm just confused why you think that
lexical scoping is equivalent to references that can't be put in lists,
or why you think this behaviour is any different from lexical scoping
everywhere else?

http://tinyurl.com/8e7tm

Ha ha ha, very amusing. But I was genuinely confused, because I know
you're an experienced Python coder and not some n00b, so I thought maybe
you were referring to something that escaped me. Your response, apt
though it may be, does nothing to clear up my confusion.

Presumably if you had a serious response you would have made it, so I'll
chalk it up to a generalised confusion field.

*wink*

 

[bartc]

I'm relatively new to python and I was very surprised by the following
behavior:

a=1
b=2
mylist=[a,b]
print mylist [1, 2]
a=3
print mylist

[1, 2]

Whoah! Are python lists only for literals? Nope:

c={}
d={}
mydlist=[c,d]
print mydlist [{}, {}]
c['x']=1
print mydlist

[{'x': 1}, {}]

So it looks like variables in a list are stored as object references.
This seems to confirm that:

mydlist[1]['y']=4[{}, {'y': 4}]

So I figure my initial example doesn't work because if you assign a

That shows a different outlook. I would have said your first example works
as expected and it was the second example that was strange, possibly due to
shallow instead of deep copies by Python.

 

[Gregory Ewing]

What makes you say that?

I think Carl is talking about cells, which *are* actually objects
(in CPython at least), but they're internal details of the
interpreter, and you can't do anything useful with them from
Python code.

 

[Aahz]

I'm relatively new to python and I was very surprised by the following
behavior:

http://starship.python.net/crew/mwh/hacks/objectthink.html
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer

 

[Alf P. Steinbach]

* Michael Pardee:

I'm relatively new to python and I was very surprised by the following behavior:

'a' refers to an object representing the integer 1.

Since 1 is an immutable value you can just as well think of it as 'a' containing
the value 1, because a reference to an immutable value is for nearly all
practical purposes indistinguishable from that value.

'b' refers to an object representing the integer 2, which, again, since integers
are immutable, you can just as well think of as 'b' containing the value 2.

mylist=[a,b]

A 'list' object (an array) is constructed, and 'mylist' is set to refer to it.
The array has two items. The first item is set to refer to same object as 'a' (a
reference copy), the second item is set to refer to the same object as 'b'.

But since integers are immutable you can just as well think of it as a copying
of the integer values.

Immutable values allow you to think of handling the values directly.

print mylist [1, 2]
a=3

This changes what 'a' refers to.

It does not change what the first array element refers to.

[1, 2]

Whoah! Are python lists only for literals?

No, that's an irrelevant notion.

Nope:

Here 'c' now refers to a dictionary object, which is mutable (can be changed).

And 'd' refers to another dictionary object.

mydlist=[c,d]
print mydlist

[{}, {}]

A 'list' array object is constructed, and 'mydlist' is set to refer to it.

The first item in the array is set to refer to the same object as 'c' refers to,
namely a mutable dictionary object (currently an empty dictionary).

The second item in the array is set to refer to the same object as 'd' refers
to, namely a mutable dictionary object (currently an empty dictionary).

It's the same that happened earlier with the integers.

The only difference, but it's a significant one, is that now the referred to
objects are mutable, that is, they can be changed.

c['x']=1

The dictionary object referred to by 'c' is updated to now have a key 'x' with
associated value 1.

In more gory detail: the key is associated with a reference to an object
representing 1, but again, since integer values are immutable you can think of
this as a direct association with the value; the reference view of this
association is mostly[1] only relevant when the referred to object is mutable.

Since the dictionary object that you're changing is referred to by both 'c' and
the first item of 'mydlist', both of these reflect the change.

[{'x': 1}, {}]
Yep.


So it looks like variables in a list are stored as object references.

You mean items in a list are references to objects. Yes.

This seems to confirm that:

mydlist[1]['y']=4[{}, {'y': 4}]

To check that you should instead have printed 'd', where you'd also see the
change, since 'd' refers to the same dictionary object as 'mydlist[1]' does.

So I figure my initial example doesn't work because if you assign a
literal to something it is changing the object.

No, it has nothing to do with literals.

With the current language definition[2], as of Python 2.x and 3.x, it's very
simple: assignments copy references, and that's all they do.

References to immutable objects allow you to think of values being copied
around, which except for checking the identities of those objects (seldom
relevant) yields the exact same conclusions about the effect of operations, but
that's only because those immutable objects never change.

What you did above was to copy references to mutable objects, objects that can
change.

Then the value-copying view breaks down.

But modifying a list
or dict (as long as you don't re-construct it) does not change the
object.

A simple way to think of this is copying references.

Objects are never copied by Python assignments.

The assignments only copy references.

I can think of some ways to work around this, including using single
element lists as "pointers":

aa=[1]
bb=[2]
myplist=[aa,bb]
print myplist [[1], [2]]
aa[0]=3
print myplist

[[3], [2]]

This is the same as your last example above, except that now you're using 'list'
arrays as the referred to mutable objects, instead of 'dict' dictionary objects.

But what would be "the python way" to accomplish "list of variables"
functionality?

Possibly, if you think of assignments as copying references, you won't need that
notion.


Cheers & hth.,

- Alf

Notes:
[1] The reference view can be relevant also for immutable objects in (at least)
two cases. One is when the program logic depends on object identity, obtainable
via the id function, which is just bad programming, but I mention it for
completeness. The other is where you have a really large immutable object, such
as in Python 2.x a very large 'long' value; then assignments would be
inefficient if the value was actually copied, but since assignments only copy
references in Python you can blissfully disregard the size of the object with
respect to assignments.
[2] Some wording in the language spec requires that an assignment behaves as if
references were copied, including preservation of object identity. But object
identity is almost never an issue for immutable objects, and a relaxation of
that wording, for immutable values, might permit some optimization. So it just
might change somewhen in the future, trading simplicity for some efficiency.