Bug in __init__?

[Zbigniew Braniecki]

I found a bug in my code today, and spent an hour trying to locate it
and then minimize the testcase.

Once I did it, I'm still confused about the behavior and I could not
find any reference to this behavior in docs.

testcase:

class A():

def add (self, el):
self.lst.extend(el)

def __init__ (self, val=[]):
print val
self.lst = val


def test ():
x = A()
x.add(["foo1","foo2"])
b = A()


So, what I would expect here is that I will create two instances of
class A with empty self.lst property. Right?

In fact (at least with my Python 2.5)

gandalf@gandalf-desktop:~/projects/pyl10n$ ./scripts/test.py
[]
['foo1', 'foo2']

This bug does not happen when I switch to __init__ (self, *args) and
assign self.lst= args[0].

Any clue on what's going on here, and/if where I should report it?

Greetings
Zbigniew Braniecki

 

[Christian Heimes]

Any clue on what's going on here, and/if where I should report it?

Congratulations! You've stumbled over a well known gotcha. Most newbies
fall for the trap.

class A:
def __init__ (self, val=[]):
print val
self.lst = val

val is created only *once* and shared across all instaces of A.

Christian

 

[Eduardo O. Padoan]

I found a bug in my code today, and spent an hour trying to locate it
and then minimize the testcase.

Once I did it, I'm still confused about the behavior and I could not
find any reference to this behavior in docs.

testcase:

class A():

def add (self, el):
self.lst.extend(el)

def __init__ (self, val=[]):
print val
self.lst = val


def test ():
x = A()
x.add(["foo1","foo2"])
b = A()


So, what I would expect here is that I will create two instances of
class A with empty self.lst property. Right?

In fact (at least with my Python 2.5)

gandalf@gandalf-desktop:~/projects/pyl10n$ ./scripts/test.py
[]
['foo1', 'foo2']

This bug does not happen when I switch to __init__ (self, *args) and
assign self.lst= args[0].

Any clue on what's going on here, and/if where I should report it?

It is a FAQ, not a bug:
http://www.python.org/doc/faq/general/#why-are-default-values-shared-between-objects
http://effbot.org/pyfaq/why-are-default-values-shared-between-objects.htm

 

[Matimus]

I found a bug in my code today, and spent an hour trying to locate it
and then minimize the testcase.

Once I did it, I'm still confused about the behavior and I could not
find any reference to this behavior in docs.

testcase:

class A():

def add (self, el):
self.lst.extend(el)

def __init__ (self, val=[]):
print val
self.lst = val

def test ():
x = A()
x.add(["foo1","foo2"])
b = A()

So, what I would expect here is that I will create two instances of
class A with empty self.lst property. Right?

In fact (at least with my Python 2.5)

gandalf@gandalf-desktop:~/projects/pyl10n$ ./scripts/test.py
[]
['foo1', 'foo2']

This bug does not happen when I switch to __init__ (self, *args) and
assign self.lst= args[0].

Any clue on what's going on here, and/if where I should report it?

Greetings
Zbigniew Braniecki

Look at item number 5. http://zephyrfalcon.org/labs/python_pitfalls.html

People run into this quite often. Basicly, you created a list as a
default argument. This doesn't create a new empty list ever time
__init__ is called. Everyone is getting the same list. How to get
around this:

class A(object): # derive from object for new-style classes
def add (self, el):
self.lst.extend(el)

def __init__ (self, val=None): # Use None instead
if val is None:
val = [] # create a new list if needed each time
print val
self.lst = val


HTH

Matt

 

[Martin Blume]

"Zbigniew Braniecki" schrieb

I found a bug in my code today, and spent an hour trying to locate it
and then minimize the testcase.
[...]
def __init__ (self, val=[]):
[...]
Any clue on what's going on here, and/if where I should report it?

I think this has to do with
http://docs.python.org/tut/node6.html#SECTION006710000000000000000
especially the "Important Warning"

Regards
Martin

 

[Zbigniew Braniecki]

Any clue on what's going on here, and/if where I should report it?

Congratulations! You've stumbled over a well known gotcha. Most newbies
fall for the trap.

class A:
def __init__ (self, val=[]):
print val
self.lst = val

val is created only *once* and shared across all instaces of A.

Thanks for help guys!

It's really a nice pitfall, I can hardly imagine anyone expecting this,
or how easily could I find this info (e.g. what query should I give to
google to get it without bothering people on this group)

Anyway, thanks

Greetings
Zbigniew Braniecki

 

[Fredrik Lundh]

It's really a nice pitfall, I can hardly imagine anyone expecting this,
or how easily could I find this info (e.g. what query should I give to
google to get it without bothering people on this group)

looking things up in the documentation *before* deciding that you might
have done something that nobody's done before is often a good idea:

http://docs.python.org/tut/node6.html#SECTION006710000000000000000

"Important warning: The default value is evaluated only once.
This makes a difference when the default is a mutable object
such as a list, dictionary, or instances of most classes.
/.../"

http://docs.python.org/ref/function.html

"Default parameter values are evaluated when the function
definition is executed. This means that the expression is
evaluated once, when the function is defined, and that
that same ``pre-computed'' value is used for each call.
This is especially important to understand when a default
parameter is a mutable object, such as a list or a
dictionary /.../

(to be precise, the default values are evaluated when the "def" state-
ment is executed, in the same scope as the "def" statement itself. if
you execute the same "def" statement multiple times, the values are
recalculated.)

</F>

 

[Neil Cerutti]

class A:
def __init__ (self, val=[]):
print val
self.lst = val

val is created only *once* and shared across all instaces of A.

Thanks for help guys!

It's really a nice pitfall, I can hardly imagine anyone expecting this,
or how easily could I find this info (e.g. what query should I give to
google to get it without bothering people on this group)

In this unfortunate case, useful searches were "default arguments",
which would be hard to guess without already knowing what's going
wrong, or "python gotchas pitfalls", which is a good general purpose
search for when you can't understand what's happening in simple code.

 

[Bart Ogryczak]

I found a bug in my code today, and spent an hour trying to locate it
and then minimize the testcase.

Once I did it, I'm still confused about the behavior and I could not
find any reference to this behavior in docs.

testcase:

class A():

def add (self, el):
self.lst.extend(el)

def __init__ (self, val=[]):
print val
self.lst = val

What you want probably is:
def __init__ (self, val=None):
if(val == None):
self.lst = []
else:
from copy import copy
### see also deepcopy
self.lst = copy(val)

def test ():
x = A()
x.add(["foo1","foo2"])
b = A()


So, what I would expect here is that I will create two instances of
class A with empty self.lst property. Right?

Wrong. default value for val gets evaluated only once, creating a list
(initialy empty). The same list for all of A's instances.

13188912

13188912

Moreover, self.lst = val, does not copy val, rather it creates binding
between self.list and val. So whatever you do to self.list, it affects
val (and vice-versa).

x = []
c = A(x)
id(x) 10508368
id(c.lst) 10508368
c.lst.append('wrong')
x

['wrong']

bart

 

[Bart Ogryczak]

It's really a nice pitfall, I can hardly imagine anyone expecting this,

AFAIR, it's described in Diving Into Python.
It's quiet elegant way of creating cache.

def calculate(x,_cache={}):
try:
return _cache[x]
except KeyError:
_cache[x] = result = _lotsa_slow_calculations(x)
return result

bart

 

[Arnaud Delobelle]

AFAIR, it's described in Diving Into Python.

Still there seems to be about one message a week about this. Indeed I
reckon the greatest benefit of early binding of default function
arguments is that it attracts lots of new people to comp.lang.python.

It's quiet elegant way of creating cache.

IMHO, calling it 'elegant' is pushing it too far!

 

[Bart Ogryczak]

Still there seems to be about one message a week about this. Indeed I
reckon the greatest benefit of early binding of default function
arguments is that it attracts lots of new people to comp.lang.python.

Generally there's lot of confusion about assigments of objects.
I guess, there are lot of ppl, who started with languages like PHP,
where $a = $b, translates to Python's a = copy(b).

IMHO, calling it 'elegant' is pushing it too far!

Ok, maybe that's not a good choice of word.
Not elegant, minimalist.

bart

 

[cokofreedom]

Is there no way of adding a possible warning message (that obviously
could be turned off) to warn users of possible problems linked to
using mutable types as parameters?

Seems to me this could save users a few minutes/hours of headaches and
headscratching. (The biggest issue affecting new programmers these
days!)

 

[Gabriel Genellina]

Is there no way of adding a possible warning message (that obviously
could be turned off) to warn users of possible problems linked to
using mutable types as parameters?

Seems to me this could save users a few minutes/hours of headaches and
headscratching. (The biggest issue affecting new programmers these
days!)

Most objects are mutable, such warning would happen so often that would
become useless.
The only immutable objects are numbers, strings, tuples, None and a few
esoteric types; all other instances, including instances of all user
defined classes, are mutable unless explicitely their attributes are
read-only.

 

[Steven D'Aprano]

Most objects are mutable, such warning would happen so often that would
become useless.

Not at all, for two reasons:

(1) Using mutable defaults in function definitions is relatively unusual.
Defaults like None, 0, 1 are far more common.

(2) We can copy the behaviour of the warnings module. Possibly even use
the warnings module. By default warnings are only shown once per session.

The only immutable objects are numbers, strings, tuples, None and a few
esoteric types; all other instances, including instances of all user
defined classes, are mutable unless explicitely their attributes are
read-only.

It is true that Python doesn't have a cheap way of recognising mutable
instances except by trying to mutate them. However, there is a case for
having it issue a warning the first time in a session it spots a default
list or dict, which would catch 99% of cases that newbies use a mutable
default.

Newbies being newbies, 30% of them will completely ignore the warning and
bitch about the "bug", but that's better than the 80% that we have now

 

[Bruno Desthuilliers]

Bart Ogryczak a écrit :

On 2008-01-18, citizen Zbigniew Braniecki testified:

(snip usual default mutable list arg problem)

class A():

def add (self, el):
self.lst.extend(el)

def __init__ (self, val=[]):
print val
self.lst = val

What you want probably is:
def __init__ (self, val=None):
if(val == None):

Better to use an identity test here - there's only one instance of the
None object, and identity test is faster than equality test (one
function call faster IIRC !-). Also, the parens are totallu useless.

if val is None:

self.lst = []
else:
from copy import copy
### see also deepcopy
self.lst = copy(val)

What makes you think the OP wants a copy ?

 

[Bart Ogryczak]

What makes you think the OP wants a copy ?

I´m guessing he doesn´t want to mutate original list, while changing
contents of self.lst.

bart

 

[Bruno Desthuilliers]

Bart Ogryczak a écrit :

I´m guessing he doesn´t want to mutate original list, while changing
contents of self.lst.

Once again: what makes you think so ?

This is a design choice depending on the concrete use case and context -
which we don't know zilch about - and by no mean the default behaviour.