default value in __init__

[kenneth]

Dear all,

I have encountered this weird problem.

I have a class definition with an __init__ argument 'd'
which defaults to {}. This argument is put in the 'self.d'
attribute at initialization

I create two independent instances of this class; the code
is as follows.

class C:
def __init__(self, i=10, d = {}):
self.d = d
self.i = i
def get(self):
print
print self.d
def set(self, dval, ival):
self.d.update(dval)
self.i+=ival

c1=C()
c1.set({'one':1},3)
c1.get()

del c1

c2=C()
c2.set({'two':2},4)
c2.get()


If I run the code I obtain:

{'one': 1}

{'two': 2, 'one': 1}

It seems that the 'self.d' argument of the second instance is the
same of the 'self.d' of the first (deleted!) instance.

Running the code in a debugger I discovered that, when I enter the
__init__ at the second initialization, before doing

self.d = d

the 'd' variable already contains the 'self.d' value of the first
instance and not the default argument {}.

Am I doing some stupid error, or this is a problem ?

Thanks in advance for any help,
Paolo

 

[Chris Rebert]

See Pitfall #5 on http://zephyrfalcon.org/labs/python_pitfalls.html
It also applies to dictionaries (and sets, any mutable object really).

Dear all,

I have encountered this weird problem.

I have a class definition with an __init__ argument 'd'
which defaults to {}. This argument is put in the 'self.d'
attribute at initialization

I create two independent instances of this class; the code
is as follows.

class C:
def __init__(self, i=10, d = {}):

Change 'd = {}' to 'd=None'
Add the line:
if d is None: d = {}

Cheers,
Chris

 

[kenneth]

No, it always contains the default argument because default values are
created just ONE TIME.http://effbot.org/pyfaq/why-are-default-values-shared-between-objects...

Wow, it's a very "dangerous" behavior ...

Just to know, is this written somewhere in the python documentation or
one has to discover it when his programs fails to work ;-) ?

Paolo

 

[Bruno Desthuilliers]

kenneth a écrit :

Wow, it's a very "dangerous" behavior ...

Well, that's usually something you only have to learn once - then you
know !-)

 

[Bruno Desthuilliers]

David C. Ullrich a écrit :

At least once a week someone discovers this "problem", makes a
post about it here, and then someone points to the spot in the
documentation where it's explained.

Seems to me that people often site the "important warning" in
the tutorial. Of course there's no reason anyone would bother
going through the tutorial

Indeed. No reason at all.

- just for fun I looked in the
official Python Reference Manual to see whether they're explicit
about this or require the reader to figure it out from something
else they say.

There's a section titled "7.6 Function definitions". About halfway
through that section there's a _bold face_ statement
"Default parameter values are evaluated when the function definition is
executed.", followed by an explanation of how that can lead to
the sort of problem above.

But there's no reason to read the reference manual neither.

So I guess it _is_ awfully dangerous. They should really explain
this aspect of the language's behavior to people who don't read
the formal definition and also don't work through the tutorial.

You mean : "to people that don't bother reading the FineManual *nor*
searching the newsgroup / ML archives ?"

Well... How to say.. Is there any chance these people will read anything
*at all* ?

 

[bearophileHUGS]

Bruno Desthuilliers:

You mean : "to people that don't bother reading the FineManual *nor*
searching the newsgroup / ML archives ?"

Are there ways to change how Python3 manages arguments and functions,
to remove this antifeature of Python, avoiding this common mistake
done by every newbie?
I don't care if it reduces the performance of Python a little.

Bye,
bearophile

 

[bearophileHUGS]

Duncan Booth:

You can't just copy the default values on every call: you would still get
people confused by the semantics whether you did a shallow or deep copy or
as now no copy.

I think I agree.

I don't think simply re-executing the default argument
expression on each call works either: that would confuse at least as many
people as the current system.

May I ask you why? I think I don't agree, but I am not sure.

It would be possible, but extremely annoying to limit default arguments to
being literal constants,

This is a possible solution, beside re-executing the default argument
expression on each call.

unless you invent some kind of scheme for declaring that
a type is safe to use as a default argument.

Well, it seems functional-style programming may become more common in
the future, and seeing languages like Scala, etc, maybe it can be
useful to add to Python some way to define immutable classes (in an
explicit way). Maybe subclasses of Immutable?

Another option might just be to generate a warning the first time a program
uses a not obviously immutable default.

I don't like this solution much.

Even if you could change the behaviour of default arguments we
would still get equivalent regular questions from the people who initialise
class attributes with lists or dictionaries.

I have seen professional programmers too use class attributes instead
of instance ones...

Well, you can't create a fool-proof language that is useful too, but
in a language that is designed for new programmers too, like Python,
and that doesn't put the running speed as its most important feature,
then I think patching the most known and common pitfalls/traps is
useful, even if you can't patch them all (without turning the language
into something useless).

Bye,
bearophile

 

[Chris Rebert]

Bruno Desthuilliers:

Are there ways to change how Python3 manages arguments and functions,
to remove this antifeature of Python, avoiding this common mistake
done by every newbie?
I don't care if it reduces the performance of Python a little.

The general idea been discussed ad-nauseum on the list several times
before, including just 2 months ago. See e.g.:

[Python-3000] default argument surprises
http://mail.python.org/pipermail/python-3000/2008-August/014658.html

[Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
http://mail.python.org/pipermail/python-ideas/2007-January/000121.html

[Python-3000] pre-PEP: Default Argument Expressions
http://mail.python.org/pipermail/python-3000/2007-February/005704.html

Cheers,
Chris

 

[bearophileHUGS]

Chris Rebert:

The general idea been discussed ad-nauseum on the list several times
before, including just 2 months ago. See e.g.:

Okay, it can't be fixed then.

Bye and thank you,
bearophile

 

[Steven D'Aprano]

objects...


Wow, it's a very "dangerous" behavior ...

No, it's very *useful* behaviour.

 

[Steven D'Aprano]

May I ask you why? I think I don't agree, but I am not sure.

x = 100
def foo(a, b=x):
return a+b

first = foo(1)
x = 101
second = foo(1)

assert first == second


I think people will be rightly surprised that this fails.

This is a possible solution, beside re-executing the default argument
expression on each call.

That's no solution at all, because default arguments should not be
limited to literal constants. That's unacceptable in my opinion.

Well, it seems functional-style programming may become more common in
the future, and seeing languages like Scala, etc, maybe it can be useful
to add to Python some way to define immutable classes (in an explicit
way). Maybe subclasses of Immutable?

You're still assuming that the behaviour is a bug. It's not, it's a
feature.

I have seen professional programmers too use class attributes instead of
instance ones...

That's only a mistake if you don't mean to use class attributes instead
of instance attributes.

 

[Aaron \Castironpi\ Brady]

(snip)
I wonder whether it is the way the default argument expressions are
embedded inside the function that causes the confusion? If for example
default arguments were defined like this:

class C:
  @default(d={})
  def __init__(self, i=10, d):
    self.d = d
    self.i = i

would moving the expression before the 'def' make people less inclined to
be suprised that the object is shared?

You could of course define a wrapper to do call-time assignment:

@calltime( d= dict, e= tuple )
def foo( self, d, e ):

If this decorator appeared in the standard library, newbies would be
able to stumble upon it.

I don't think either semantic is more obvious from the syntax alone.
It could mean either thing just as reasonably, and if Python defined
the opposite, we'd be getting opposite complaints.

def f(): return [ 0, 1 ] ....
f().append( 2 )
f()

[0, 1]

It constructs a new object each time. In light of this, the burden of
proof could even fall on Python for the inconsistency. That is,
assuming that it's well- and always defined.

 

[Aaron \Castironpi\ Brady]

That's only a mistake if you don't mean to use class attributes instead
of instance attributes.

Off topic: That gives me an idea for an interesting technique.

class Foo( HasInstanceVars ):
class InstanceVars:
x= 0
y= None
z= [ 0, 1 ]

The __init__ method in HasInstanceVars adds any InstanceVars members
to the instance members. It's not terribly different than using
__init__-- slightly slower, slightly clearer. It could even include a
'__initvars__' variable which adds constructor parameters by name to
the instance. It's marginally problematic how to create new objects
each time Foo is instantiated. You could require factories, pickle-
unpickle the contents, require 'deepcopy' compatibility, execute a
string, or call a function which uniquely executes the class
statement.
1

 

[Bruno Desthuilliers]

David C. Ullrich a écrit :
Hum... Who are "they", exactly ?

Yes. Also add "don't read any books".
Indeed.

I think I started with some
book

As far as I'm concerned, I started with the FineManual(tm)'s tutorial.

(snip)

No. That's exactly the point!

Yeps. But I don't think we derive the same conclusions from that point.

Basic Python is so transparent that
you can start using it without reading anything, just looking at
a few examples.

And immediatly fall into a well-known (and well-documented) "gotcha" -
like forgetting to apply the call operator to function, using mutable
default arguments, using mutable class attributes instead of instance
attributes, etc, etc, etc...

While it has been heavily publicized that Python is easy to get started
with - which, AFAICT, seems to still be true for any programmer already
having some experience *and* serious enough to actually _read_ that
fucking manual -, it doesn't mean that, by the virtue of some
extraterrestrial magic, one could master the language just reading a
couple dummy examples. Learning a language requires some home work,
period.

_Because_ of that it's their

who are "they" in this "their" ?

responsibility to
ensure that if you look at a few examples you then have a complete
understanding of the language.

I'm afraid I have to say this is *totally* utopic - except perhaps for
the most braindead and limited language one could imagine. Heck, even
Java - which has explicitely been designed for morons^Mdummies - would
fail this test. How could "a few examples" provide complete
understanding of attribute lookup rules, the descriptor protocol,
metaclasses, iterators, and generator expression - just to name a few ?

Granted, Python is easy to get started with. This doesn't mean anyone
should expect it to work like any language XXX they already know. Every
language has it's own pecularities, every language requires a serious
learning effort, and Python is no exception here.

FWIW, on most programming language-related newgroups, the only answer to
such FAQs would be (at best) a raw RTFM. Not that I'd advertise such a
behaviour here - this newsgroup is mostly newbies-friendly and I
*really* appreciate it to be that way (and happily answer the same 101
questions year after year FWIW), but I really fail to see how anything
more could be done for points that are 1/ clearly documented and 2/
reexplained here month after month.

In particular default parameters should work the way the user
expects! The fact that different users will expect different
things here is no excuse...

If different users expect different - mostly incompatible - things, how
would it be possible to have it working "the way the user expect" ?
Should Python grow some telepathic features to guess the user's
expectations and modifies itself to meet these expectations ?-)

 

[Aaron \Castironpi\ Brady]

David C. Ullrich a écrit :


snip

If different users expect different - mostly incompatible - things, how
would it be possible to have it working "the way the user expect" ?
Should Python grow some telepathic features to guess the user's
expectations and modifies itself to meet these expectations ?-)

No. Just have a user community that only has one user.

 

[Lawrence D'Oliveiro]

No, it's very *useful* behaviour.

Can you give an example of how useful it is? Something worth the pain of
newbies tripping over it every week?

 

[Aaron \Castironpi\ Brady]

Can you give an example of how useful it is? Something worth the pain of
newbies tripping over it every week?

Not to be overly practical, but what kind of community push would van
Rossum need in order to make a change, especially with 3.0 almost
out? Even if everyone agrees, it seems too late even for the entire
3.x series, to be changing something that deeply embedded in not only
_syntax_, but programmers' minds. I'd say a decorator would be a
viable alternative, but even that would be hard to get into the
language now.

 

[Steve Holden]

Aaron "Castironpi" Brady wrote:
[about how default argument behavior should, in his opinion, be changed]

Say what you like. The language is as it is by choice. Were it, for some
reason, to change we would then be receiving posts every week that
didn't understand the *new* behavior.

Sometimes people just have to learn to confirm with reality instead of
requiring reality to confirm with their preconceptions. This is one such
case.

regards
Steve

 

[Aaron \Castironpi\ Brady]

Aaron "Castironpi" Brady wrote:

[about how default argument behavior should, in his opinion, be changed]

Say what you like. The language is as it is by choice. Were it, for some
reason, to change we would then be receiving posts every week that
didn't understand the *new* behavior.

Sometimes people just have to learn to confirm with reality instead of
requiring reality to confirm with their preconceptions. This is one such
case.

regards
 Steve

I am not convinced it should either stay or go, but it's hard to argue
one way or the other about something so deeply entrenched. However,
what are your thoughts, whatever the default behavior is, on a
decorator that provides the alternative? That is, a decorator that
either reevaluates default arguments each time when the language
evaluates them once, or a decorator that evaluates arguments once,
when the languages evaluates them each time?

P.S.

we would then be receiving posts every week that
didn't understand the *new* behavior.

That is not obvious and I don't know of any empirical evidence that
entails it. Hard to search the standard library for that figure.

 

[Chris Rebert]

Aaron "Castironpi" Brady wrote:

[about how default argument behavior should, in his opinion, be changed]

Say what you like. The language is as it is by choice. Were it, for some
reason, to change we would then be receiving posts every week that
didn't understand the *new* behavior.

Sometimes people just have to learn to confirm with reality instead of
requiring reality to confirm with their preconceptions. This is one such
case.

regards
Steve

I am not convinced it should either stay or go, but it's hard to argue
one way or the other about something so deeply entrenched. However,
what are your thoughts, whatever the default behavior is, on a
decorator that provides the alternative? That is, a decorator that
either reevaluates default arguments each time when the language
evaluates them once, or a decorator that evaluates arguments once,
when the languages evaluates them each time?

P.S.

we would then be receiving posts every week that
didn't understand the *new* behavior.

That is not obvious and I don't know of any empirical evidence that
entails it. Hard to search the standard library for that figure.

Although primitive and likely somewhat flawed, you may find the
statistics in the "Compatibility Issues" section of
http://mail.python.org/pipermail/python-3000/2007-February/005704.html
to be of interest.

Cheers,
Chris[/QUOTE]