Default Value

[Ahmed Abdulshafy]

I'm reading the Python.org tutorial right now, and I found this part rather strange and incomprehensible to me>

Important warning: The default value is evaluated only once. This makes a difference when the default is a mutable object such as a list, dictionary, or instances of most classes
def f(a, L=[]):
L.append(a)
return L

print(f(1))
print(f(2))
print(f(3))

This will print
[1]
[1, 2]
[1, 2, 3]

How the list is retained between successive calls? And why?

 

[Jussi Piitulainen]

I'm reading the Python.org tutorial right now, and I found this part
rather strange and incomprehensible to me

Important warning: The default value is evaluated only once. This
makes a difference when the default is a mutable object such as a
list, dictionary, or instances of most classes

def f(a, L=[]):
L.append(a)
return L

print(f(1))
print(f(2))
print(f(3))

This will print
[1]
[1, 2]
[1, 2, 3]

How the list is retained between successive calls? And why?

Functions are objects with their own attributes. The default values
are computed and stored when the def statement is executed.

dir(f) will show the attributes of f. Their names start and end with a
double underscore, __. This indicates that they are not usually used
explicitly. Still, f.__defaults__ seems to be where the default value
is kept between (and during) calls.

 

[Rick Johnson]

I'm reading the Python.org tutorial right now, and I found
this part rather strange and incomprehensible to me>

Important warning: The default value is evaluated only
once. This makes a difference when the default is a
mutable object such as a list, dictionary, or instances of
most classes

def f(a, L=[]):
L.append(a)
return L

print(f(1))
print(f(2))
print(f(3))

This will print

[1]
[1, 2]
[1, 2, 3]

How the list is retained between successive calls? And
why?

By the evil hands of an illogical consistency.

Have you ever heard the old adage: "The road to hell is
paved in good intentions"? Well, apparently the original
designers of the language called in sick the day that class
was taught. It's unfortunate because the same functionality
that this "intention" claims to solve can be reproduced
easily, and in a less astonishing manner, by the programmer
himself.

http://en.wikipedia.org/wiki/Principle_of_least_astonishment

 

[Gary Herron]

I'm reading the Python.org tutorial right now, and I found this part rather strange and incomprehensible to me>

Important warning: The default value is evaluated only once. This makes a difference when the default is a mutable object such as a list, dictionary, or instances of most classes

This code:

def f(a, L=[]):
L.append(a)
return L

does the same as this code:

M=[]
def f(a, L=M):
L.append(a)
return L

where it's slightly more obvious that the list is created once, and
modified with each call to the function (or rather with each call to the
function that does not supply its own value for L).

Gary Herron

print(f(1))
print(f(2))
print(f(3))

This will print
[1]
[1, 2]
[1, 2, 3]

How the list is retained between successive calls? And why?

 

[Steven D'Aprano]

I'm reading the Python.org tutorial right now, and I found this part
rather strange and incomprehensible to me>

Important warning: The default value is evaluated only once. This makes
a difference when the default is a mutable object such as a list,
dictionary, or instances of most classes def f(a, L=[]):
L.append(a)
return L

print(f(1))
print(f(2))
print(f(3))

This will print
[1]
[1, 2]
[1, 2, 3]

How the list is retained between successive calls? And why?

"How" is easy: it is stored inside the function object, where the code
can get to it. To be precise, in the function object's __defaults__
attribute:

py> def f(a=1, b=5, c="hello world"):
.... pass
....
py> f.__defaults__
(1, 5, 'hello world')


"Why" is even easier: because the creator of Python decided to do it this
way.

This is called "early binding" -- the default value is bound (assigned)
to the parameter early in the process, when the function is created. This
has a few advantages:

- It is the most efficient: the cost of evaluating the defaults
only happens once, when the function is created, not over and
over again, every time the function is called.

- It is usually what people expect. If you have a function with
a default, you normally expect the default to be set once, and
not re-evaluated each time.

- If you start with "early binding", it is trivial to get "late
binding" semantics instead; but if you start with late binding,
it's less convenient to get early binding semantics.

The usual way to get the effect of late binding in Python is with a
sentinel value to trigger the re-evaluation of the default. Normally, we
would use None and a call-time check:

def function(arg, value=None):
if value is None:
# Re-calculate the default.
value = ...
# rest of code

 

[rusi]

I'm reading the Python.org tutorial right now, and I found
this part rather strange and incomprehensible to me>

Important warning: The default value is evaluated only
once. This makes a difference when the default is a
mutable object such as a list, dictionary, or instances of
most classes

def f(a, L=[]):
    L.append(a)
    return L

print(f(1))
print(f(2))
print(f(3))

This will print

[1]
[1, 2]
[1, 2, 3]

How the list is retained between successive calls? And
why?

By the evil hands of an illogical consistency.

Have you ever heard the old adage: "The road to hell is
paved in good intentions"? Well, apparently the original
designers of the language called in sick the day that class
was taught. It's unfortunate because the same functionality
that this "intention" claims to solve can be reproduced
easily, and in a less astonishing manner, by the programmer
himself.

 http://en.wikipedia.org/wiki/Principle_of_least_astonishment

Every language has gotchas. This is one of python's.

If you are a beginning python programmer, really the best answer is:
Just dont do it! Dont do what? Dont use mutable arguments as function
defaults.
And when you feel that you need to, use Steven's trick: use a
immutable indicator 'None' for the mutable [].

Once you cross the noob stage you can check that its a standard
gotcha: Just run a search for 'python gotcha default []'

Its even been discussed repeatedly on the python-ideas list
Heres a correction suggestion: http://mail.python.org/pipermail/python-ideas/2007-January/000073.html
Here's Terry Reedy's nicely explaining the 'why-of-the-how' :
http://mail.python.org/pipermail/python-ideas/2009-May/004680.html

FWIW here is a conceptual/philosophical explanation of your confusion:

There are 2 fundamental ways for approaching the programming task --
functional and imperative.
In FP, the primary requirement is that the behavior of a function
should be entirely determinable by its arguments.
In Imp. P, computation proceeds by changing state.

Now from the time of Backus turing award http://amturing.acm.org/award_winners/backus_0703524.cfm
it is known that the two philosophies are mutually incompatible. Now
python is in a rather unique position in this respect: it supports
many of the features of modern FPLs and one can almost do functional
programming in it. In a few cases the imperative/Object-based
internals leak out, this is one of those cases.

That such leakage will occasionally happen see:
http://www.joelonsoftware.com/articles/LeakyAbstractions.html

And you've had a little excess of the FP koolaid read :
http://blog.languager.org/2012/08/functional-programming-philosophical.html

 

[Roy Smith]

def f(a, L=[]):
    L.append(a)
    return L

Every language has gotchas. This is one of python's.

One of our pre-interview screening questions for Python programmers at
Songza is about this. I haven't been keeping careful statistics, but
I'd guess only about 50% of the candidates get this right.

 

[rusi]

def f(a, L=[]):
    L.append(a)
    return L

Every language has gotchas. This is one of python's.

One of our pre-interview screening questions for Python programmers at
Songza is about this.  I haven't been keeping careful statistics, but
I'd guess only about 50% of the candidates get this right.

See http://www.cs.utexas.edu/~EWD/transcriptions/EWD04xx/EWD480.html
(search forward to 'disastrous blending' )
No I am not saying that such knowledge is not required in the 'real
world'

However I do feel that just as a view of literature that does not go
beyond gothic horror is a poor view;
and just because for languages like C (even more C++) expertise more
or less equivales gothicness,
for a cleaner world like python we should distinguish the seedy areas
we know about but avoid and the clean areas we live in.
Clearly flagging gotchas as such helps in that direction.

In short (and to the OP): If you did NOT find this confusing, it would
be cause for more concern

 

[Rick Johnson]

Every language has gotchas. This is one of python's.

So are we purposely injecting illogic into our language just
to be part of some "cool crowd" of programming languages with
gotchas.

"You thought intuitiveness was a virtue? Haha, we gotcha!"

Or maybe this is reminiscent of the fraternity hazing rituals
of days gone by:

*POP*
"Thank you Sir, may i have another?"

If you are a beginning python programmer, really the best
answer is: Just dont do it! Dont do what? Dont use
mutable arguments as function defaults. Once you cross the
noob stage you can check that its a standard gotcha: Just
run a search for 'python gotcha default []'

And when you feel that you need to, use Steven's trick:
use a immutable indicator 'None' for the mutable []. Once
you cross the noob stage you can check that its a standard
gotcha: Just run a search for 'python gotcha default []'
Its even been discussed repeatedly on the python-ideas
list

Your attempting to excuse an inexcusable language flaw by
pointing out that the effects of the flaw can be avoided by
using a workaround. And, to add insult to injury, you
provide no good reason for the flaw to exist:

"Just do x, y, and z and shut up. Yes we made a mistake
but we are not about to admit our mistake and the onerous
will be on all the noobs to re-invent the workaround each
time"

To me that is unacceptable.

Heres a correction suggestion: [...] Here's Terry Reedy's
nicely explaining the 'why-of-the-how' : [...] FWIW here
is a conceptual/philosophical explanation of your
confusion: There are 2 fundamental ways for approaching
the programming task -- functional and imperative.

All these "explanations" ignore a fundamental fact about
subroutines[1].

A call to a subroutine should exists as a unique transaction
within the entire program. It is expected to optionally take
inputs, and optionally return an output (depending on the
definition).

When the subroutine is completed, all inputs and local
variables are expected to be destroyed. If the programmer
wants a return value, he need simply ask. Data persistence
is not a function of subroutines! Finally, a subroutine
should never have side effects UNLESS the programmer
explicitly ask for a side effect.

However, the Python flaw of allowing mutable sequence
arguments to persist between successive calls of a
subroutine is breaking the entire definition of a
subroutine, and for what noble purpose i ask? What purpose
is SO important that you change a well established interface
in a totally unintuitive manner?

If your answer is that recursion is easier, then i say that
is foolish because a programmer can keep a reference
to a mutable sequence OUTSIDE the subroutine and you can
save the "gotchas" for Guido's next birthday party.

[1]: http://en.wikipedia.org/wiki/Subroutine

 

[Chris Angelico]

When the subroutine is completed, all inputs and local
variables are expected to be destroyed. If the programmer
wants a return value, he need simply ask. Data persistence
is not a function of subroutines!

Funny, C violates your description too.

int counter()
{
static int count;
return ++count;
}

Function defaults in Python, being implemented as attributes on the
function object, are very similar in nature to static variables in C.
They're constructed once at function creation time, they're available
to that function (okay, C doesn't have any concept of "reaching into"
a function from the outside, Python does), they out-last any
particular execution of that function.

Finally, a subroutine
should never have side effects UNLESS the programmer
explicitly ask for a side effect.

Bogus.

http://www.postgresql.org/docs/current/static/sql-createfunction.html

By default, a function is assumed to have side effects. The
programmer/sysadmin has to explicitly ask for PostgreSQL to treat the
function as NOT having side effects (the IMMUTABLE attribute, or
possibly its weaker cousin STABLE).

ChrisA

 

[rusi]

You know Rick, you are good at python, you are better at polemics.
If only you would cut the crap I would (be able to) agree with you.
See below

So are we purposely injecting illogic into our language just
to be part of some "cool crowd" of programming languages with
gotchas.

 "You thought intuitiveness was a virtue? Haha, we gotcha!"

Python (and all the other 'cool' languages) dont have gotchas because
someone malevolently put them there.
In most cases, the problem is seen too late and the cost of changing
entrenched code too great.
Or the problem is clear, the solution is not etc etc.

Or maybe this is reminiscent of the fraternity hazing rituals
of days gone by:

 *POP*
 "Thank you Sir, may i have another?"

If you are a beginning python programmer, really the best
answer is: Just dont do it!  Dont do what? Dont use
mutable arguments as function defaults. Once you cross the
noob stage you can check that its a standard gotcha: Just
run a search for 'python gotcha default []'

And when you feel that you need to, use Steven's trick:
use a immutable indicator 'None' for the mutable []. Once
you cross the noob stage you can check that its a standard
gotcha: Just run a search for 'python gotcha default []'
Its even been discussed repeatedly on the python-ideas
list

Your attempting to excuse an inexcusable language flaw by
pointing out that the effects of the flaw can be avoided by
using a workaround. And, to add insult to injury, you
provide no good reason for the flaw to exist:

 "Just do x, y, and z and shut up. Yes we made a mistake
 but we are not about to admit our mistake and the onerous
 will be on all the noobs to re-invent the workaround each
 time"

To me that is unacceptable.

Heres a correction suggestion: [...] Here's Terry Reedy's
nicely explaining the 'why-of-the-how' : [...] FWIW here
is a conceptual/philosophical explanation of your
confusion: There are 2 fundamental ways for approaching
the programming task -- functional and imperative.

All these "explanations" ignore a fundamental fact about
subroutines[1].

A call to a subroutine should exists as a unique transaction
within the entire program. It is expected to optionally take
inputs, and optionally return an output (depending on the
definition).

When the subroutine is completed, all inputs and local
variables are expected to be destroyed. If the programmer
wants a return value, he need simply ask. Data persistence
is not a function of subroutines! Finally, a subroutine
should never have side effects UNLESS the programmer
explicitly ask for a side effect.

However, the Python flaw of allowing mutable sequence
arguments to persist between successive calls of a
subroutine is breaking the entire definition of a
subroutine, and for what noble purpose i ask? What purpose
is SO important that you change a well established interface
in a totally unintuitive manner?

If your answer is that recursion is easier, then i say that
is foolish because a programmer can keep a reference
to a mutable sequence OUTSIDE the subroutine and you can
save the "gotchas" for Guido's next birthday party.

[1]:http://en.wikipedia.org/wiki/Subroutine

You are saying in different language what I said: Functional
programming is a good idea, imperative programming is a bad idea.
From the invention of subroutines (ie 50 years) PL designers are
hearing this truth but somehow or other fail to listen; for more
details see http://blog.languager.org/2012/11/imperative-programming-lessons-not.html

 

[Chris Angelico]

Python (and all the other 'cool' languages) dont have gotchas because
someone malevolently put them there.
In most cases, the problem is seen too late and the cost of changing
entrenched code too great.
Or the problem is clear, the solution is not etc etc.

Or, in many MANY cases, the choice was the right one, but isn't
obvious to everyone.

ChrisA

 

[rusi]

Or, in many MANY cases, the choice was the right one, but isn't
obvious to everyone.

ChrisA

Ha!

If you've ever taught a first programming class you would know how
frequent is the complaint (said by a beginning student in a loud
confident voice): THE COMPILER HAS A BUG!

 

[Ian Kelly]

Or, in many MANY cases, the choice was the right one, but isn't
obvious to everyone.

I think it's worth pointing out that changing function defaults to
late-binding would merely change the nature of the gotcha, not eliminate it.

words = ("one", "two", "red", "blue", "fish")

def join_strings(strings=words):
return ' '.join('%s %s' % (s, strings[-1]) for s in strings[:-1])

# Later:
words = open("gettysburg_address.txt", "r").read().split()
# Oops, the default argument of join_strings just changed.


Additionally, with late-binding semantics the default values would no
longer be default *values*. They would be initialization code
instead, which sort of flies in the face of the idea that late-binding
would somehow be better for functional programming -- if the default
expressions have no side effects and since they don't depend on the
function arguments, why should they need to run more than once? If
the goal is indeed to make the the functions more functional, then the
proper solution would be to keep the binding early but just disallow
mutable defaults altogether -- which is tricky to achieve in Python,
so we simply emulate it with the advice "don't use mutable function
defaults".

 

[Rick Johnson]

Function defaults in Python, being implemented as
attributes on the function object, are very similar in
nature to static variables in C.

Oh wait a minute. i think it's becoming clear to me now!

Python functions are objects that take arguments, of which
(the arguments) are then converted to attributes of the
function object. Ah-Ha! Urm, but wait! We already have a
method to define Objects. Heck, I can even create my own
callable objects if i want!

Observe:
py> class FuncAdd(object):
... def __init__(self, ivalue):
... self.ivalue = ivalue
... def __call__(self, numeric):
... return self.ivalue + numeric
...
py> fa = FuncAdd(10)
py> fa(10)
20
py> fa(20)
30

I can even emulate (quite easily) this idea of
"persistence of function arguments" with mutables
too, Yippee!

py> class ArgPersist(object):
... def __init__(self, lst):
... self.lst = lst
... def __call__(self, arg):
... self.lst.append(arg)
... return self.lst
...
py> fp = ArgPersist([1,2,3])
py> fp(4)
[1, 2, 3, 4]
py> fp([5,6,7])
[1, 2, 3, 4, [5, 6, 7]]

But then, i can even do it WITHOUT creating an object
definition:

py> mutable = []
py> def expandMutable(arg):
... mutable.append(arg)
...
py> expandMutable(1)
py> mutable
[1]
py> expandMutable([2,3,4])
py> mutable
[1, [2, 3, 4]]

ANY of those approaches are much less confusing than the
current flaw and do not violate the least astonishment law..
I'm quite Okay with Python functions being first class
objects, however, i am not okay with violating the
fundamental nature of subroutines, especially when that
violation can offer no clear and arguable benefits and is in
fact unreasonably esoteric in nature.

 

[Rick Johnson]

Python (and all the other 'cool' languages) dont have
gotchas because someone malevolently put them there. In
most cases, the problem is seen too late and the cost of
changing entrenched code too great.

Okay. So now you are admitting the problem. That is a good
start. Thanks for being honest.

Or the problem is clear, the solution is not etc etc.

Both the problem and solution are clear, however, the will to execute
the solution remains to be seem. Are you telling me we can
justify breaking backwards compatibility for the sake of
"print"[1] but we cannot justify breaking it for this
monstrosity? Come on Rusi, you've gone too far with the
comedy now.

[1]: FYI I personally think print as function is the correct implementation.

 

[rusi]

If
the goal is indeed to make the the functions more functional, then the
proper solution would be to keep the binding early but just disallow
mutable defaults altogether -- which is tricky to achieve in Python,
so we simply emulate it with the advice "don't use mutable function
defaults".

Nicely put. "Imperative programming is a bad idea; FP is a good idea"
is an attractive religious dogma.
When put into practice with full religious zeal, we then need
something like monads to make realistic programming possible.
Which then becomes a case of "remedy worse than disease".

Which is why I prefer to keep FP in the limbo region between ideology
and technology

 

[Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC]]

Or, in many MANY cases, the choice was the right one, but isn't
obvious to everyone.

I think it's worth pointing out that changing function defaults to
late-binding would merely change the nature of the gotcha, not eliminate it..

words = ("one", "two", "red", "blue", "fish")

def join_strings(strings=words):
return ' '.join('%s %s' % (s, strings[-1]) for s in strings[:-1])

# Later:
words = open("gettysburg_address.txt", "r").read().split()
# Oops, the default argument of join_strings just changed.

+1. This is what convinces me that keeping references to keyword arguments is actually the right thing to do.

Perhaps I'm biased because I'm used to the mutable-argument behavior by now, but the gotcha described above seems to be much harder to track down and understand than any bugs caused by mutable arguments. With mutable arguments, at least you know the cause is in the function definition. If you initialized variables each time, you'd have to track the problem down across the entire namespace.

I, for one, would much rather follow the rule "don't use mutable arguments in function definitions" than "don't ever re-use the name of your variables.."

ML

 

[Steven D'Aprano]

def f(a, L=[]):
    L.append(a)
    return L

Every language has gotchas. This is one of python's.

One of our pre-interview screening questions for Python programmers at
Songza is about this. I haven't been keeping careful statistics, but
I'd guess only about 50% of the candidates get this right.

What exactly is the question? Because it's not always a bug to use a
mutable default, there are good uses for it:


def func(arg, cache={}):
...


Now you can pass your own cache as an argument, otherwise the function
will use the default. The default cache is mutable, and so will remember
values from one invocation to the next.

 

[Roy Smith]

def f(a, L=[]):
    L.append(a)
    return L

Every language has gotchas. This is one of python's.

One of our pre-interview screening questions for Python programmers at
Songza is about this. I haven't been keeping careful statistics, but
I'd guess only about 50% of the candidates get this right.

What exactly is the question? Because it's not always a bug to use a
mutable default, there are good uses for it:

It's a screening question; I'd rather not reveal all the details (it's
really annoying when you find your screening questions posted to stack
overflow). But, yes, I understand that there are good uses.