Should Python raise a warning for mutable default arguments?

[Steven D'Aprano]

Sometimes it seems that barely a day goes by without some newbie, or not-
so-newbie, getting confused by the behaviour of functions with mutable
default arguments. No sooner does one thread finally, and painfully, fade
away than another one starts up.

I suggest that Python should raise warnings.RuntimeWarning (or similar?)
when a function is defined with a default argument consisting of a list,
dict or set. (This is not meant as an exhaustive list of all possible
mutable types, but as the most common ones that I expect will trip up
newbies.) The warning should refer to the relevant FAQ or section in the
docs.

What do people think?

 

[Diez B. Roggisch]

Sometimes it seems that barely a day goes by without some newbie, or not-
so-newbie, getting confused by the behaviour of functions with mutable
default arguments. No sooner does one thread finally, and painfully, fade
away than another one starts up.

I suggest that Python should raise warnings.RuntimeWarning (or similar?)
when a function is defined with a default argument consisting of a list,
dict or set. (This is not meant as an exhaustive list of all possible
mutable types, but as the most common ones that I expect will trip up
newbies.) The warning should refer to the relevant FAQ or section in the
docs.

I just suggested a documentation enhancement in one of the plethora of
threads... so I'm certainly +1 for any enhancement in this area.

Diez

 

[Peter Otten]

-0 from me. I'd rather feature it more prominently in the tutorial, a
section "The five most common pitfalls" or something like that.

I don't see a chance for your proposal. How are you going to detect
mutable objects? Custom types can be mutable as well as immutable.

A check at compilation time for list literals would catch 90 % of the cases.
The warning would be targeted at newbies after all. It might still be a
source of confusion when they try to import someone else's code that uses
mutable defaults intentionally.

Peter

 

[bearophileHUGS]

DrScheme is an implementation of Scheme that is very newbie-friendly.
It has several limited sub-languages, etc.

So maybe a command line option can be added to Python3 ( -
newbie ? ) that just switches on similar warnings, to help newbies
(in schools, where there's a teacher that encourages to always use
that command line option) avoid some of the most common traps.

Bye,
bearophile

 

[Peter Otten]

DrScheme is an implementation of Scheme that is very newbie-friendly.
It has several limited sub-languages, etc.

So maybe a command line option can be added to Python3 ( -
newbie ? ) that just switches on similar warnings, to help newbies
(in schools, where there's a teacher that encourages to always use
that command line option) avoid some of the most common traps.

Or maybe bundle pychecker with idle?

$ cat tmp.py
def test(x, a=[]):
a.append(x)
return a

for i in range(5):
print test(i)

$ pychecker tmp.py
Processing tmp...
[0]
[0, 1]
[0, 1, 2]
[0, 1, 2, 3]
[0, 1, 2, 3, 4]

Warnings...

tmp.py:2: Modifying parameter (a) with a default value may have unexpected
consequences

Though it might be interesting to ask a newbie what he expects when warned
of "unexpected consequences"

Peter

 

[castironpi]

Sometimes it seems that barely a day goes by without some newbie, or not-
so-newbie, getting confused by the behaviour of functions with mutable
default arguments. No sooner does one thread finally, and painfully, fade
away than another one starts up.

I suggest that Python should raise warnings.RuntimeWarning (or similar?)
when a function is defined with a default argument consisting of a list,
dict or set. (This is not meant as an exhaustive list of all possible
mutable types, but as the most common ones that I expect will trip up
newbies.) The warning should refer to the relevant FAQ or section in the
docs.

What do people think?

I like the idea of calling it to your attention. +1.

If a warning, you should be able to silence it with an annotation,
decorator, or a module-level flag.

Or perhaps, require it be declared explicit, and make it an error.

def test(x, a=defmut([])):

Python raises an actual error unless default arguments are known
immutable or instances of 'defmut'.

 

[Emile van Sebille]

Sometimes it seems that barely a day goes by without some newbie, or not-
so-newbie, getting confused by the behaviour of functions with mutable
default arguments. No sooner does one thread finally, and painfully, fade
away than another one starts up.

I suggest that Python should raise warnings.RuntimeWarning (or similar?)
when a function is defined with a default argument consisting of a list,
dict or set. (This is not meant as an exhaustive list of all possible
mutable types, but as the most common ones that I expect will trip up
newbies.) The warning should refer to the relevant FAQ or section in the
docs.

What do people think?

-1

People that have worked through the tutorial, something everyone should
do when they're starting out, will find this explicitly discussed. See

http://docs.python.org/tut/node6.html#SECTION006710000000000000000

People that just skim the surface get stung -- sorry.

Emile

 

[MRAB]

I don't see a chance for your proposal. How are you going to detect
mutable objects? Custom types can be mutable as well as immutable.

Could there be a new special method __mutable__?

 

[Diez B. Roggisch]

-1

People that have worked through the tutorial, something everyone should
do when they're starting out, will find this explicitly discussed. See

http://docs.python.org/tut/node6.html#SECTION006710000000000000000

People that just skim the surface get stung -- sorry.

But obviously enough, it's not emphazized enough. Even if the
interpreter isn't touched, at least the docs should be.

Diez

 

[Emile van Sebille]

But obviously enough, it's not emphazized enough. Even if the
interpreter isn't touched, at least the docs should be.

Well, that might be why it's the docs the way it is, and the only place
in the docs that carries the "Important warning" label, and in bold at that.

Further, the tutorial is the first link on the python for programmers
page, and on the non-programmers page it's the last link, so presumably
any non-programmer continuing on is pointed to the next step.

.... so as to emphasized enough, how more?

Emile

 

[Emile van Sebille]

I'd suggest that at the
end of the tutorial, when people have a better general idea of how
Python works, there would be a Python Gotchas section.

Hmmm, OK -- mutable defaults, integer division, name mangling...

I'd think decimal precision is more a general problem than a python
problem, but still one that throws newbies...

Any other ideas for gotcha's (as opposed to faqs)?

Emile

 

[Steven D'Aprano]

I don't see a chance for your proposal. How are you going to detect
mutable objects? Custom types can be mutable as well as immutable.

It is not meant to catch all conceivable mutable types, just the three
most likely to surprise newbies.

The analogy is with the sum() built-in, which raises an exception if you
use it to add strings. (Personally, I think it should raise a warning,
not an exception, and allow people to write slow code if they want.)
Python doesn't attempt to detect every single type that leads to O(n**2)
behaviour in sum(), just the one most likely to trip up newbies.

But if people decide that it is important to warn on any conceivable
mutable type, the easy way is to approach the question from the other
direction. If the default value *isn't* a string, int, float, None or
frozenset, assume it's mutable.

Personally I don't think it's important to catch every case. After all,
it's just a warning, not an error.

 

[Steven D'Aprano]

Steven D'Aprano wrote:
-1

People that have worked through the tutorial, something everyone should
do when they're starting out,

I never worked through the Python tutorial. I bought a book. By the time
I even knew the tutorial existed, I had finished the book and spent six
months programming in Python and the tutorial was far too basic for me.
I'm sure there are thousands of other Python developers who have had
similar experiences. You can't expect every beginner's book on Python
programming to feature a discussion on mutable defaults.

Perhaps Python should print a warning when you start up: "If you haven't
worked through the tutorial, you are not allowed to ask questions about
surprising behaviour".

will find this explicitly discussed. See

http://docs.python.org/tut/node6.html#SECTION006710000000000000000

People that just skim the surface get stung -- sorry.

You assume that people will read the tutorial and then immediately trip
over a real life example. There's a lot of stuff in the tutorial and it
doesn't all stick the first time you read it. Just recently, perhaps a
week or so ago, we had a case of a fellow who had been programming in
Python for many years before he stumbled across this behaviour. (I forget
the name of the thread, but I'm sure you can find it.) It's not just
noobs who trip over this.

 

[Steven D'Aprano]

Hmmm, OK -- mutable defaults, integer division, name mangling...

I'd think decimal precision is more a general problem than a python
problem, but still one that throws newbies...

Any other ideas for gotcha's (as opposed to faqs)?

Augmented assignment: x ?= y is not always the same as x = x ? y.

Repeated string addition can be very slow. For that matter, so can list
addition.

Inserting at the beginning of lists is slow.

Everything about unicode is a Gotcha! *wink*

Raw strings are not designed for Windows paths, they're designed for
regexes. Consequently, you can't write the following:

r'C:\dir\'


list.sort() and list.reverse() return None.

sorted() returns a list, but reversed() returns an iterator.

urllib2.urlopen() will automatically detect the proxy in your environment
and use that. That's usually a feature, but sometimes it can be a gotcha.

urllib2 doesn't work well with some HTTPS proxies. This is, I believe, a
known bug, but until it is fixed, it can be a gotcha.

 

[George Sakkis]

Augmented assignment: x ?= y is not always the same as x = x ? y.

Repeated string addition can be very slow. For that matter, so can list
addition.

Inserting at the beginning of lists is slow.

Everything about unicode is a Gotcha! *wink*

Raw strings are not designed for Windows paths, they're designed for
regexes. Consequently, you can't write the following:

r'C:\dir\'

list.sort() and list.reverse() return None.

sorted() returns a list, but reversed() returns an iterator.

urllib2.urlopen() will automatically detect the proxy in your environment
and use that. That's usually a feature, but sometimes it can be a gotcha.

urllib2 doesn't work well with some HTTPS proxies. This is, I believe, a
known bug, but until it is fixed, it can be a gotcha.

My "favorite": comparisons between disparate types are allowed by
default. Thankfully fixed in 3.0.

George

 

[Carl Banks]

Sometimes it seems that barely a day goes by without some newbie, or not-
so-newbie, getting confused by the behaviour of functions with mutable
default arguments. No sooner does one thread finally, and painfully, fade
away than another one starts up.

I suggest that Python should raise warnings.RuntimeWarning (or similar?)
when a function is defined with a default argument consisting of a list,
dict or set. (This is not meant as an exhaustive list of all possible
mutable types, but as the most common ones that I expect will trip up
newbies.) The warning should refer to the relevant FAQ or section in the
docs.

What do people think?

-1

There's nothing questionable about using a mutable default argument,
as long as you don't mutate it. Python shouldn't raise a warning just
because something *might* be due to a misunderstanding.

I usefully use mutable default arguments all the time. Most commonly
in situations like this:

def f(data,substitutions = {}):
...
name = data['name']
obj.name = substitutions.get(name,name)
...


-0 on adding a warning that's disbaled by default.


Carl Banks

 

[Paddy]

DrScheme is an implementation of Scheme that is very newbie-friendly.
It has several limited sub-languages, etc.

So maybe a command line option can be added to Python3 ( -
newbie ? ) that just switches on similar warnings, to help newbies
(in schools, where there's a teacher that encourages to always use
that command line option) avoid some of the most common traps.

Or maybe bundle pychecker with idle?

$ cat tmp.py
def test(x, a=[]):
    a.append(x)
    return a

for i in range(5):
    print test(i)

$ pychecker tmp.py
Processing tmp...
[0]
[0, 1]
[0, 1, 2]
[0, 1, 2, 3]
[0, 1, 2, 3, 4]

Warnings...

tmp.py:2: Modifying parameter (a) with a default value may have unexpected
consequences

Though it might be interesting to ask a newbie what he expects when warned
of "unexpected consequences"

Peter

+1 on this.

It seems an obvious think to add to a lint-like tool rather than
burdening core Python.

- Paddy.

 

[Steven D'Aprano]

-1

There's nothing questionable about using a mutable default argument, as
long as you don't mutate it.

There's nothing questionable about using a mutable default argument, so
long as you know what behaviour to expect. I too use that behaviour, I
like that behaviour, and I'm tired of people who want it "fixed".

Nevertheless, it is surprising to many people. My aim is to make it a
little less surprising.

Python shouldn't raise a warning just
because something *might* be due to a misunderstanding.

That's one opinion.

As I've eluded to in an early post, I don't believe Python should refuse
to perform an operation just because it might be slow. Nevertheless,
that's precisely what the sum() function does. I'm suggesting a warning
rather than an exception, but other than that, I suggest that there's
precedence to what I am suggesting.

 

[BJörn Lindqvist]

Or maybe bundle pychecker with idle?

I think that is an excellent idea! In fact why can't pychecker be
included in the standard distribution? I'd love it if compilation was
done with pychecker checking by default. Python could definitely use a
-Wall mode.

 

[Carl Banks]

There's nothing questionable about using a mutable default argument, so
long as you know what behaviour to expect. I too use that behaviour, I
like that behaviour, and I'm tired of people who want it "fixed".

Nevertheless, it is surprising to many people. My aim is to make it a
little less surprising.


That's one opinion.

1. When you print spurious warnings, the overall effectiveness of the
warning system is diminished. People start to ignore them, either by
disabling them or by mentally tuning out. This in turn makes people
less likely to notice if a real warning is printed.

When you print a warning, you better be %99.9 sure that it's something
worth warning about, otherwise you are doing more harm than good.

(Story time: I once worked on a system that displayed warnings to jet
fighter pilots. Our requirements were not to show the pilot a warning
unless the airplane actually tries something and fails, even if the
computer is absolutely sure that it would fail. Ex: if computer knows
for sure the engine can't produce more than (say) 50% rated thrust,
the pilot does not get a warning unless he actually requests more than
50% thrust. The reason is for this, according to the senior engineers
on the team, was that pilots would start to ignore the warning lights
REALLY FAST.)


2. It's rude to be presumptuous, which is what the compiler would be
if it printed this warning.


Carl Banks