mutable default parameter problem [Prothon]

[Mark Hahn]

I like more as an example:

def foo(x):
... if not hasattr(foo,'list'):
... foo.list = []
... foo.list.append(x)
... print foo.list

In Prothon:

def foo(x):
print foo.list.append!(x)
foo.list = []

(Sorry. I couldn't resist bragging.)

About what?

Is there an echo in here?

 

[Christos TZOTZIOY Georgiou]

I like more as an example:

def foo(x):
... if not hasattr(foo,'list'):
... foo.list = []
... foo.list.append(x)
... print foo.list

In Prothon:

def foo(x):
print foo.list.append!(x)
foo.list = []

(Sorry. I couldn't resist bragging.)

About what?

Is there an echo in here?

Probably not. Given the python version:

def foo(x):
foo.list.append(x)
print foo.list
foo.list = []

Dave's question is legitimate. Are you bragging about the two lines
combined into one?

 

[Mark Hahn]

Dave's question is legitimate. Are you bragging about the two lines
combined into one?

As I replied before to the same exact question, I didn't know that Python
could do the function attribute assignment outside of the function (yes it
was stupid of me) and I was bragging about taking six lines to three. It
turns out that Prothon can only improve it by one line. I also said I was
sorry in my very first posting. If you want me to repeat this a third time
I will.

 

[=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=]

2) Evaluate the default expression once at each call time when the
default
value is needed. The default expression would be evaluated in the
context
of the function definition (like a closure).

I like Choice 2 because I've always wanted to do the following :

def func( x, y=2*x ):
do stuff...

ie. use default values for function parameters which depend on previous
function parameters.
This can be very practical at times :

before (suppose 'info' is a class which has "text" and "htmlclass" as
members):

def format_information( info, htmlclass = None ):
if htmlclass == None:
htmlclass = info.htmlclass
return "<p class=%s>%s</p>" % (htmlclass, info.text )

after:

def format_information( info, htmlclass = info.htmlcass ):
return "<p class=%s>%s</p>" % (htmlclass, info.text )


the intended use would be :
format_information( info )
format_information( info, 'red_text' ) overrides the html_class in info.


The former example could be simplified (below) but I still think the
second example using your choice 2 is more elegant.

def format_information( info, htmlclass = None ):
return "<p class=%s>%s</p>" % (htmlclass or info.htmlclass, info.text )

 

[=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=]

I'm new to python. To my eyes this is a pretty poor attempt to
have static variables. I've implemented in the past a few

static variables in python

def accumulate( x ):
accumulate.accumulator += x
return accumulate.accumulator

accumulate.accumulator = 0

 

[=?ISO-8859-1?Q?Gr=E9goire_Dooms?=]

Troy Melhase wrote:

Here's an idea: if it ain't broke, don't fix it.

Seriously, you see a "wart" and a "problem". I see a pleasant
side-effect of the documented semantics. True, new folks are
surprised by the behavior, but once it's understood, it becomes more
powerful.

All four of the Python gotcha's, wart's and regrets lists I have found
included this problem. It is not only a newbie's problem as I showed in my
posting.

How do you intend to account for code like this:

def F(a, b, cache={}):
try:
return cache[(a,b)]
except (IndexError, ):
value = cache[(a,b)] = do_some_long_calc(a,b)
return value

Or even this:

shared_cache = {}

def F(a, b, cache=shared_cache):
...

The first example is very unreadable and uncool in general. Your second
example will work just fine with our fix.

Of course you can argue that this is bad style,

Yes I (and many others) will.

but the counter
argument is just as strong: this is quite pythonic and quite
readable.

I disagree strongly. I would never be caught coding something like that and
I love Python dearly.

Python is a tool, and you decrease the utility of that tool when you
limit it's idioms.

So far you have only shown me an idiom that many say should not be used.
Show me one that everyone agrees is useful.

How much Python code would these different proposals break?

A lot. I ran this:

$ find /usr/lib/python2.3/ -name "*.py" -exec grep "def.*=\[\]" {}
\; | wc

And see 67 instances just in the standard library. Multiply that by
a factor of 1000, 10000 or more to reflect code in the field, and you
might start to understand the significance of changing the language
definition.

That count is not accurate. Fixing this will not break every use of [] as a
default formal param. Using [] in __init__ for example would break nothing.
I can think of many other cases where it is legal to use []. The only case
I can think of that would break would be the idiom we disagree on above. If
I am wrong, then show me other cases.

Right. I'd like to see how many of these 67 instances of mutable def
args actually mutates them. I hope it is 0. In this case the proposed
modification would break nothing at all.

But I wonder what the actual semantics of the second proposal are.

You say the default argument is evaluated in its declaration context at
each call. But is that evaluation context garanteed to be the same at
every call ?

#Ex:
x=0
def t(a,b,c=x+1):
# could I assert c==1 if t called with 2 args ?

def make_some_obj():
return []
def t(a,b,c=make_some_obj()):
# can I assert c==[] if t called with 2 args ?

I think every data coming from outside the expression (the variables in
the expression including the global variables accessed from a function)
should be preserved. Let aside the eventual mutable state stored in an
object:

#Ex:
class Inc:
def __init__(self):
self.x=0
def newval():
self.x += 1
return self.x
x=Inc()
def t(a,b,c=x.newval()):
# can I assert c will increase by 1 at each call with 2 args ?

If I also might make a general argument for the fix then let me continue.
Doing a late evaluation of the default expression makes the language more
dynamic, which fits the overall goal of making Prothon more dynamic. Using
prototypes instead of classes, dynamic var scoping, this fix, and many other
Prothon changes from Python all work towards that goal.

Dynamic var scoping fixed another Python gotcha which doesn't break
anything. Here are the two versions of code showing the problem and the
fix:

--- Python ---



... x = x + 1
... print x
...


UnboundLocalError: local variable 'x' referenced before assignment

--- Prothon ---

O>> x = 1
1
O>> def f():
... x = x + 1
... print x
...
O>> f()
2

Prothon's scoping rules are dynamic which means that x comes from outside
the function until the actual assignment happens. At that point x becomes a
local variable. This, along with the fact that vars are inherited from
ancestors along with methods, allow for some intuitive and simple var
initialization techniques.

So then
0>> f()
2
0>> x
1

Obviously it is the responsibility of the programmer to make sure that the
outer x has the proper initialization value for the local x. This can cause
a hiding-of-uninitialized-vars bug if the programmer uses the same names for
unrelated variables but it is worth the extra power and intuitiveness.

Nice, I had never heard about Prothon. I'll give it a look.

 

[=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=]

2) Evaluate the default expression once at each call time when the
default
value is needed. The default expression would be evaluated in the
context
of the function definition (like a closure).

I like Choice 2 because I've always wanted to do the following :

def func( x, y=2*x ):
do stuff...

ie. use default values for function parameters which depend on previous
function parameters.
This can be very practical at times :

before (suppose 'info' is a class which has "text" and "htmlclass" as
members):

def format_information( info, htmlclass = None ):
if htmlclass == None:
htmlclass = info.htmlclass
return "<p class=%s>%s</p>" % (htmlclass, info.text )

after:

def format_information( info, htmlclass = info.htmlcass ):
return "<p class=%s>%s</p>" % (htmlclass, info.text )


the intended use would be :
format_information( info )
format_information( info, 'red_text' ) overrides the html_class in info.


The former example could be simplified (below) but I still think the
second example using your choice 2 is more elegant.

def format_information( info, htmlclass = None ):
return "<p class=%s>%s</p>" % (htmlclass or info.htmlclass, info.text )

 

[=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=]

I'm new to python. To my eyes this is a pretty poor attempt to
have static variables. I've implemented in the past a few

static variables in python

def accumulate( x ):
accumulate.accumulator += x
return accumulate.accumulator

accumulate.accumulator = 0

 

[Mark Hahn]

But I wonder what the actual semantics of the second proposal are.

You say the default argument is evaluated in its declaration context
at each call. But is that evaluation context garanteed to be the same
at every call ?

#Ex:
x=0
def t(a,b,c=x+1):
# could I assert c==1 if t called with 2 args ?

No, because x could change value. If you wanted that then you would say
c=1. When you say c = x+1 you get x+1, no more, no less. You can code it
to get whatever you want.

def make_some_obj():
return []
def t(a,b,c=make_some_obj()):
# can I assert c==[] if t called with 2 args ?

This case would always give you [], unless you redefined make_some_obj
This is a dynamic language after all.

I think every data coming from outside the expression (the variables
in the expression including the global variables accessed from a
function) should be preserved. Let aside the eventual mutable state
stored in an object:

#Ex:
class Inc:
def __init__(self):
self.x=0
def newval():
self.x += 1
return self.x
x=Inc()
def t(a,b,c=x.newval()):
# can I assert c will increase by 1 at each call with 2 args ?

If you really want this then you should be asking for solution 3, which just
keeps the results of the expression evalutation at definition time and then
makes a copy at each call. You would be the only one asking for solution 3
so far. It appears that we are going with solution 2.

Nice, I had never heard about Prothon. I'll give it a look.

You can find it at http://prothon.org. Hang out on the Prothon mailing list
for a while. We are discussing some really fun stuff.

 

[Mark Hahn]

I like Choice 2 because I've always wanted to do the following :

def func( x, y=2*x ):

It looks like you will get your wish. The voting has been pretty much
unanimous for option 2.

 

[=?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?=]

Nice !
I wish Python had this, too.

Also, re-evaluating each time will allow one to use a global variable
whose value could change as a default parameter. Which looke suspicious.
Or something like that, which is GOOD :

class youpi( object ):
def mymethod( self, a=self.a, b=self.b ):
...

 

[Hung Jung Lu]

But python has static variables.

def another( x ):
y = getattr( another, 'static', 10 )
another.static = x
return y

print another(1), another(2), another(4)

What about the following case:

def f():
f.static = getattr(f, 'static', 0)
f.static += 1
return f.static

print f(), f(), f() # prints 1 2 3

As opposed to C++, you now have a line of code that is always executed
in subsequent calls, for no good reason. This is worse than:

def f(static=[0]):
static[0] += 1
return static[0]

in the sense that you have a wasteful call ("getattr") that doesn't do
anything productive in subsequent calls. (You could change that to
f.static = getattr(f, 'static', 0) + 1, but the "getattr" is surely
inefficient compared to f.static += 1, and internally likely incurs
some conditional statement at some level.)

Maybe one can do instead:

def f():
global f
def f():
f.static += 1
return f.static
f.static = 0 # initialization
return f()

print f(), f(), f() # prints 1 2 3

The advantage is there is no more wasteful statements in subsequent
calls. No "if" conditional statement. The above is of course a toy
example to illustrate the case of a function that needs to perform
something special the first time it is called. (I am well aware of the
outside assignment like:

def f():
f.static += 1
return f.static
f.static = 0

mentioned in this thread, but I am talking about something more
general. Notice that in the latter case, f.static=0 is done before f()
is called, which may not be what one wants. E.g.: if f() is never
called, this assignment is wasteful. Not a problem in this case, for
if complicated initialization is needed, like requiring a timestamp,
it may not be a good idea.)

In code refactoring, the equivalent is to replace conditional
statements by polymorphism. In terms of codeblocks, what I mean is
dynamic hook-on and hook-off of codeblocks. If the underlying language
is powerful enough, one should be able to achieve runtime
restructuring of code, without performance impact for subsequent
calls.

regards,

Hung Jung

 

[Mark Hahn]

In code refactoring, the equivalent is to replace conditional
statements by polymorphism. In terms of codeblocks, what I mean is
dynamic hook-on and hook-off of codeblocks. If the underlying language
is powerful enough, one should be able to achieve runtime
restructuring of code, without performance impact for subsequent
calls.

What about this Prothon code:

def f():
if 'static' not in f.attrs_:
f.static = 0
f.static += 1

tmp = f.static
def outer.f():
f.static += 1
return f.static
f.static = tmp

return f.static

print f() # 1
print f() # 2
print f() # 3

I'm sure Python can do this better so please don't barrage me with replies
saying so. I'm not going to brag this time

 

[Peter Hansen]

I also said I was
sorry in my very first posting. If you want me to repeat this a third time
I will.

It's likely that Christos' Usenet link (if that's how he gets
this group) is much slower than yours or mine... he may not
have received (or at least read) either of the other replies
by the time he sent his own...

Or, he might just want you to repeat it a third time. ;-)

-Peter

 

[Rob Williscroft]

Hung Jung Lu wrote in
in comp.lang.python:

What about the following case:

def f():
f.static = getattr(f, 'static', 0)
f.static += 1
return f.static

print f(), f(), f() # prints 1 2 3

Yep this would be the way I would instinctivly write it. However
Andrea suggested:

def f():
if not hasattr( f, 'static', 0 ):
f.static = 0
f.static += 1
return f.static

Which my primitive timing tests show to be faster.

As opposed to C++, you now have a line of code that is always executed
in subsequent calls, for no good reason. This is worse than:

Well C++ has `if ( __compiler_generated_bool__ )` that will always
be executed ( except in the case of POD's (char, int, double etc) ).

def f(static=[0]):
static[0] += 1
return static[0]

I don't know but surely the interpreter is doing some kind of
extra if/attribute lookup in there, its just faster than the
other versions.

I've timed both, and this was by far the fastest. I don't think
the problems is the `if` though. I don't know but I suspect this
version finds the local/paramiter 'static' with __slots__ like
performance, all other version's suffer from attribute lookup
problems.

in the sense that you have a wasteful call ("getattr") that doesn't do
anything productive in subsequent calls. (You could change that to
f.static = getattr(f, 'static', 0) + 1, but the "getattr" is surely
inefficient compared to f.static += 1, and internally likely incurs
some conditional statement at some level.)

Maybe one can do instead:

def f():
global f
def f():
f.static += 1
return f.static
f.static = 0 # initialization
return f()

print f(), f(), f() # prints 1 2 3

The advantage is there is no more wasteful statements in subsequent
calls. No "if" conditional statement.

Intresting, but it looses (in my timing tests) to the f(static=[0])
version, static[0] must be faster than f.static I guess.

The above is of course a toy
example to illustrate the case of a function that needs to perform
something special the first time it is called. (I am well aware of the
outside assignment like:

def f():
f.static += 1
return f.static
f.static = 0

mentioned in this thread, but I am talking about something more
general. Notice that in the latter case, f.static=0 is done before f()
is called, which may not be what one wants. E.g.: if f() is never
called, this assignment is wasteful. Not a problem in this case, for
if complicated initialization is needed, like requiring a timestamp,
it may not be a good idea.)

Indeed, the fast version is:

_f_static = 0
def f():
global _f_static
_f_static += 1
return _f_static

This is directly equivalent to
<c++>
static int f_static = 0;
int f() { return ++f_static; }
</c++>

C++ name hiding is with the static keyword, python name hiding
is with a leading underscore.

In code refactoring, the equivalent is to replace conditional
statements by polymorphism. In terms of codeblocks, what I mean is
dynamic hook-on and hook-off of codeblocks. If the underlying language
is powerful enough, one should be able to achieve runtime
restructuring of code, without performance impact for subsequent
calls.

Nice.

<c++>
static int f_static;
static int f_init();
static int f_run();
int (*f)() = f_init();

static int f_init()
{
f_static = /* dynamic value */ 0;
f = f_run;
return f();
}
static int f_run()
{
return ++f_static;
}
</c++>

In C++ the (performance) cost is visible and only f() and its callers
pay for it, in Python the the (performance) cost is invisible and
everybody pays for it.

In C++ I write:

int f()
{
static int var = 0;
return ++var;
}

And I let the compiler worry about the best way to implement it,
in Python I write:

class f( object ):
def __init__( self ):
self.var = 0;
def run( self ):
self.var += 1
return self.var

Though in a simple case (as all the examples have been) I
might write:

def f():
if not hasattr( f, 'static' ):
f.static = 0
f.static += 1
return f.static

Clarity (my clairty of purpose, as a programmer) wins in
both languages.

I won't be writing:

def f(static=[0]):
#etc

It simply isn't clear.

Rob.

 

[Christos TZOTZIOY Georgiou]

It's likely that Christos' Usenet link (if that's how he gets
this group) is much slower than yours or mine... he may not
have received (or at least read) either of the other replies
by the time he sent his own...

Or, he might just want you to repeat it a third time. ;-)

I am actually reading through Usenet... the newsserver link is fast, but
the arrival order of posts is not guaranteed, so I don't read messages
in the same order as people using the mail list. Often I post a reply
to a single message, only to find in the next synchronisation that
others did reply even earlier than me saying more or less the same
things. Other times I don't bother, saying "somebody else will reply to
this trivial question", and next day I find out that everybody else
thought the same as I did. That's Usenet (sigh).

 

[Mark Hahn]

Wait, so is the exclamation point required or not? IOW, say you have a class
like this:

class List(list):
def append(self, what):
list.append(self, what)
return self

a = List()
b = a.append(5)

So in the call to append, is it a.append!(5) or just a.append(5) ? If it's
the former, then does the compiler detect that it's required because the
function returns 'self' or is the determining factor something else?

Or, does the append method not really return anything, and the language
takes care of substituting in the object? (in which case, does that mean you
can override the return value by adding '!' - such that z=foo.bar!() ignores
the return value of bar and z references foo?)

As I said above: It's not that the exclamation mark that causes append to
return the sequence. The exclamation mark is always there and the sequence
is always returned. In Prothon (and Ruby and other languages before) the
exclamation mark is just part of the method name and is there to warn you
that in-place modification is happening.

Personally, I don't like the modify-in-place-and-return-the-object
'feature' - it's not needed _that_ often, but more importantly, it makes the
code harder to read (to me at least).

If you use the Prothon append!() exactly as you use the Python append() you
will get the exact same results. This is just an extra feature for those
that want it.

Guido avoided returning values from in-place modification functions because
of the confusion as to whether in-place mods were happening or not. We have
solved that confusion with the exclamation mark. Our code is very readable
because of this.

 

[Mark Hahn]

I am actually reading through Usenet... the newsserver link is fast,
but the arrival order of posts is not guaranteed, so I don't read
messages in the same order as people using the mail list. Often I
post a reply to a single message, only to find in the next
synchronisation that others did reply even earlier than me saying
more or less the same things. Other times I don't bother, saying
"somebody else will reply to this trivial question", and next day I
find out that everybody else thought the same as I did. That's
Usenet (sigh).

I'm sorry if I was rude. It is always when I make a stupid remark on a
public forum that I am asked to repeat it No one ever asks me to repeat
a witty gem. (Of course maybe that never occurs).

 

[Christos TZOTZIOY Georgiou]

No one ever asks me to repeat
a witty gem.

You can say that again

 

[Dave Brueck]

As I said above: It's not that the exclamation mark that causes append to
return the sequence. The exclamation mark is always there and the sequence
is always returned. In Prothon (and Ruby and other languages before) the
exclamation mark is just part of the method name and is there to warn you
that in-place modification is happening.

Ahh...so the method name is just _spelled_ with an exclamation point? IOW,
the ! is a token you can use at the end of an identifier, but it is not
actually used by the language itself - it's some sort of pseudo-syntax? I
think I understand now. But is it truly part of the name in that you are
required to include the ! when calling the method? (I'm still thinking of
the confusion I'd experience with something like w = x.y.z!() )

So if I want a reference to one of those methods I could end up doing

ref = obj.method! or ref! = obj.method!

and the program runs the same either way, it's just that in one case the
code is misleading?

If it's up to the programmer to remember to add it (meaning that it doesn't
cause an error to forget to use it), and if is really just part of the name,
then it's just a naming convention, right? Wouldn't you get the same result
by establishing the convention that e.g. method names ending in a single
underscore signify in-place modification (foo.append_() ) ? Seems like a
waste to reserve a symbol for something so rarely needed.

If you use the Prothon append!() exactly as you use the Python append() you
will get the exact same results. This is just an extra feature for those
that want it.

Guido avoided returning values from in-place modification functions because
of the confusion as to whether in-place mods were happening or not. We have
solved that confusion with the exclamation mark. Our code is very readable
because of this.

Clearly, readability is in the eye of the beholder.

-Dave