Relax Syntax for Augmented Arithmetic?

[andrew cooke]

Context - http://docs.python.org/3.0/reference/datamodel.html?highlight=data
model#object.__iadd__

Just a suggestion I thought I'd throw out... There's a restriction in
the language implementation on exactly what can go the left of an
augmented arithmetic expression.

For example:
is ok, but:.... def __init__():
.... self.a = 3
.... def __ipow__(self, x):
.... self.a **= x
.... File "<stdin>", line 1
SyntaxError: illegal expression for augmented assignment

Now unless I've done something stupid above (always a possibility )
the implementation seems a bit strict (is it really a *syntax* error?
- I am not sure exactly what the restriction is).

This may seems like a small issue, but operators can really help with
making embedded DSLs in Python - they give quite a bit of "wiggle
room" to invent a syntax that is compact and intuitive. The
restriction above cuts into that (OK, so it's still a small
issue... )

Cheers,
Andrew

 

[andrew cooke]

Improved link - http://docs.python.org/3.0/reference/datamodel.html#object.__iadd__

 

[Chris Rebert]

Context - http://docs.python.org/3.0/reference/datamodel.html?highlight=data
model#object.__iadd__

Just a suggestion I thought I'd throw out... There's a restriction in
the language implementation on exactly what can go the left of an
augmented arithmetic expression.

For example:

is ok, but:
... def __init__():
... self.a = 3
... def __ipow__(self, x):
... self.a **= x
...
File "<stdin>", line 1
SyntaxError: illegal expression for augmented assignment

Now unless I've done something stupid above (always a possibility )
the implementation seems a bit strict (is it really a *syntax* error?
- I am not sure exactly what the restriction is).

IIRC, you can only assign to:
- variables (x = ...)
- attributes (x.y = ...)
- elements (x[y] = ...)

Anything else doesn't make sense to assign to because it's not a
"storage box" so to speak. There's no way to work out what is meant.
In your case, you're assigning to a *value*, specifically a new
instance of the Foo class, which is nonsensical; instead of a "box",
you're trying to assign to a "value", something that gets stored in
boxes. By comparison, '2 = 5' and '[1,2] = 7' would seem to have some
sort of meaning under your system, which IMHO seems preposterous.
Now true, you are using augmented assignment, which in certain cases
is translated to a method call, but in principle the augmented
assignment (e.g. x += y) should have roughly the same effect as the
non-augmented equivalent (x = x + y), and the fact that a method call
is involved is merely an implementation detail of sorts.
Therefore, Python requires you to rewrite the code in some other way
that makes your intentions more clear. For instance, why not use the
<< operator instead?

Cheers,
Chris

 

[andrew cooke]

Therefore, Python requires you to rewrite the code in some other way

that makes your intentions more clear. For instance, why not use the
<< operator instead?

Right, but you're guessing what the context is. Within a DSL it often
makes a lot of sense to use operators for reasons that weren't
originally intended.

You even make the same case yourself indirectly. The same argument
you make could be made to say that << should only operator on values
that can be shifted. Now thankfully there is no way to test for that,
so there is no restriction and, consequently, it is now widely
accepted that no-one (even people arguing the case for constraints!)
think it odd to use << for something other than its initial use.

Obviously this kind of discussion has gone on since languages were
first invented - it's the "how much rope" argument. So rather than
continue down that road I would just like to say that this feels like
an inconsistency. The other operators are *not* as restricted and
this is making my life harder.

It may sound crazy, but this may force me to use * and ** instead (the
context is a language feature related to *args and **kargs, so the *
and ** help convey the meaning). And they have a much much stronger
meaning to users, which will make my DSL harder to understand. So in
this case a blunt knife is making life harder.

Andrew

 

[Chris Rebert]

Right, but you're guessing what the context is. Within a DSL it often
makes a lot of sense to use operators for reasons that weren't
originally intended.

You even make the same case yourself indirectly. The same argument
you make could be made to say that << should only operator on values
that can be shifted. Now thankfully there is no way to test for that,
so there is no restriction and, consequently, it is now widely
accepted that no-one (even people arguing the case for constraints!)
think it odd to use << for something other than its initial use.

Obviously this kind of discussion has gone on since languages were
first invented - it's the "how much rope" argument. So rather than
continue down that road I would just like to say that this feels like
an inconsistency. The other operators are *not* as restricted and
this is making my life harder.

Indeed. Python happens to in this case draw the line at using the
augmented assignment operators for non-assignment. I personally see
this as reasonable because the = symbol has a consistent meaning in
Python (assignment) whereas the other plain operators, as you bring
up, consistently have no predetermined meaning; but I do agree that it
is in a sense an arbitrary restriction, like many programming language
design choices. However, Python was not explicitly designed for
creating DSLs, so it's kinda odd to complain about something Python
never claimed to support in the first place (although I do favor the
DSL in general-purpose-PL paradigm).

It may sound crazy, but this may force me to use * and ** instead (the
context is a language feature related to *args and **kargs, so the *
and ** help convey the meaning). And they have a much much stronger
meaning to users, which will make my DSL harder to understand. So in
this case a blunt knife is making life harder.

Perhaps if you explained your particular predicament in more depth,
someone might be able to offer a workable suggestion.

Cheers,
Chris

 

[andrew cooke]

Not sure if you were saying this, but the underlying technical reason
for this issue is that they are treated as assignment rather than
operators in the language spec -
http://docs.python.org/3.0/reference/simple_stmts.html#augmented-assignment-statements

I think this explains why they are not listed in the operator
precedence table http://docs.python.org/3.0/reference/expressions.html#summary

I think that's unfortunate in a language with mutable objects, but it
makes the decision seem much less arbitrary...

Cheers,
Andrew

 

[andrew cooke]

Indeed. Python happens to in this case draw the line at using the
augmented assignment operators for non-assignment. I personally see
this as reasonable because the = symbol has a consistent meaning in
Python (assignment) whereas the other plain operators, as you bring
up, consistently have no predetermined meaning;

my argument was that *= is not treated as = and *, but as a completely
new operator (the docs even say that the implementation need not
return self which suggests some pretty extreme semantics were
envisaged). however, as i've just commented elsewhere, this
commitment to operators was only half-baked because they are parsed as
assignments.

anyway, to reply to your comment - *= is not predetermined. it is
determined by __imul__ which is user-definable.

but I do agree that it
is in a sense an arbitrary restriction, like many programming language
design choices. However, Python was not explicitly designed for
creating DSLs,

python is a general programming language. as far as i can make any
sense at all of your argument it seems to be "you are asking for
change, but this is not how the current system works". to which the
obvious answer is: if it did work that way i wouldn't be asking for
change.

andrew

 

[Marc 'BlackJack' Rintsch]

my argument was that *= is not treated as = and *, but as a completely
new operator (the docs even say that the implementation need not return
self which suggests some pretty extreme semantics were envisaged).

What do you mean by "suggests … extreme semantics"? Most natural thing
is to use numbers and there you *have* to be able to return something
different than `self` to get anything useful. For instance:

n *= 3

with `n` bound to a number different from zero can't return `self` from
`__imul__`.

Ciao,
Marc 'BlackJack' Rintsch

 

[andrew cooke]

What do you mean by "suggests … extreme semantics"?  Most natural thing
is to use numbers and there you *have* to be able to return something
different than `self` to get anything useful.  For instance:

 n *= 3

with `n` bound to a number different from zero can't return `self` from
`__imul__`.

in your example, n is not a number, it is a mutable variable, and its
value changes.

when n is an instance implementing __imul__ the natural analogue is
that the internal state of the instance changes.

either i have misundertstood you, or you have misunderstood __imul__,
or you are treating = as equality, or maybe you are thinking of a pure
language that creates new instances? python is impure.

anyway, my original request is moot. i was assuming that this was a
"capricious" restriction. in fact it's related to what i thought were
operators actually being assignments, and so no change is possible
(until python 4.0 when guido will finally see the light and move to s-
expressions, at which point everyone will stop using the language ;o)

andrew

 

[andrew cooke]

either i have misundertstood you

ah, i see your point. sorry, andrew

 

[andrew cooke]

http://bugs.python.org/issue4986

Sorry for the noise,
Andrew

 

[Aaron Brady]

in your example, n is not a number, it is a mutable variable, and its
value changes.

when n is an instance implementing __imul__ the natural analogue is
that the internal state of the instance changes.

either i have misundertstood you, or you have misunderstood __imul__,
or you are treating = as equality, or maybe you are thinking of a pure
language that creates new instances?  python is impure.

anyway, my original request is moot.  i was assuming that this was a
"capricious" restriction.  in fact it's related to what i thought were
operators actually being assignments, and so no change is possible
(until python 4.0 when guido will finally see the light and move to s-
expressions, at which point everyone will stop using the language ;o)

andrew

Not sure if this ties in, but:

['a'].__imul__(2)

['a', 'a']

 

[Terry Reedy]

Context - http://docs.python.org/3.0/reference/datamodel.html?highlight=data
model#object.__iadd__

Just a suggestion I thought I'd throw out... There's a restriction in
the language implementation on exactly what can go the left of an
augmented arithmetic expression.

For example:

is ok, but:
... def __init__():
... self.a = 3
... def __ipow__(self, x):
... self.a **= x
...

Calls return objects and therefore cannot be the target of an
assignment, augmented or otherwise. The target of an assignment is a
name or collection slot, both of which are grammatical constructs, not
objects.

 

[Steven D'Aprano]

Calls return objects and therefore cannot be the target of an
assignment, augmented or otherwise. The target of an assignment is a
name or collection slot, both of which are grammatical constructs, not
objects.

There's a practical reason too. You create a new Foo instance, mutate it
with the augmented assignment operator, and then a tenth of a millisecond
later the garbage collector throws it away because it has a reference
count of zero.

 

[Mark Wooding]

There's a practical reason too. You create a new Foo instance, mutate
it with the augmented assignment operator, and then a tenth of a
millisecond later the garbage collector throws it away because it has
a reference count of zero.

Only in this specific example. A function can easily return a
well-known object, which it's sensible to mutate.

Besides,

class mumble (object): pass
mumble().foo = 1

is accepted without fuss, and is just as useless.

I think I see the confusion here.

* Python assignment (`=') is fairly simple. The left-hand side is
analyzed syntactically: if it's a plain name then the variable it
denotes is modified; otherwise an appropriate method is invoked to
mutate some object.

* Python augmented-assignment (`+=', for example) is inconsistent.
Depending on what type of object the left-hand side evaluates to, it
may /either/ mutate that object, /or/ assign a new value to the
expression.

What do I mean? Well, consider this function.

def assg(x, y):
x = y

Under no circumstances does calling this function have any effect (other
than wasting time and memory). But:

def aug(x, y):
x += y

Calling this function might or might not have an observable effect,
depending on the type of x. For example,

x = 5
aug(x, 3)

is useless, but

x = [1, 2, 3]
aug(x, [4])

is not.

The `aug' function can be used to bypass the syntactic restriction on
augmented assignment, where it makes sense:

aug(func(), 17)

is always syntactically valid, and may or may not be useless depending
on the type of thing returned by `func'.

The Python language refuses to let the programmer write something which
is (a) possibly meaningful, and (b) possibly useful because augmented
assignment inherits the syntactic restriction of simple assignment that
the left-hand side expression designate a `place' -- i.e., one of the
things that there's a rule for assigning to, e.g., VAR, EXPR[INDEX],
EXPR.ID -- because it /might/ need to perform such an assignment, though
it might not.

My personal view is that augmented-assignment operators which work by
mutation rather than assignment (e.g., `+=' on lists, rather than `+='
on numbers) are one of Python's least pleasant warts. But if they're
going to exist then I think

list() += [1]

ought to be valid syntax, since semantically it's actually clear what it
should do (namely, construct a fresh empty list, append a `1' to it, and
then throw the whole thing away). Of course,

tuple() += 1,

is still meaningless, and ought to be an error. Of course, this removes
a static error-check, but if we were petty about getting all our errors
at compile time we wouldn't be writing in Python in the first place.

-- [mdw]

 

[Aahz]

* Python augmented-assignment (`+=', for example) is inconsistent.
Depending on what type of object the left-hand side evaluates to, it
may /either/ mutate that object, /or/ assign a new value to the
expression.

Actually, that is not correct. The augmented assignment always binds a
new value to the name; the gotcha is that with a mutable object, the
object returns ``self`` from the augmented assignment method rather than
creating a new object and returning that. IOW, the smarts are always
with the object, not with the augmented assignment bytecode.

The best way to illustrate this:

a = (1, ['foo'], 'xyzzy')
a[1].append('bar')
a (1, ['foo', 'bar'], 'xyzzy')
a[1] = 9

Traceback (most recent call last):

a (1, ['foo', 'bar'], 'xyzzy')
a[1] += ['spam']

Traceback (most recent call last):
(1, ['foo', 'bar', 'spam'], 'xyzzy')

 

[Mark Wooding]

Actually, that is not correct.

You're right, evidently.

[snip]

a (1, ['foo', 'bar'], 'xyzzy')
a[1] += ['spam']

Traceback (most recent call last):
(1, ['foo', 'bar', 'spam'], 'xyzzy')

Ugh!

-- [mdw]

 

[Steve Holden]

* Python augmented-assignment (`+=', for example) is inconsistent.
Depending on what type of object the left-hand side evaluates to, it
may /either/ mutate that object, /or/ assign a new value to the
expression.

Actually, that is not correct. The augmented assignment always binds a
new value to the name; the gotcha is that with a mutable object, the
object returns ``self`` from the augmented assignment method rather than
creating a new object and returning that. IOW, the smarts are always
with the object, not with the augmented assignment bytecode.

The best way to illustrate this:

a = (1, ['foo'], 'xyzzy')
a[1].append('bar')
a (1, ['foo', 'bar'], 'xyzzy')
a[1] = 9

Traceback (most recent call last):

a (1, ['foo', 'bar'], 'xyzzy')
a[1] += ['spam']

Traceback (most recent call last):
(1, ['foo', 'bar', 'spam'], 'xyzzy')

I understand what you are saying, but if the id() associated with a name
doesn't change after augmented assignment it seems a little wrong-headed
to argue that "the augmented assignment always binds a new value to the
name".

What you are actually saying is that it's up to the method that
implements the augmented assignment whether the same (mutated) object or
a different one is returned, right? And that the left-hand side of the
assignment is always bound to the result of that method.

regards
Steve

 

[Aahz]

I understand what you are saying, but if the id() associated with a name
doesn't change after augmented assignment it seems a little wrong-headed
to argue that "the augmented assignment always binds a new value to the
name".

What you are actually saying is that it's up to the method that
implements the augmented assignment whether the same (mutated) object or
a different one is returned, right? And that the left-hand side of the
assignment is always bound to the result of that method.

That's overall more correct, but I wanted to emphasize that there is
*always* a binding operation being performed. Whether what gets bound to
the target is a new object or an existing object is up to the augmented
assignment method.

 

[Steve Holden]

That's overall more correct, but I wanted to emphasize that there is
*always* a binding operation being performed. Whether what gets bound to
the target is a new object or an existing object is up to the augmented
assignment method.

<nods>

Yes, we're on the same page. Maybe I was being too pedantic ...

regards
Steve