Some syntactic sugar proposals

[Dmitry Groshev]

Here are some proposals. They are quite useful at my opinion and I'm
interested for suggestions. It's all about some common patterns.
First of all: how many times do you write something like
t = foo()
t = t if pred(t) else default_value
? Of course we can write it as
t = foo() if pred(foo()) else default_value
but here we have 2 foo() calls instead of one. Why can't we write just
something like this:
t = foo() if pred(it) else default_value
where "it" means "foo() value"?
Second, I saw a lot of questions about using dot notation for a
"object-like" dictionaries and a lot of solutions like this:
class dotdict(dict):
def __getattr__(self, attr):
return self.get(attr, None)
__setattr__= dict.__setitem__
__delattr__= dict.__delitem__
why there isn't something like this in a standart library?
And the third. The more I use python the more I see how "natural" it
can be. By "natural" I mean the statements like this:
[x.strip() for x in reversed(foo)]
which looks almost like a natural language. But there is some
pitfalls:
if x in range(a, b): #wrong!
it feels so natural to check it that way, but we have to write
if a <= x <= b
I understand that it's not a big deal, but it would be awesome to have
some optimisations - it's clearly possible to detect things like that
"wrong" one and fix it in a bytecode.

x in range optimisation
dot dict access
foo() if foo() else bar()

 

[Dmitry Groshev]

Here are some proposals. They are quite useful at my opinion and I'm
interested for suggestions. It's all about some common patterns.
First of all: how many times do you write something like
    t = foo()
    t = t if pred(t) else default_value
? Of course we can write it as
    t = foo() if pred(foo()) else default_value
but here we have 2 foo() calls instead of one. Why can't we write just
something like this:
    t = foo() if pred(it) else default_value
where "it" means "foo() value"?
Second, I saw a lot of questions about using dot notation for a
"object-like" dictionaries and a lot of solutions like this:
    class dotdict(dict):
        def __getattr__(self, attr):
            return self.get(attr, None)
        __setattr__= dict.__setitem__
        __delattr__= dict.__delitem__
why there isn't something like this in a standart library?
And the third. The more I use python the more I see how "natural" it
can be. By "natural" I mean the statements like this:
    [x.strip() for x in reversed(foo)]
which looks almost like a natural language. But there is some
pitfalls:
    if x in range(a, b): #wrong!
it feels so natural to check it that way, but we have to write
    if a <= x <= b
I understand that it's not a big deal, but it would be awesome to have
some optimisations - it's clearly possible to detect things like that
"wrong" one and fix it in a bytecode.

x in range optimisation
dot dict access
foo() if foo() else bar()

Oh, I'm sorry. I forgot to delete my little notes at the bottom of
message.

 

[Chris Rebert]

Here are some proposals. They are quite useful at my opinion and I'm
interested for suggestions. It's all about some common patterns.

Second, I saw a lot of questions about using dot notation for a
"object-like" dictionaries and a lot of solutions like this:
   class dotdict(dict):
       def __getattr__(self, attr):
           return self.get(attr, None)
       __setattr__= dict.__setitem__
       __delattr__= dict.__delitem__
why there isn't something like this in a standart library?

There is:
http://docs.python.org/library/collections.html#collections.namedtuple

The "bunch" recipe is also fairly well-known; I suppose one could
argue whether it's std-lib-worthy:
http://code.activestate.com/recipes/52308-the-simple-but-handy-collector-of-a-bunch-of-named/

Cheers,
Chris

 

[Dmitry Groshev]

There is:http://docs.python.org/library/collections.html#collections.namedtuple

The "bunch" recipe is also fairly well-known; I suppose one could
argue whether it's std-lib-worthy:http://code.activestate.com/recipes/52308-the-simple-but-handy-collec...

Cheers,
Chris

namedtuple is not a "drop-in" replacement like this "dotdict" thing -
you first need to create a new namedtuple instance. As for me it's a
bit too complicated.

 

[alex23]

First of all: how many times do you write something like
    t = foo()
    t = t if pred(t) else default_value
? Of course we can write it as
    t = foo() if pred(foo()) else default_value
but here we have 2 foo() calls instead of one. Why can't we write just
something like this:
    t = foo() if pred(it) else default_value
where "it" means "foo() value"?

Could you provide an actual use case for this. This seems weird to me:
you're creating an object, testing the object, then possibly throwing
it away and using a default instead. Are you sure you can't
restructure your code as such:

Second, I saw a lot of questions about using dot notation for a
"object-like" dictionaries and a lot of solutions like this:
    class dotdict(dict):
        def __getattr__(self, attr):
            return self.get(attr, None)
        __setattr__= dict.__setitem__
        __delattr__= dict.__delitem__
why there isn't something like this in a standart library?

Personally, I like keeping object attribute references separate from
dictionary item references. If you're someone who doesn't mind
muddying that distinction, then - as you've discovered - it's a simple
addition to your own code.

    if x in range(a, b): #wrong!

Only in Python 3.x, it's perfectly valid in Python 2.x. To achieve the
same in Python 3.x, try:

if x in list(range(a, b,)): # BUT SEE MY COMMENT BELOW

it feels so natural to check it that way, but we have to write
    if a <= x <= b
I understand that it's not a big deal, but it would be awesome to have
some optimisations - it's clearly possible to detect things like that
"wrong" one and fix it in a bytecode.

This seems more like a pessimisation to me: your range version
constructs a list just to do a single container check. That's a _lot_
more cumbersome than two simple comparisons chained together.

 

[alex23]

   t = foo(x) if <text on x> else default

This should read 'test' instead of 'text', sorry.

 

[Dmitry Groshev]

Could you provide an actual use case for this. This seems weird to me:
you're creating an object, testing the object, then possibly throwing
it away and using a default instead. Are you sure you can't
restructure your code as such:

   t = foo(x) if <text on x> else default

Sure. Let's pretend you have some string foo and compiled regular
expression bar.
Naive code:
t = bar.findall(foo)
if len(t) < 3:
t = []
Code with proposed syntactic sugar:
t = bar.findall(foo) if len(it) > 2 else []

Personally, I like keeping object attribute references separate from
dictionary item references.

Your Python doesn't - dot notation is just a sugar for __dict__ lookup
with default metaclass.

This seems more like a pessimisation to me: your range version
constructs a list just to do a single container check. That's a _lot_
more cumbersome than two simple comparisons chained together.

By "wrong" I meant exactly this. I told about "compiler" optimisation
of statements like this so it would not construct a list.

 

[alex23]

Your Python doesn't - dot notation is just a sugar for __dict__ lookup
with default metaclass.

That's a gross oversimplification that tends towards wrong:
.... def __init__(self):
.... self._x = None
.... @property
.... def x(self): return self._x
.... @x.setter
.... def x(self, val): self._x = val
....

c = C()
c.x = 1
c.x 1
c.__dict__['x']

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'x'

But my concern has _nothing_ to do with the implementation detail of
how objects hold attributes, it's solely over the clarity that comes
from being able to visually tell that something is an object vs a
dictionary.

By "wrong" I meant exactly this. I told about "compiler" optimisation
of statements like this so it would not construct a list.

So you want this:

if x in range(1,10):

....to effectively emit the same bytecode as a chained comparison while
this:

for x in range(1,10):

....produces a list/generator?

Never going to happen. "Special cases aren't special enough to break
the rules." The standard form for chained comparisons can handle far
more complex expressions which your 'in' version could not: 0 <= min
<= max <= 100

 

[Dmitry Groshev]

Your Python doesn't - dot notation is just a sugar for __dict__ lookup
with default metaclass.

That's a gross oversimplification that tends towards wrong:

...   def __init__(self):
...     self._x = None
...   @property
...   def x(self): return self._x
...   @x.setter
...   def x(self, val): self._x = val
...>>> c = C()

c.x = 1
c.x 1
c.__dict__['x']

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'x'

But my concern has _nothing_ to do with the implementation detail of
how objects hold attributes, it's solely over the clarity that comes
from being able to visually tell that something is an object vs a
dictionary.

Oh, now I understand you. But this "dotdict" (or "bunch") things don't
break anything. You still need to use it explicitly and it is very
useful if you need to serialize some JSON data about some entities.
s = """[{"name": "Jhon Doe", "age": "12"}, {"name": "Alice",
"age": "23"}]"""
t = map(dotdict, json.loads(s))
t[0] #{'age': '12', 'name': 'Jhon Doe'}
t[0].age #'12'
Of course you can do this with namedtuple, but in fact this isn't a
tuple at all. It's a list of entities.

So you want this:

  if x in range(1,10):

...to effectively emit the same bytecode as a chained comparison while
this:

  for x in range(1,10):

...produces a list/generator?

Never going to happen. "Special cases aren't special enough to break
the rules." The standard form for chained comparisons can handle far
more complex expressions which your 'in' version could not: 0 <= min
<= max <= 100

I know about chained comparisons, thanks. It's not about them. It's
about bytecode optimisation. But maybe you are right about "Special
cases aren't special enough to break the rules". I kinda forgot that

 

[Hrvoje Niksic]

which looks almost like a natural language. But there is some
pitfalls:
if x in range(a, b): #wrong!
it feels so natural to check it that way, but we have to write
if a <= x <= b

For the record, you have to write:

if a <= x < b:

Ranges are open on the ending side.

 

[Mark Wooding]

First of all: how many times do you write something like
t = foo()
t = t if pred(t) else default_value
? Of course we can write it as
t = foo() if pred(foo()) else default_value
but here we have 2 foo() calls instead of one. Why can't we write just
something like this:
t = foo() if pred(it) else default_value
where "it" means "foo() value"?

How about

t = (lambda y: y if pred(y) else default_value)(foo(x))

You could even package the lambda into a named function if you get bored
of typing or your aesthetic senses are offended.

And the third. The more I use python the more I see how "natural" it
can be. By "natural" I mean the statements like this:
[x.strip() for x in reversed(foo)]
which looks almost like a natural language. But there is some
pitfalls:
if x in range(a, b): #wrong!
it feels so natural to check it that way, but we have to write
if a <= x <= b

This, I think, is your error. The test `x in range(a, b)' means the
same as `a <= x < b' (only in Python 2 it builds a list and then throws
it away again). Such half-open intervals turn out to be what you want
most of the time, and I think you'll avoid many bugs if you embrace them
rather than trying to cling to the fully-closed intervals above.

Python very definitely made the right decision to use half-open
intervals pervasively, e.g., for `rangeand 'in sequence slicing. It's a
bad idea to fight against it.

Advantages of half-open intervals [a, b):

* The number of elements is exactly b - a; closed intervals contain an
extra element which is often forgotten.

* They compose and split nicely: if a <= c <= b then [a, b) is exactly
the disjoint union of [a, c) and [c, b).

* Sums over these intervals are often better behaved; indeed, there's
a rather pretty analogy between sums on half-open intervals and
definite integrals which is lost if you use closed intervals. (The
above two observations are special cases of this one.) See Concrete
Mathematics (Graham, Knuth, Patashnik) for more on this.

* It's easy to express an empty interval. You can't do that if you
work entirely with closed intervals, but empty sets are an important
boundary case and it's annoying to have to handle them separately.

-- [mdw]

 

[Tim Chase]

x in range optimisation

I've often thought this would make a nice O(1)-test lookup on an
xrange() generator. I don't have strong use-cases for it, but a
bit of well-tested code in the standard library would save me
from doing the math (along with its potential fenceposting and
sign errors) the couple times I've reached for it. Using the

x in list(xrange(...)) # or "range()" depending your version

ends up with an O(n) lookup, though I suppose one could create a
set() for O(1) lookups, but that still requires O(N) storage
(where a mathematical lookup would involve O(1) storage *and* time)

foo() if foo() else bar()

This is usually indicative that you need to cache the
object...something like

etcc = expensive_to_calculate_constructor()
result = foo(etcc) if test(etcc) else bar()

That way, you cache the potentially-expensive calculation,
indicate to other readers-of-your-code that it's
potentially-expensive, and enforce the use of that cache to
ensure that you don't duplicate the expensive op.

-tkc

 

[Terry Reedy]

Here are some proposals. They are quite useful at my opinion and I'm
interested for suggestions. It's all about some common patterns.
First of all: how many times do you write something like
t = foo()
t = t if pred(t) else default_value

Never. t=t unbinds and rebinds 't' to the same object. A waste.
Only rebind if needed.
if not pred(t): t = default_value

? Of course we can write it as
t = foo() if pred(foo()) else default_value
but here we have 2 foo() calls instead of one. Why can't we write just
something like this:
t = foo() if pred(it) else default_value
where "it" means "foo() value"?

Too magical.

I agree with most other comments.

 

[Brian Blais]

if x in range(a, b): #wrong!
it feels so natural to check it that way, but we have to write
if a <= x <= b
I understand that it's not a big deal, but it would be awesome to have
some optimisations - it's clearly possible to detect things like that
"wrong" one and fix it in a bytecode.

I don't think anyone has pointed this out, but these are *not* the same thing. observe:


x=3
if x in range(1,10):
print "yay!"

if 1<=x<10:
print "yay too!"

x=3.2
if x in range(1,10):
print "yay!"

if 1<=x<10:
print "yay too!"

output:

yay!
yay too!
yay too!


bb

 

[John Nagle]

Personally, I like keeping object attribute references separate from
dictionary item references.

Right. This isn't JavaScript. If you need a "dict", use a
"dict". Don't use attributes as named storage. It leads to
problems. Functions and various built-in objects are in the
attribute namespace, and this can lead to name clashes.
Maybe security holes, if the attribute keys come from
external input. There are also some restrictions on
the syntax of attribute names, restrictions "dict"
does not have.

Remember, you can inherit from "dict", which
allows you to write

obj['abc']

which is almost as short as

obj.abc

but safer.

John Nagle

 

[Steven D'Aprano]

Here are some proposals. They are quite useful at my opinion and I'm
interested for suggestions. It's all about some common patterns. First
of all: how many times do you write something like
t = foo()
t = t if pred(t) else default_value
?

Hardly ever. Not often enough to need special syntax for it.


Of course we can write it as

t = foo() if pred(foo()) else default_value
but here we have 2 foo() calls instead of one. Why can't we write just
something like this:
t = foo() if pred(it) else default_value
where "it" means "foo() value"?

t = foo()+bar()+baz() if pred(it) else baz()-foo()-bar()

What does "it" mean here?

Second, I saw a lot of questions about using dot notation for a
"object-like" dictionaries and a lot of solutions like this:
class dotdict(dict):
def __getattr__(self, attr):
return self.get(attr, None)
__setattr__= dict.__setitem__
__delattr__= dict.__delitem__
why there isn't something like this in a standart library?

Because dot notation for dictionaries is not something we should
encourage.

And the
third. The more I use python the more I see how "natural" it can be. By
"natural" I mean the statements like this:
[x.strip() for x in reversed(foo)]
which looks almost like a natural language. But there is some pitfalls:
if x in range(a, b): #wrong!

Why do you say it's wrong? It's perfectly correct:

1 in range(1, 10)
=> returns True

1.5 in range(1, 10)
=> returns False

5 in range(1, 10)
=> returns True

10 in range(1, 10)
=> returns False

exactly as I expect for element testing in a half-open interval. So
where's the problem? If you want interval testing, you need to perform an
interval test, not an element test.

it feels so natural to check it that way, but we have to write
if a <= x <= b
I understand that it's not a big deal, but it would be awesome to have
some optimisations - it's clearly possible to detect things like that
"wrong" one and fix it in a bytecode.

If I write:

x in range(1, 10)

how do you expect the compiler to read my mind and know if I want the
half-open interval or the closed interval?

 

[Ian Kelly]

t = foo()+bar()+baz() if pred(it) else baz()-foo()-bar()

What does "it" mean here?

"it" would mean the result of the expression foo()+bar()+baz(). What
else could it mean? There are valid objections to the proposal, but the
intended semantics seem perfectly clear.

Cheers,
Ian

 

[Steven D'Aprano]

"it" would mean the result of the expression foo()+bar()+baz(). What
else could it mean?

It could mean the last expression, baz(). Or the entire compound
expression, foo()+bar()+baz().

There are valid objections to the proposal, but the
intended semantics seem perfectly clear.

Fair point, my example was terrible and didn't show the point that was
clear in my head. Mea culpa. How about this instead?

t = foo()+it if pred(it) else bar()

Should that be a SyntaxError, or is `it` a variable that holds its value
from statement to statement?

t = it

t = (foo() if pred(it) else bar()) if cond(it) else baz()


For what it's worth, Apple's defunct Hypertalk language had a couple of
syntax elements very much like that. Hypertalk had a special variable,
"it", which you used like this:

get the number of cards
put it into field "Card number"

I trust I don't have to explain this?

Hypertalk also had a special function, "the result", which worked
something like this:

ask "This is a dialog box. Please type your answer here:"
put the result into field "Your answer"

(or you could use "result()" instead). Both of these worked quite well
with Hypertalk, particularly with it's focus on non-programmers, but they
were designed into the language from the beginning. In Python they would
be the proverbial round peg in a square hole.

BTW, I frequently use "it" for iterators, and making "it" a reserved word
would break a lot of my code. This would make me quite peeved.

 

[Ian Kelly]

It could mean the last expression, baz(). Or the entire compound
expression, foo()+bar()+baz().

Unless the precedence rules were to change, the three expressions in the
example that are operated on by the ternary are these:

1. foo()+bar()+baz()
2. pred(it)
3. baz()-foo()-bar()

So the antecedent would have to be one of those in order to make any
sense at all, and obviously the only choice of those that would be
useful is the first one.

Fair point, my example was terrible and didn't show the point that was
clear in my head. Mea culpa. How about this instead?

t = foo()+it if pred(it) else bar()

Should that be a SyntaxError, or is `it` a variable that holds its value
from statement to statement?

SyntaxError. Implementing this without making 'it' a keyword would be
very sloppy, IMO.

Another option would be to use a special variable named '__', similar to
the variable '_' that the REPL uses to store the result of the last
command. This might break some existing code, but probably not very
much. Mostly it would just be confusing.

t = (foo() if pred(it) else bar()) if cond(it) else baz()

More problematic:

t = foo() if pred(bar() if pred2(it) else str(it)) else bar()

Does the first 'it' belong to the inner or outer ternary? Probably the
inner one. What about the second 'it'?

BTW, I frequently use "it" for iterators, and making "it" a reserved word
would break a lot of my code. This would make me quite peeved.

I do the same, and yes, it would make upgrading a real pain.

Cheers,
Ian

 

[André]

Here are some proposals. They are quite useful at my opinion and I'm
interested for suggestions. It's all about some common patterns.
First of all: how many times do you write something like
    t = foo()
    t = t if pred(t) else default_value
Never!

[snip]

And the third. The more I use python the more I see how "natural" it
can be. By "natural" I mean the statements like this:
    [x.strip() for x in reversed(foo)]
which looks almost like a natural language. But there is some
pitfalls:
    if x in range(a, b): #wrong!

This is true only if x is an integer such that a <= x < b

it feels so natural to check it that way, but we have to write
    if a <= x <= b

This is true if x is an integer OR a float. Two very different
cases, deserving of different notation.

André