Generator expressions vs. comprehensions

[Ian Kelly]

Hi all,

I just ran into an interesting but rather subtle little wart today.
The following expressions are not functionally equivalent, even in
Python 3:

tuple(iterator.next() for i in xrange(n))

tuple([iterator.next() for i in xrange(n)])

In the first case, if iterator.next() raises a StopIteration, the
exception is swallowed by the generator expression. The expression
evaluates to a truncated tuple, and the StopIteration is not
propagated.

In the second case, the StopIteration exception is propagated as
expected by the list comprehension. Set and dict comprehensions also
behave this way in Python 3.

Is this distinction generally known? The generator expression
behavior is understandable since a generator would do the same thing,
but I'm disappointed that the inconsistency exists and wasn't fixed in
Python 3 when we had the chance.

Cheers,
Ian

 

[Carl Banks]

Hi all,

I just ran into an interesting but rather subtle little wart today.
The following expressions are not functionally equivalent, even in
Python 3:

tuple(iterator.next() for i in xrange(n))

tuple([iterator.next() for i in xrange(n)])

In the first case, if iterator.next() raises a StopIteration, the
exception is swallowed by the generator expression.  The expression
evaluates to a truncated tuple, and the StopIteration is not
propagated.

In the second case, the StopIteration exception is propagated as
expected by the list comprehension.  Set and dict comprehensions also
behave this way in Python 3.

Is this distinction generally known?  The generator expression
behavior is understandable since a generator would do the same thing,
but I'm disappointed that the inconsistency exists and wasn't fixed in
Python 3 when we had the chance.

As a general rule you shouldn't call the next() method/function
without arranging to catch StopIteration, unless you know the iterator
will never raise StopIteration (such as it if it itertools.count()).
The problem is that if a StopIteration "leaks", as it does with the
generator examples, another iterator further up the stack might catch
it and exit.

The situation here is known. It can't be corrected, even in Python 3,
without modifying iterator protocol to tie StopIteration to a specific
iterator. This is possible and might be worth it to avoid hard-to-
diagnose bugs but it would complicate iterator protocol, which becomes
less useful as it becomes more complex.


Carl Banks

 

[Antoine Pitrou]

Yes, it is.

Why don't you use itertools.islice()?

 

[Michele Simionato]

The situation here is known.  It can't be corrected, even in Python 3,
without modifying iterator protocol to tie StopIteration to a specific
iterator.  This is possible and might be worth it to avoid hard-to-
diagnose bugs but it would complicate iterator protocol, which becomes
less useful as it becomes more complex.

The situation here is a known and could be corrected by changing the
meaning of list comprehension,
for instance by having [x for x in iterable] to be an alias for list(x
for x in iterable). In such a way the StopIteration exception would be
always swallowed and there would be consistency with generator
expressions (by construction). However, the list comprehension would
become non-equivalent to the corresponding for-loop with an .append,
so somebody would be un happy anyway :-/

 

[Peter Otten]

The situation here is known. It can't be corrected, even in Python 3,
without modifying iterator protocol to tie StopIteration to a specific
iterator. This is possible and might be worth it to avoid hard-to-
diagnose bugs but it would complicate iterator protocol, which becomes
less useful as it becomes more complex.

The situation here is a known and could be corrected by changing the
meaning of list comprehension,
for instance by having [x for x in iterable] to be an alias for list(x
for x in iterable). In such a way the StopIteration exception would be
always swallowed and there would be consistency with generator
expressions (by construction). However, the list comprehension would
become non-equivalent to the corresponding for-loop with an .append,
so somebody would be un happy anyway :-/

But the list comprehension is already non-equivalent to the for loop as the
loop variable isn't leaked anymore. We do have three similar constructs with
subtle differences.

I think not turning the list-comp into syntactic sugar for list(genexp) in
py3 is a missed opportunity.

Peter

 

[Michele Simionato]

But the list comprehension is already non-equivalent to the for loop as the
loop variable isn't leaked anymore. We do have three similar constructs with
subtle differences.

I think not turning the list-comp into syntactic sugar for list(genexp) in
py3 is a missed opportunity.

Actually I do agree with the feeling, and this is not the only missed
opportunity in Python 3 :-/

 

[Terry Reedy]

Michele Simionato wrote:

I think not turning the list-comp into syntactic sugar for list(genexp) in
py3 is a missed opportunity.

Implementing it that way was tried but was much slower than the current
implementation. If one uses StopIteration as it is intended to be used
(and is so documented), then, I believe, they are equivalent. There was
a conscious decision to not slow comprehensions for the many to cater to
the very few.

Terry Jan Reedy

 

[Terry Reedy]

Implementing it that way was tried but was much slower than the current
implementation. If one uses StopIteration as it is intended to be used
(and is so documented), then, I believe, they are equivalent. There was
a conscious decision to not slow comprehensions for the many to cater to
the very few.

And those few can always write list(genexp) instead of [genexp] (or
set...) when the minute difference actually matters.