Generator expressions v/s list comprehensions

[Steve Holden]

Michael J. Fromberger <[email protected]>
wrote:
...



Almost all, not all. As I repeatedly explained, for example, generator
comprehensions are not quite like anything in other languages (though
you can surely find _similarities_, of course).

Indeed, and the reason I like Python is that it has some aspects of Icon
mixed in with perhaps the most easily-accessible object-oriented
features of any of the programming languages I've learned.

I think most people's reaction to (==against) Lisp's parentheses and
prefix syntax is more intense than that to Python's significant
whitespace, which appears quite natural to most non-programmers even
though many programmers inured to block delimiters do react strongly
against it. It's hard enough for Python to keep its community welcoming
and friendly when you hear some clueless dweeb rant against indendation
for the thousandth time, I think the Lisp community would have had to be
made up of saints to be still friendly and welcoming after a _million_
newcomers raised their whining against parentheses.

.... which it clearly isn't .

I do think the Python community is just a little out-of-the-ordinary,
which is what keeps me (and, I suspect, you: great to see you on the
wires again) coming back to c.l.py.

regards
Steve

 

[Antoon Pardon]

Hi,

When list comprehension was added to the language, I had a lot of
trouble understanding it but now that I am familiar with it, I am not
sure how I programmed in Python without it.

Oh good.

Now I see that generator expressions have been added to the language
with 2.4 and I question the need for it. I know that it allows for lazy
evaluation which speeds things up for larger lists but why was it
necessary to add it instead of improving list comprehension?

Was there some sort of limitation that prevented list comprehension from
taking over the functionality that generator expressions provide?

Sure, and that limitation is: list comprehensions return lists. This
one "limitation" (together with Python's applicative order evaluation,
and you couldn't change THAT without breaking the whole caboodle of
existing programs!) implies everything else.

I have always liked the fact that Python has limited capabilities for
having more than one way to do it and it seem to me that generator
expressions break that philosophy. It is similar to how you have to use
range or xrange depending on how large the range is.

Can someone please explain the reasoning behind it?

Generator comprehensions are wonderful and there is no way Python list
comprehensions can provide the same features, since lists need to be
lists. Sure, list(e(x) for x in foo) IS just the same thing as [e(x)
for x in foo]. We'll remove the redundancy in 3.0 -- not earlier
because it will break backwards compatibility. The only sensible way I
can see right now for 3.0 to remove this redundancy is by removing list
comprehensions and leaving only generator comprehensions, btw.

The one problem I have with generator comprehensions is that, because
list comprhension came first they look like tuple comprehensions.

In order to avoid this same kind of confusion with new comers I think
it would be best if generator comprehensions are explained first and
that list comprehension is just syntax sugar for list(genexp)

 

[Raymond Hettinger]

[Mahesh Padmanabhan]

Now I see that generator expressions have been added to the language
with 2.4 and I question the need for it. I know that it allows for lazy
evaluation which speeds things up for larger lists but why was it
necessary to add it instead of improving list comprehension?

FWIW, we improved list comprehensions too. They run much faster now.


Raymond

 

[Nicolas Lehuen]

[Mahesh Padmanabhan]

Now I see that generator expressions have been added to the language
with 2.4 and I question the need for it. I know that it allows for lazy
evaluation which speeds things up for larger lists but why was it
necessary to add it instead of improving list comprehension?

FWIW, we improved list comprehensions too. They run much faster now.


Raymond

IIRC, list constructors and list comprehensions now use the length of
the underlying sequence (if it defines __len__) to preallocate its
internal array.

Thus, the list comprehension is theoritically more efficient in this
case :

[x for x in range(100000)] vs [x for x in xrange(100000)]

So [x for x in whatever] may be preferable to list(x for x in
whatever) if you don't know what 'whatever' is and it happens in
define __len__.

BTW, I guess this is no longer true with an if statement :

[x for x in range(100000) if x%2==0] may be less efficient than [x for
x in xrange(100000) if x%2==0]

Regards,

Nicolas Lehuen

 

[Raymond Hettinger]

FWIW, we improved list comprehensions too. They run much faster now.

[Nicolas Lehuen]

IIRC, list constructors and list comprehensions now use the length of
the underlying sequence (if it defines __len__) to preallocate its
internal array.

Yes for list().
No for list comps.

Thus, the list comprehension is theoritically more efficient

Aside from the theory being wrong, this is a testable assertion. Use
timeit.

The relative speed performance of genexps vs list comps depends on
length, cache, processor, memory allocator, use of re-usable
containers (tuples, dicts, etc), and other factors. Try this for
example:
dict( (i,i+i) for i in xrange(n) )

My recommendation is to always use list comps when you need a list and
to use genexps whenever values get consumed right away by an enclosing
function.


Raymond