When is List Comprehension inappropriate?

[Ben]

I have recently learned how list comprehension works and am finding it
extremely cool. I am worried, however, that I may be stuffing it into
places that it does not belong.

What's the most "pythony" way to do this:

even = []
for x in range(0,width,2):
for y in range(0,height,2):
color = im.getpixel((x,y))
even.append(((x,y), color))

versus list comprehension:

even2 = [((x,y), im.getpixel((x,y))) for x in range(0,width,2) for y
in range(0,height,2)]

Is there a computational difference in creating a blank list and
appending to it versus doing a list comprehension? Are there
advantages to it outside of short and pretty code?

Feel free to tell me a different way to do this, as well.

Thanks,
Ben

 

[Alex Martelli]

I have recently learned how list comprehension works and am finding it
extremely cool. I am worried, however, that I may be stuffing it into
places that it does not belong.

What's the most "pythony" way to do this:

even = []
for x in range(0,width,2):
for y in range(0,height,2):
color = im.getpixel((x,y))
even.append(((x,y), color))

versus list comprehension:

even2 = [((x,y), im.getpixel((x,y))) for x in range(0,width,2) for y
in range(0,height,2)]

Is there a computational difference in creating a blank list and
appending to it versus doing a list comprehension? Are there
advantages to it outside of short and pretty code?

Feel free to tell me a different way to do this, as well.

I like list comprehensions when I'm building up a list, originally
empty, with .append calls within for loops (possibly with an if guard),
unless I need (e.g.) a conditional break, which LCs don't support.

IOW, I would use a LC in your example. However, I would format it more
neatly:

even2 = [((x,y), im.getpixel((x,y)))
for x in range(0,width,2)
for y in range(0,height,2)]

though I guess that's a matter of taste.

Some people think that LCs should not be used except for extremely
simple cases, but I personally disagree with that stance.

The cases where an LC should NOT be used are those in which you are not
really building a list as above described, but e.g. "just looping".

To check whether LC is faster than, slower than, or equal to a more
extensive loop, use timeit, e.g.:

brain:~/py25/Doc alex$ python -mtimeit -s'xs=range(83)' 'L=[]' 'for x in
xs: L.append(x*x)'
10000 loops, best of 3: 34.6 usec per loop

brain:~/py25/Doc alex$ python -mtimeit -s'xs=range(83)' 'L=[x*x for x in
xs]'
100000 loops, best of 3: 19.4 usec per loop

So for this simple case, it may look like the LC is much faster;
however:

brain:~/py25/Doc alex$ python -mtimeit -s'xs=range(83)'
'L=[];ap=L.append' 'for x in xs: ap(x*x)'
10000 loops, best of 3: 22.3 usec per loop

....as you can see, hoisting the L.append lookup out of the loop accounts
for most of the difference.


Alex

 

[Paul McGuire]

even = []
for x in range(0,width,2):
for y in range(0,height,2):
color = im.getpixel((x,y))
even.append(((x,y), color))

versus list comprehension:

even2 = [((x,y), im.getpixel((x,y))) for x in range(0,width,2) for y
in range(0,height,2)]

To simplify access to individual pixels, you can make even2 into a
dict using:

even2asDict = dict(even2)

and then you can directly get the pixel in the 4th row, 5th column
(zero-based) as:

even2asDict[(4,5)]

If you really want something more like a 2-D array, then create a list
comp of lists, as in:

even2 = [ [im.getpixel((x,y)) for x in range(0,width,2) ]
for y in range(0,height,2) ]

which allows you to use list indexing to get individual pixel values,
as in:

even2[4][5]

to get the same pixel as above.

-- Paul

 

[=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=]

I have recently learned how list comprehension works and am finding it
extremely cool. I am worried, however, that I may be stuffing it into
places that it does not belong.

What's the most "pythony" way to do this:

even = []
for x in range(0,width,2):
for y in range(0,height,2):
color = im.getpixel((x,y))
even.append(((x,y), color))

versus list comprehension:

even2 = [((x,y), im.getpixel((x,y))) for x in range(0,width,2) for y
in range(0,height,2)]

I would definitely not use list comprehension in this case. While they
may be faster, Psyco is great here. Also, if you have lots of 2d-loops
like "for x in something: for y in something:", then it could be more
beautiful to separate the iteration from the task:

def iterimage(im, width, height, step = 1):
for y in range(0, height, step):
for x in range(0, width, step):
yield (x, y), im.getpixel((x, y))

Then the list comprehension becomes a little more manageable:

even2 = [(pos, col) for pos, col in iterimage(im, width, height, 2)]

Although this must definitely be the slowest of all the different approaches.

 

[Duncan Booth]

What's the most "pythony" way to do this:

even = []
for x in range(0,width,2):
for y in range(0,height,2):
color = im.getpixel((x,y))
even.append(((x,y), color))

versus list comprehension:

even2 = [((x,y), im.getpixel((x,y))) for x in range(0,width,2) for y
in range(0,height,2)]
....

Feel free to tell me a different way to do this, as well.

Untested code, but I would try to avoid calling getpixel:

data = list(im.getdata())
width, height = im.size
even = [ data[i:i+width:2] for i in range(0, width*height, 2*width)]

That creates a 2 dimensional list rather than one long list, and doesn't
create the x,y tuples, but since they are implied by the position in the
list I don't actually see why you would want to create them at all. You can
calculate them separately if you actually need them.

 

[Steven D'Aprano]

I have recently learned how list comprehension works and am finding it
extremely cool. I am worried, however, that I may be stuffing it into
places that it does not belong.

Others have suggested reasons why you might or might not want to use list
comprehensions. Here's three more reasons:

* You have a LOT of data to handle. (But remember that a lot to you might
not be a lot to your computer.)

* You don't need to handle the items all at once, and can handle the items
one at a time instead.

Use an iterator, generator expression, or other lazily-evaluated function
instead. That way you avoid forming the list all at once.

* You have to write code that's backwards-compatible to an old version of
Python.

Use a list and a for-loop.

 

[Alex Martelli]

even2 = [(pos, col) for pos, col in iterimage(im, width, height, 2)]

list(iterimage(etc etc))

is surely a better way to express identical semantics. More generally,
[x for x in whatever] (whether x is a single name or gets peculiarly
unpacked and repacked like here) is a good example of inappropriate LC,
to get back to the question in the subject: list(whatever) is the "one
obvious way" to perform the same task.


Alex

 

[bearophileHUGS]

BJörn Lindqvist:

While they
may be faster, Psyco is great here. Also, if you have lots of 2d-loops
like "for x in something: for y in something:", then it could be more
beautiful to separate the iteration from the task:

def iterimage(im, width, height, step = 1):
for y in range(0, height, step):
for x in range(0, width, step):
yield (x, y), im.getpixel((x, y))

Just a note: Psyco usually isn't able to speed up generators (ShedSkin
recently has hadded a support of them too, and it seem fast enough).

Bye,
bearophile

 

[Steve Holden]

even2 = [(pos, col) for pos, col in iterimage(im, width, height, 2)]

list(iterimage(etc etc))

is surely a better way to express identical semantics. More generally,
[x for x in whatever] (whether x is a single name or gets peculiarly
unpacked and repacked like here) is a good example of inappropriate LC,
to get back to the question in the subject: list(whatever) is the "one
obvious way" to perform the same task.

Clearly the comprehension you complain about is sub-optimal.

The essential difference, however, is between

[x for x in iterimage(im, width, height, 2)]

and

list(iterimage(im, width, height, 2))

I agree that the latter is the obvious way, but the difference isn't as
large as your leap makes it look - and we had to await the invention of
the generator expression for it to be a practical choice.

regards
Steve

 

[Aahz]

list(iterimage(etc etc))

is surely a better way to express identical semantics. More generally,
[x for x in whatever] (whether x is a single name or gets peculiarly
unpacked and repacked like here) is a good example of inappropriate LC,
to get back to the question in the subject: list(whatever) is the "one
obvious way" to perform the same task.

Except of course, when it's

[x for x in whatever if x]

I'm exceedingly fond of replacing filter() with listcomps. They're so
much more readable and often faster.

 

[Alex Martelli]

list(iterimage(etc etc))

is surely a better way to express identical semantics. More generally,
[x for x in whatever] (whether x is a single name or gets peculiarly
unpacked and repacked like here) is a good example of inappropriate LC,
to get back to the question in the subject: list(whatever) is the "one
obvious way" to perform the same task.

Except of course, when it's

[x for x in whatever if x]

I'm exceedingly fond of replacing filter() with listcomps. They're so
much more readable and often faster.

Sure, if there are other clauses in the LC (be they for or if ones) you
can't just call list(...) -- and I do entirely agree that filter can be
put out to pasture. Similarly, you need the LC if you're performing
some processing on the items -- for example, if you have an iterator
yielding pairs,
[(y,x) for x,y in whatever]
you do need the unpacking and repacking to achieve this swapping of
items within each pair -- what I was pointing out was re the simpler and
most common case:
[(x,y) for x,y in whatever]
no processing needed, no if clauses, etc, and thus better expressed as
list(whatever)


Alex

 

[Paddy]

I have recently learned how list comprehension works and am finding it
extremely cool. I am worried, however, that I may be stuffing it into
places that it does not belong.

What's the most "pythony" way to do this:

even = []
for x in range(0,width,2):
for y in range(0,height,2):
color = im.getpixel((x,y))
even.append(((x,y), color))

versus list comprehension:

even2 = [((x,y), im.getpixel((x,y))) for x in range(0,width,2) for y
in range(0,height,2)]

Is there a computational difference in creating a blank list and
appending to it versus doing a list comprehension? Are there
advantages to it outside of short and pretty code?

Feel free to tell me a different way to do this, as well.

Thanks,
Ben

I have found that I have gone too far when I used listcomps for their
sideeffects rather than wanting the list produced, for example the
second listcomp below is an expression as statement I don't want the
list produced - just the effect on data.

# some random ranges
data = [range(random.randrange(3,7)) for x in range(4)]
# but I want each range jumbled
[ random.shuffle(d) for d in data] [None, None, None, None]
data [[2, 0, 3, 1], [0, 2, 1], [3, 4, 1, 0, 2], [2, 1, 0, 3]]

(I do know how to re-write it).

- Paddy.

 

[John J. Lee]

even2 = [(pos, col) for pos, col in iterimage(im, width, height, 2)]

list(iterimage(etc etc))
is surely a better way to express identical semantics. More
generally,
[x for x in whatever] (whether x is a single name or gets peculiarly
unpacked and repacked like here) is a good example of inappropriate LC,
to get back to the question in the subject: list(whatever) is the "one
obvious way" to perform the same task.

Clearly the comprehension you complain about is sub-optimal.

The essential difference, however, is between

[x for x in iterimage(im, width, height, 2)]

and

list(iterimage(im, width, height, 2))

I agree that the latter is the obvious way, but the difference isn't
as large as your leap makes it look - and we had to await the
invention of the generator expression for it to be a practical choice.

What generator expression? I don't see a genexp in your examples.


John

 

[Alex Martelli]

I have found that I have gone too far when I used listcomps for their
sideeffects rather than wanting the list produced, for example the

I agree.

second listcomp below is an expression as statement I don't want the
list produced - just the effect on data.

# some random ranges
data = [range(random.randrange(3,7)) for x in range(4)]
# but I want each range jumbled
[ random.shuffle(d) for d in data] [None, None, None, None]
data [[2, 0, 3, 1], [0, 2, 1], [3, 4, 1, 0, 2], [2, 1, 0, 3]]

(I do know how to re-write it).

for d in data: random.shuffle(d)


Alex

 

[Paddy]

...

I have found that I have gone too far when I used listcomps for their
sideeffects rather than wanting the list produced, for example the

I agree.

second listcomp below is an expression as statement I don't want the
list produced - just the effect on data.

# some random ranges
data = [range(random.randrange(3,7)) for x in range(4)]
# but I want each range jumbled
[ random.shuffle(d) for d in data] [None, None, None, None]
data

[[2, 0, 3, 1], [0, 2, 1], [3, 4, 1, 0, 2], [2, 1, 0, 3]]

(I do know how to re-write it).

for d in data: random.shuffle(d)

Alex

Hah,
"You just couldn't let it lie" ;-)

I just knew someone would bite, and re-write it.
Now to reel, you in. Its fish for dinner today!


- Paddy.