list comprehension question

[John Yeung]

Apart from the presence of 'item' at the beginning of the
list comprehension as opposed to 'b.append(item)' at the
end of the for-loop, how exactly does the listcomp force
you to "bounce [..] back and forth" to follow the logic?

It's precisely the fact that the item is at the beginning which is the
main instigator of the bounce. To tranlate

[f(x) for x in a]

into a loop, you start in the middle, work your way toward the right,
then bounce back to the left.

I think some of us, perhaps subconsciously, get used to reading that
small chunk on the right, then bouncing left. See enough small, flat
LCs, and we may be conditioned to think that the general rule is to
start with that small chunk on the right and bounce left.

When confronted with a long, nested LC then, some of us have the
instinct to read the rightmost chunk, then bounce left to the next-
rightmost chunk, etc.

Essentially, if you see [B A] over and over and over again, when
finally confronted with LCs of more "elements", it's not immediately
clear that the pattern of increasing nestedness is

[B A] => [C A B] => [D A B C] => etc.

rather than

[B A] => [C B A] => [D C B A] => etc.

Maybe we're wired wrong, but there are many of us who find the second
pattern more logical, which is why it is so easy for us to mess up the
order of the nesting in a more complex LC.

John

 

[alex23]

It's precisely the indentation and colons (plus newlines) that makes
nested for-loops easier to read than list-comps with multiple fors.

You can get back *nearly* all the readability by splitting the list comp
into multiple lines:

It was less the overall readability I was commenting on, and more the
claim that the listcomp required a 'back-and-forth' parsing to
understand. As you've reinforced, the listcomp can be readily
converted back to the multiple-line form by the inclusion of colons &
EOL markers, which means you can make as much sequential sense from a
listcomp as you can a for-loop.

 

[Roel Schroeven]

John Yeung schreef:

Essentially, if you see [B A] over and over and over again, when
finally confronted with LCs of more "elements", it's not immediately
clear that the pattern of increasing nestedness is

[B A] => [C A B] => [D A B C] => etc.

rather than

[B A] => [C B A] => [D C B A] => etc.

Maybe we're wired wrong, but there are many of us who find the second
pattern more logical, which is why it is so easy for us to mess up the
order of the nesting in a more complex LC.

Indeed, that is exactly the reason why I find it difficult to read
nested loop comprehensions, and why I almost never use them.

--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
-- Isaac Asimov

Roel Schroeven

 

[J Kenneth King]

On 5/5/2009 9:15 AM J Kenneth King said...


Can you provide a link for this? I'd like to see specifically what's
being discouraged, as I'd be surprised to find routine usage frowned
upon.

Emile

http://docs.python.org/tutorial/datastructures.html#nested-list-comprehensions


"If you’ve got the stomach for it, list comprehensions can be
nested. They are a powerful tool but – like all powerful tools – they
need to be used carefully, if at all."

and

"In real world, you should prefer builtin functions to complex flow
statements."

 

[Boris Borcic]

If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
return a new list of each individual element in these tuples, I can do
it with a nested for loop but when I try to do it using the list
comprehension b = [j for j in i for i in a], my output is b =
[5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
doing wrong?

just fyi, in python 2.6

list(itertools.chain.from_iterable(a))

would do it

in python 2.5

list(itertools.chain(*a))

would do it too, but I wouldn't try it with arbitrarily long a

 

[Emile van Sebille]

On 5/5/2009 9:15 AM J Kenneth King said...
Can you provide a link for this? I'd like to see specifically what's
being discouraged, as I'd be surprised to find routine usage frowned
upon. http://docs.python.org/tutorial/datastructures.html#nested-list-comprehensions

Thanks.


"If you’ve got the stomach for it, list comprehensions can be
nested. They are a powerful tool but – like all powerful tools – they
need to be used carefully, if at all."[/QUOTE]

Yes, although this is in reference to nested list comprehensions. Eg,

[xx for xx in [yy for yy in [zz for zz in iterable if z] if y] if x]

The section above on list comprehensions advocates their use saying:

"List comprehensions provide a concise way to create lists without
resorting to use of map(), filter() and/or lambda. The resulting list
definition tends often to be clearer than lists built using those
constructs."

Emile

 

[Steven D'Aprano]

comprehensions


"If you’ve got the stomach for it, list comprehensions can be nested.
They are a powerful tool but – like all powerful tools – they need to be
used carefully, if at all."

How does this discourage the use of list comprehensions? At most, it
warns that complicated list comps are tricky. Complicated *anything* are
tricky.

and

"In real world, you should prefer builtin functions to complex flow
statements."

That's ridiculous. The example given is a special case. That's like
saying "Loops are hard, so in the real world, if you want a loop, find a
builtin function that does what you want instead."

What's the "builtin function" we're supposed to prefer over a "complex
flow statement" like this?

# split text into word fragments of length <= 3
sentence = "a sentence is a series of words"
new = [word[i:i+3] for word in sentence.split() for i in range(0, len(word), 3)]

 

[Lie Ryan]

"If you’ve got the stomach for it, list comprehensions can be nested.
They are a powerful tool but – like all powerful tools – they need to be
used carefully, if at all."

How does this discourage the use of list comprehensions? At most, it
warns that complicated list comps are tricky. Complicated *anything* are
tricky.

and

"In real world, you should prefer builtin functions to complex flow
statements."

That's ridiculous. The example given is a special case. That's like
saying "Loops are hard, so in the real world, if you want a loop, find a
builtin function that does what you want instead."

What's the "builtin function" we're supposed to prefer over a "complex
flow statement" like this?

# split text into word fragments of length <= 3
sentence = "a sentence is a series of words"
new = [word[i:i+3] for word in sentence.split() for i in range(0, len(word), 3)]

I often found list comprehension *more* readable than the equivalent
for-loop because of its density. Seeing complex for-loop requires one
step of thinking for each line, but in list comprehension I can skip
some of the steps because I know what list comprehension would roughly
look like.

i.e.

when I process this:

lst = [] #1
for i in range(10): #2
lst.append(i) #3

I do 3 steps of thinking for each line

but seeing this is only one step, since I know it is going to create new
list (skipping step #1) and I know i will be appended to that list
(skipping step #3)

 

[Lie Ryan]

"If you’ve got the stomach for it, list comprehensions can be nested.
They are a powerful tool but – like all powerful tools – they need to be
used carefully, if at all."

How does this discourage the use of list comprehensions? At most, it
warns that complicated list comps are tricky. Complicated *anything*
are tricky.

and

"In real world, you should prefer builtin functions to complex flow
statements."

That's ridiculous. The example given is a special case. That's like
saying "Loops are hard, so in the real world, if you want a loop, find
a builtin function that does what you want instead."

What's the "builtin function" we're supposed to prefer over a "complex
flow statement" like this?

# split text into word fragments of length <= 3
sentence = "a sentence is a series of words"
new = [word[i:i+3] for word in sentence.split() for i in range(0,
len(word), 3)]

I often found list comprehension *more* readable than the equivalent
for-loop because of its density. Seeing complex for-loop requires one
step of thinking for each line, but in list comprehension I can skip
some of the steps because I know what list comprehension would roughly
look like.

i.e.

when I process this:

lst = [] #1
for i in range(10): #2
lst.append(i) #3

I do 3 steps of thinking for each line

but seeing this is only one step, since I know it is going to create new
list (skipping step #1) and I know i will be appended to that list
(skipping step #3)

I don't know what I was thinking before sending that last line, here is
what it was supposed to be:

but in list comprehension I see this in only one step, since I know it
is going to create new list (skipping step #1) and I know 'i' will be
appended to that list (skipping step #3) and this also continues to
nested list comprehensions

 

[Lie Ryan]

or (even better)
if isinstance(el, (list, tuple))

However, it is my contention that you shouldn't be flattening by type --
you should know where, explicitly to flatten. If I have 3-D points as
triples, a tree of points would, by this code, get flattened into a
a list of numbers. If, however, I create a class with the same
elements, but a method or two, suddenly flatten will produce a list
of points. It is my contention that the tree abstraction should be
responsible for producing its leaves, rather than a blunderbus that
doesn't know where one container abstraction ends, and its contents
begin. In other words, I'd like to see thigs like "flatten one layer."

Maybe that is why some modules like PIL or pygame accepts list of
coordinates in flattened list of number instead of nested (i.e. they
accepts [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] and do the grouping
themself).

 

[J Kenneth King]

How does this discourage the use of list comprehensions? At most, it
warns that complicated list comps are tricky. Complicated *anything* are
tricky.

They are tricky and need to be used carefully, *if at all*.

IMO this means that if there's a way to do it without a nested list
comprehension, then that solution should be preferred.

That's ridiculous. The example given is a special case. That's like
saying "Loops are hard, so in the real world, if you want a loop, find a
builtin function that does what you want instead."

What's the "builtin function" we're supposed to prefer over a "complex
flow statement" like this?

It's not ridiculous and says nothing of the sort. You're jumping to
conclusions. If you have a problem with it, I suggest you complain to
the Python documentation people. I don't make this stuff up.

Just look at many of the example solutions provided in this thread to
the OP's problem. The examples in the link I provided are also good as
are ones provided in the itertools documentation.

Keep in mind that nested comprehensions are still available because
they do have a use case that justifies their existence. However, I
think the Python documentation warns against their use because people
might rely on them for problems where they aren't necessary and since
they are difficult to read... it can lead to difficult to read code.

 

[Terry Reedy]

Keep in mind that nested comprehensions are still available because
they do have a use case that justifies their existence.

Nested comprehensions are available because because the syntax makes
them available by default and making a fiddly exception would be
contrary to Python's style. A list comp creates an iterable. A list
comp requires use of an iterable. Therefore, a list comp may use a list
comp.

However, I
think the Python documentation warns against their use because people
might rely on them for problems where they aren't necessary and since
they are difficult to read... it can lead to difficult to read code.

Whenever the expression that results in the iterable used by a list comp
is sufficiently complex, readability is improved by pulling it out as a
separate statement. Nested list comps are often examplex of such
sufficiently complex expressions, but not the only possible one.

tjr

 

[J Kenneth King]

Nested comprehensions are available because because the syntax makes
them available by default and making a fiddly exception would be
contrary to Python's style. A list comp creates an iterable. A list
comp requires use of an iterable. Therefore, a list comp may use a
list comp.


Whenever the expression that results in the iterable used by a list
comp is sufficiently complex, readability is improved by pulling it
out as a separate statement. Nested list comps are often examplex of
such sufficiently complex expressions, but not the only possible one.

tjr

Agreed. A very succinct explanation of the point I was trying to make.

 

[Steven D'Aprano]

They are tricky and need to be used carefully, *if at all*.

IMO this means that if there's a way to do it without a nested list
comprehension, then that solution should be preferred.

Earlier, you claimed that list comps in general were discouraged:

"List comprehensions can make a reader of your code apprehensive because
it can read like a run-on sentence and thus be difficult to parse. The
Python documentation discourages their use and I believe for good reason."

Emile said "I'd be surprised to find routine usage frowned upon", giving
you the opportunity to correct his (mis)understanding. You failed to do
so, which I took as meaning that you agreed that routine usage of simple
list comps were frowned upon. Now you're talking about *nested* list
comps. You started off (apparently) objecting to list comps in general,
because they "can" make readers apprehensive. Now you seem to be saying
that it's only the complicated, overly-dense ones which rightly make
readers apprehensive which you object to.

I suppose we're making progress if we agree that the Python docs warn
against unnecessarily complicated nested list comps. Whether it
discourages as well as warns is a matter of interpretation. But there's
certainly no sign that list comps as a general technique are discouraged
just because overly-complicated list comps are tricky to read. The same
can be said about *any* piece of code.

 

[J Kenneth King]

Earlier, you claimed that list comps in general were discouraged:

"List comprehensions can make a reader of your code apprehensive because
it can read like a run-on sentence and thus be difficult to parse. The
Python documentation discourages their use and I believe for good reason."

Ooops. Typo. My bad. Had it been quoted earlier it would have saved a
few posts for sure. Such is the Internet.

Emile said "I'd be surprised to find routine usage frowned upon", giving
you the opportunity to correct his (mis)understanding. You failed to do
so, which I took as meaning that you agreed that routine usage of simple
list comps were frowned upon. Now you're talking about *nested* list
comps. You started off (apparently) objecting to list comps in general,
because they "can" make readers apprehensive. Now you seem to be saying
that it's only the complicated, overly-dense ones which rightly make
readers apprehensive which you object to.

I was rather confused by that. I use list comps all the time and said
several times that I don't object to list comps, just nested ones.

I suppose we're making progress if we agree that the Python docs warn
against unnecessarily complicated nested list comps. Whether it
discourages as well as warns is a matter of interpretation. But there's
certainly no sign that list comps as a general technique are discouraged
just because overly-complicated list comps are tricky to read. The same
can be said about *any* piece of code.

Well I did my best to make my interpretation clear. If the
documentation says that nested list comps are difficult to read and
should be used rarely, if at all, then I generally consider that
"discouraging" their use. Meaning of course that in the right
situation one may be able to justify their own use of a nested comp.
However, for every day problems one wouldn't be encouraged to use them
so liberally as some tend to do.


So.. we have some sort of consensus then? This might be a rare
phenomenon on Usenet...

Cheers,

J.