do you master list comprehensions?

[Will Stuyvesant]

Here is a question about list comprehensions [lc]. The
question is dumb because I can do without [lc]; but I am
posing the question because I am curious.

This:

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result = []
for d in data:

.... for w in d:
.... result.append(w)['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

puts all the words in a list, like I want.

How to do this with [lc] instead of for-loops?

I tried funnies like [[w for w in L] for L in data],
that is correct syntax, but you'd never guess.

I know, silly! No need for [lc]! So there's my
question. I am sure a one-liner using [lc] will be very
enlightening. Like studying LISP.

 

[Max M]

I tried funnies like [[w for w in L] for L in data],
that is correct syntax, but you'd never guess.

That is absolutely correct. It's not a funnie at all. If you find it odd
it's only because you are not used to list comprehensiones.

In that case you might be more comfortable with:

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result = []
for l in data:
result += l


--

hilsen/regards Max M, Denmark

http://www.mxm.dk/
IT's Mad Science

 

[Peter Otten]

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result = []
for d in data:

... for w in d:
... result.append(w)['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

puts all the words in a list, like I want.

How to do this with [lc] instead of for-loops?

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
[w for d in data for w in d]

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

See how the for expressions in the list comprehension exactly match your
nested for loops? That's all there is to it.

Peter

 

[Steven Bethard]

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result = []
for d in data:

... for w in d:
... result.append(w)

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

Take advantage of the fact that you can have more than one 'for' in a
list comprehension:

>>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
>>> [item for item_list in data for item in item_list]

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

Steve

 

[Diez B. Roggisch]

data = [['foo','bar','baz'],['my','your'],['holy','grail']]

[e for l in data for e in l]

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

 

[Fredrik Lundh]

I tried funnies like [[w for w in L] for L in data],
that is correct syntax, but you'd never guess.

That is absolutely correct. It's not a funnie at all. If you find it odd it's only because you are
not used to list comprehensiones.

well, syntactically correct or not, it doesn't do what he want...

In that case you might be more comfortable with:

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result = []
for l in data:
result += l

how about (slightly evil):

result = []; map(result.extend, data)

</F>

 

[Max M]

Max M wrote:

I tried funnies like [[w for w in L] for L in data],

That is absolutely correct. It's not a funnie at all.

well, syntactically correct or not, it doesn't do what he want...

Doh! *I* might not be used to list comprehensions then... You are right.

That example could have been expressed more clearly as:

result = data

;-)

--

hilsen/regards Max M, Denmark

http://www.mxm.dk/
IT's Mad Science

 

[Luis M. Gonzalez]

I guess the simplest to do it is like this:

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result=[w for d in data for w in d]
result ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

 

[Luis M. Gonzalez]

I guess the simplest way to do it is like this:

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result=[w for d in data for w in d]
result ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

 

[James Stroud]

Here is one for arbitrary depth:

def unroll(ary):
unrolled = []
for item in ary:
# add test for your favorite sequence type
if ( type(item) == types.ListType or \
type(item) == types.TupleType \
):
unrolled.extend(unroll(item))
else:
unrolled.append(item)
return unrolled

unroll([[1, 2, 3], ('fred', 'barney', ['wilma', 'betty']), 'dino'])

[1, 2, 3, 'fred', 'barney', 'wilma', 'betty', 'dino']

Here is a question about list comprehensions [lc]. The
question is dumb because I can do without [lc]; but I am
posing the question because I am curious.

This:

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result = []
for d in data:

... for w in d:
... result.append(w)

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

puts all the words in a list, like I want.

How to do this with [lc] instead of for-loops?

I tried funnies like [[w for w in L] for L in data],
that is correct syntax, but you'd never guess.

I know, silly! No need for [lc]! So there's my
question. I am sure a one-liner using [lc] will be very
enlightening. Like studying LISP.

--
James Stroud, Ph.D.
UCLA-DOE Institute for Genomics and Proteomics
611 Charles E. Young Dr. S.
MBI 205, UCLA 951570
Los Angeles CA 90095-1570
http://www.jamesstroud.com/

 

[Jeremy Bowers]

Max M wrote:

I tried funnies like [[w for w in L] for L in data],

That is absolutely correct. It's not a funnie at all.

well, syntactically correct or not, it doesn't do what he want...

Doh! *I* might not be used to list comprehensions then... You are right.

That example could have been expressed more clearly as:

result = data

result = data[:]

 

[Fredrik Lundh]

Here is one for arbitrary depth:

def unroll(ary):
unrolled = []
for item in ary:
# add test for your favorite sequence type
if ( type(item) == types.ListType or \
type(item) == types.TupleType \
):
unrolled.extend(unroll(item))
else:
unrolled.append(item)
return unrolled

unroll([[1, 2, 3], ('fred', 'barney', ['wilma', 'betty']), 'dino'])

[1, 2, 3, 'fred', 'barney', 'wilma', 'betty', 'dino']

or, shorter:

(alright, it returns a tuple, but that can be easily fixed, if necessary)

</F>

 

[Timothy Babytch]

data = [['foo','bar','baz'],['my','your'],['holy','grail']]

sum(data, [])

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

The second parameter passed to sum is just to overrride default
initial value "zero".

 

[Will Stuyvesant]

Okay that was fun. Enlightening as I hoped. unroll() in Python, for
arbitrary depth, _flatten in Tkinter (what else is in Tkinter!), sum()
abuse.

The sum(data,[]) was funniest, it works like ((['foo','bar'] + []) +
['my','your']) + ['holy','grail']. Before I think of such things I
have already coded an algorithm in imperative style. Guess I have not
been exposed to functional programming enough.

 

[Steven Bethard]

data = [['foo','bar','baz'],['my','your'],['holy','grail']]

sum(data, [])

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

The second parameter passed to sum is just to overrride default
initial value "zero".

It's worth keeping in mind that this solution has the same efficiency
problems that a loop that =+ strings does:

> python -m timeit -s "data = [range(10) for _ in range(100)]"

"sum(data, [])"
1000 loops, best of 3: 530 usec per loop

> python -m timeit -s "data = [range(10) for _ in range(100)]" "[w for

d in data for w in d]"
10000 loops, best of 3: 151 usec per loop

> python -m timeit -s "data = [range(10) for _ in range(1000)]"

"sum(data, [])"
10 loops, best of 3: 54.2 msec per loop

> python -m timeit -s "data = [range(10) for _ in range(1000)]" "[w for

d in data for w in d]"
100 loops, best of 3: 1.75 msec per loop

The sum function used in this way (or a loop with a +=) is O(N**2) while
the LC is O(N).

Steve

 

[Fredrik Lundh]

python -m timeit -s "data = [range(10) for _ in range(1000)]"

"sum(data, [])"
10 loops, best of 3: 54.2 msec per loop

python -m timeit -s "data = [range(10) for _ in range(1000)]" "[w for

d in data for w in d]"
100 loops, best of 3: 1.75 msec per loop

The sum function used in this way (or a loop with a +=) is O(N**2) while the LC is O(N).

also:

timeit -s "data = [range(10) for _ in range(1000)]" "L = sum(data, [])"
10 loops, best of 3: 4.02e+004 usec per loop

timeit -s "data = [range(10) for _ in range(1000)]" "L = [w for d in data for w in d]"
1000 loops, best of 3: 1.12e+003 usec per loop

timeit -s "data = [range(10) for _ in range(1000)]" "L = []; map(L.extend, data)"
1000 loops, best of 3: 283 usec per loop

timeit -s "data = [range(10) for _ in range(1000)]; from Tkinter import _flatten" "L =
_flatten(data)"
1000 loops, best of 3: 308 usec per loop

(the last one supports arbitrary nestings, the others don't)

</F>

 

[Nick Coghlan]

Here is a question about list comprehensions [lc]. The
question is dumb because I can do without [lc]; but I am
posing the question because I am curious.

This:

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result = []
for d in data:

... for w in d:
... result.append(w)

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

puts all the words in a list, like I want.

There's a way to avoid generating the intermediate list if you don't actually
need it (e.g. you want to feed the sequence to another method):

..>>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
..>>> from itertools import chain
..>>> result = "".join(chain(*data))
'foobarbazmyyourholygrail'


Some timing with integers:

C:\>python -m timeit -s "data = [range(x) for x in range(1000)]" "L= []; map(L.e
xtend, data); max(L)"
10 loops, best of 3: 78.5 msec per loop

C:\>python -m timeit -s "data = [range(x) for x in range(1000)]; from Tkinter im
port _flatten" "max(_flatten(data))"
10 loops, best of 3: 58.4 msec per loop

C:\>python -m timeit -s "data = [range(x) for x in range(1000)]; from itertools
import chain" "max(chain(*data))"
10 loops, best of 3: 43 msec per loop

And with strings:

C:\>python -m timeit -s "data = [map(str, range(x)) for x in range(1000)]" "L= [
]; map(L.extend, data); ''.join(L)"
10 loops, best of 3: 106 msec per loop

C:\>python -m timeit -s "data = [map(str, range(x)) for x in range(1000)]; from
Tkinter import _flatten" "''.join(_flatten(data))"
10 loops, best of 3: 85.4 msec per loop

C:\>python -m timeit -s "data = [map(str, range(x)) for x in range(1000)]; from
itertools import chain" "''.join(chain(*data))"
10 loops, best of 3: 1.2 sec per loop ****** OUCH!!

C:\>python -m timeit -s "data = [map(str, range(x)) for x in range(1000)]; from
itertools import chain" "''.join(list(chain(*data)))"
10 loops, best of 3: 107 msec per loop


Yikes - looks like chain() really sucks for str.join. However, the addition of
the 'list' call makes a big difference. Maybe PySequence_Fast should be called
PySequence_SlowAsADeadDingo() in this case

(FYI, I filed bug report #1085744 on SF about this)

Cheers,
Nick.

 

[Matthew Moss]

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
[e for l in data for e in l]

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

Okay, I tried this in an interactive Python session and it works as
stated. My question is, why? How is the interpreter parsing that list
expression that it makes sense?

 

[Fredrik Lundh]

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
[e for l in data for e in l]

['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

Okay, I tried this in an interactive Python session and it works as
stated. My question is, why? How is the interpreter parsing that list
expression that it makes sense?

the language reference has the answer:

http://www.python.org/doc/2.3.4/ref/lists.html

"... the elements of the new list are those that would be produced
by considering each of the 'for' or 'if' clauses a block, nesting
from left to right, and evaluating the expression to produce a list
element each time the innermost block is reached."

or, conceptually, the compiler strips off the expression, and then
inserts newlines, colons and indentation, and wraps the expression
in an append statement.

so

[e for l in data for e in l]

behaves like:

result = []
for l in data:
for e in l:
result.append(e)

except that it's an expression, and the result variable is hidden. (and
the compiler and the runtime is of course free to implement this in a
more efficient way)

</F>

 

[Stefan Behnel]

data = [['foo','bar','baz'],['my','your'],['holy','grail']]
result = []
for d in data:

.>>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
.>>> from itertools import chain
.>>> result = "".join(chain(*data))
'foobarbazmyyourholygrail'

This is the first time I see that and I totally like the idea of writing
".>>>" instead of ">>>" at the beginning of a line. Thank you Dr. Dobb!
It's unfortunate for c.l.py that Python uses ">>>" as the default prompt
as it messes up the display on mail/news readers that provide "syntax
highlighting" for quotes.

I wish everyone would write examples that way! On the other hand - three
levels of quoting are really rare, maybe it would be easier to change that
in the mail readers...

.... or in Py3k ?

Stefan