generator / iterator mystery

[Dave Abrahams]

Please consider:
[('a', 0), ('a', 1), ('a', 2)]


# Rewrite the same expression four different ways:
[('a', 0), ('a', 1), ('a', 2), ('b', 0), ('b', 1), ('b', 2), ('c', 0), ('c', 1), ('c', 2)]
[('a', 0), ('a', 1), ('a', 2), ('b', 0), ('b', 1), ('b', 2), ('c', 0), ('c', 1), ('c', 2)]

.... (('a',n) for n in range(3)),
.... (('b',n) for n in range(3)),
.... (('c',n) for n in range(3)) ))
[('a', 0), ('a', 1), ('a', 2), ('b', 0), ('b', 1), ('b', 2), ('c', 0), ('c', 1), ('c', 2)]
[('c', 0), ('c', 1), ('c', 2), ('c', 0), ('c', 1), ('c', 2), ('c', 0), ('c', 1), ('c', 2)]

Huh? Can anyone explain why the last result is different?
(This is with Python 2.6)

Thanks in advance!

 

[Hrvoje Niksic]

[('c', 0), ('c', 1), ('c', 2), ('c', 0), ('c', 1), ('c', 2), ('c', 0), ('c', 1), ('c', 2)]

Huh? Can anyone explain why the last result is different?

list(chain(*EXPR)) is constructing a tuple out of EXPR. In your case,
EXPR evaluates to a generator expression that yields generator
expressions iterated over by chain and then by list. It is equivalent
to the following generator:

def outer():
for x in 'abc':
def inner():
for n in range(3):
yield x, n
yield inner()

list(chain(*outer()))
.... the same result as above ...

The problem is that all the different instances of the inner() generator
refer to the same "x" variable, whose value has been changed to 'c' by
the time any of them is called. The same gotcha is often seen in code
that creates closures in a loop, such as:

fns = [(lambda: x+1) for x in range(3)]
map(apply, fns)

[3, 3, 3] # most people would expect [1, 2, 3]

In your case the closure is less explicit because it's being created by
a generator expression, but the principle is exactly the same. The
classic fix for this problem is to move the closure creation into a
function, which forces a new cell to be allocated:

def adder(x):
return lambda: x+1

fns = [adder(x) for x in range(3)]
map(apply, fns)

[1, 2, 3]

This is why your enum3 variant works.