free variables /cell objects question

[gangesmaster]

why does CPython require to wrap the free variables if
closure functions by a cell objects?
why can't it just pass the object itself?
.... def g():
.... return x+2
.... return g
.... 3 0 LOAD_DEREF 0 (x)
3 LOAD_CONST 1 (2)
6 BINARY_ADD
7 RETURN_VALUE 2 0 LOAD_CLOSURE 0 (x)
3 BUILD_TUPLE 1
6 LOAD_CONST 1 (<code objec...
9 MAKE_CLOSURE 0
12 STORE_FAST 1 (g)

4 15 LOAD_FAST 1 (g)
18 RETURN_VALUE
i don't see why dereferencing is needed. why not just pass
the object itself to the MAKE_CLOSURE? i.e.

LOAD_FAST 0 (x)
LOAD_CONST 1 (the code object)
MAKE_CLOSURE 0

what problem does the cell object solve?


-tomer

 

[Peter Otten]

why does CPython require to wrap the free variables if
closure functions by a cell objects?
why can't it just pass the object itself?

... def g():
... return x+2
... return g
...
3 0 LOAD_DEREF 0 (x)
3 LOAD_CONST 1 (2)
6 BINARY_ADD
7 RETURN_VALUE
2 0 LOAD_CLOSURE 0 (x)
3 BUILD_TUPLE 1
6 LOAD_CONST 1 (<code objec...
9 MAKE_CLOSURE 0
12 STORE_FAST 1 (g)

4 15 LOAD_FAST 1 (g)
18 RETURN_VALUE

i don't see why dereferencing is needed. why not just pass
the object itself to the MAKE_CLOSURE? i.e.

LOAD_FAST 0 (x)
LOAD_CONST 1 (the code object)
MAKE_CLOSURE 0

what problem does the cell object solve?

If I understand you correctly:

def f(x):
def g():
return x + 2
x = 42
return g
assert f(0)() == 44

Peter

 

[Duncan Booth]

what problem does the cell object solve?

The closure represents the variable, not the object. So if x is rebound to
a different object your inner function g() will now access the new object.
If the object itself was passed to MAKE_CLOSURE then g would only ever see
the value of x from the instant when g was defined.
def g():
print "x is", x
g()
x += 1
g()

x is 1
x is 2

 

[gangesmaster]

ugliness

so this is why [lambda: i for i in range(10)] will always return 9.
imho that's a bug, not a feature.


thanks.
-tomer

 

[Paul Rubin]

so this is why [lambda: i for i in range(10)] will always return 9.
imho that's a bug, not a feature.

Use [(lambda j: lambda: j)(i) for i in range(10)]
or the Python idiom [(lambda i=i: i) for i in range(10)]

 

[Duncan Booth]

ugliness

so this is why [lambda: i for i in range(10)] will always return 9.
imho that's a bug, not a feature.

Horses for courses.

There are cases where you need to get the latest value of the bound
variable, and there are cases where you want the value at the time it was
bound. For the latter case you have always been able to use default
arguments.

 

[Terry Reedy]

| so this is why [lambda: i for i in range(10)] will always return 9.

No, it returns a list of 10 identical functions which each return the
current (when executed) global (module) variable i. Except for names,
'lambda:i' abbreviates 'def f(): return i'.

a=[lambda: i for i in range(10)]
i=42
for j in range(10): print a[j]()

42
42
42
42
42
42
42
42
42
42

for i in range(10): print a()

0
1
2
3
4
5
6
7
8
9

del i
for j in range(10): print a[j]()

Traceback (most recent call last):
File "<pyshell#23>", line 1, in -toplevel-
for j in range(10): print a[j]()
File "<pyshell#8>", line 1, in <lambda>
a=[lambda: i for i in range(10)]
NameError: global name 'i' is not defined

| imho that's a bug, not a feature.

The developers now think it a mistake to let the list comp variable 'leak'
into the global scope. It leads to the sort of confusion that you
repeated. In Py3, the leak will be plugged, so one will get an exception,
as in the last example, unless i (or whatever) is defined outside the list
comp.

Terry Jan Reedy

 

[Terry Reedy]

| or the Python idiom [(lambda i=i: i) for i in range(10)]

While the idiom is commonly written this way, I think it would be clearer
if the two related but different variables had two different names:
[(lambda j=i:j) for i in range(10)]

Terry Jan Reedy

 

[Carl Banks]

ugliness

so this is why [lambda: i for i in range(10)] will always return 9.
imho that's a bug, not a feature.

It's a feature.

If you were to write a nested function like this, where a, b, c, and d,
are cell variables:

print_values():
print a,b,c,d

then it wouldn't work as intended if you passed the variable itself to
the cell rather than a reference. Nested functions have to support
this as well as to create lexical closures.

The thing is, newbies are very likely to use nested functions like I've
shown above, whereas few newbies are likely to use closures. Python
has correctly, IMHO, chosen the way that minimizes surprises for
newbies.


Carl Banks

 

[gangesmaster]

no, it has nothing to do with "i" being global.

tuple(lambda: i for i in range(10))[0]() 9
tuple(lambda: i for i in range(10))[1]()

9

what i see as a bug is this code not working as expected:
.... funcs = []
.... for n in names:
.... def foo():
.... print "my name is", n
.... funcs.append(foo)
.... return funcs
....

foos = make_foos(["hello", "world", "spam", "bacon"])
foos[0]() my name is bacon
foos[1]() my name is bacon
foos[2]() my name is bacon

i have to create yet another closure, make_foo, so that the name
is correctly bound to the object, rather than the frame's slot:
.... def foo():
.... print "my name is", name
.... return foo
........ return [make_foo(n) for n in names]
....

foos = make_foos(["hello", "world", "spam", "bacon"])
foos[0]() my name is hello
foos[1]() my name is world
foos[2]()

my name is spam


-tomer

| so this is why [lambda: i for i in range(10)] will always return 9.

No, it returns a list of 10 identical functions which each return the
current (when executed) global (module) variable i. Except for names,
'lambda:i' abbreviates 'def f(): return i'.

a=[lambda: i for i in range(10)]
i=42
for j in range(10): print a[j]()42

42
42
42
42
42
42
42
42
42

for i in range(10): print a()0

1
2
3
4
5
6
7
8
9

del i
for j in range(10): print a[j]()Traceback (most recent call last):

File "<pyshell#23>", line 1, in -toplevel-
for j in range(10): print a[j]()
File "<pyshell#8>", line 1, in <lambda>
a=[lambda: i for i in range(10)]
NameError: global name 'i' is not defined

| imho that's a bug, not a feature.

The developers now think it a mistake to let the list comp variable 'leak'
into the global scope. It leads to the sort of confusion that you
repeated. In Py3, the leak will be plugged, so one will get an exception,
as in the last example, unless i (or whatever) is defined outside the list
comp.

Terry Jan Reedy

 

[Paul Rubin]

what i see as a bug is this code not working as expected:
... funcs = []
... for n in names:
... def foo():
... print "my name is", n
... funcs.append(foo)
... return funcs

But it does work as expected, if your expectations are based on what
closures actually do.

i have to create yet another closure, make_foo, so that the name
is correctly bound to the object, rather than the frame's slot:

The Python idiom is:

def make_foos(names):
funcs = []
for n in names:
def foo(n=n):
print "my name is", n
funcs.append(foo)
return funcs

The n=n in the "def foo" creates the internal binding that you need.

 

[Steven D'Aprano]

what i see as a bug is this code not working as expected:

def make_foos(names):

... funcs = []
... for n in names:
... def foo():
... print "my name is", n
... funcs.append(foo)
... return funcs

But it does work as expected, if your expectations are based on what
closures actually do.

i have to create yet another closure, make_foo, so that the name
is correctly bound to the object, rather than the frame's slot:

The Python idiom is:

def make_foos(names):
funcs = []
for n in names:
def foo(n=n):
print "my name is", n
funcs.append(foo)
return funcs

The n=n in the "def foo" creates the internal binding that you need.

Hmmm... I thought that the introduction of nested scopes removed the need
for that idiom. Its an ugly idiom, the less I see it the happier I am.

And I worry that it will bite you on the backside if your "n=n" is a
mutable value.

My solution is, don't try to have one function do too much. Making a list
of foos should be a separate operation from making a single foo:
.... def foo():
.... return "my name is " + name
.... return foo
........ foos = []
.... for name in names:
.... foos.append(makefoo(name))
.... return foos
....

L = makefoos(["fred", "wilma"])
L[0]() 'my name is fred'
L[1]()

'my name is wilma'


That makes it easier to do unit testing too: you can test your makefoo
function independently of your makefoos function, if that's important.

If you absolutely have to have everything in one function:
.... def makefoo(name):
.... def foo():
.... return "my name is " + name
.... return foo
.... L = []
.... for name in names:
.... L.append(makefoo(name))
.... return L
....

L = makefoos(["betty", "barney"])
L[0]() 'my name is betty'
L[1]()

'my name is barney'


Best of all, now I don't have to argue as to which binding behaviour is
more correct for closures!!! *wink*

 

[gangesmaster]

[Steven]

My solution is, don't try to have one function do too much. Making a list
of foos should be a separate operation from making a single foo:

that's exactly what my second example does (as well as my production
code)

[Paul]

But it does work as expected, if your expectations are based on what
closures actually do.

yet, i find what closures actually do, to be logically wrong.
moreover, it means the frame object must be kept alive for no reason...
or in my case, two frame objects per foo-function.

The Python idiom is: ....
def foo(n=n):

besides of being ugly, the def f(n=n) idiom is very bad,
programatically speaking. what if the user chooses to be a smartass
and call with n = 7? or what if the function's signature is
meaningful? (as it is in my case)

anyway, this talk is not going anywhere.
thanks for the info, and i'll see how i manage to optimize my code
from here.

-tomer

what i see as a bug is this code not working as expected:
def make_foos(names):
... funcs = []
... for n in names:
... def foo():
... print "my name is", n
... funcs.append(foo)
... return funcs

But it does work as expected, if your expectations are based on what
closures actually do.

The Python idiom is:

def make_foos(names):
funcs = []
for n in names:
def foo(n=n):
print "my name is", n
funcs.append(foo)
return funcs

The n=n in the "def foo" creates the internal binding that you need.Hmmm... I thought that the introduction of nested scopes removed the need

for that idiom. Its an ugly idiom, the less I see it the happier I am.

And I worry that it will bite you on the backside if your "n=n" is a
mutable value.

My solution is, don't try to have one function do too much. Making a list
of foos should be a separate operation from making a single foo:
... return "my name is " + name
... return foo
...>>> makefoo("fred")()
'my name is fred'

def makefoos(names):... foos = []

... for name in names:
... foos.append(makefoo(name))
... return foos
...>>> L = makefoos(["fred", "wilma"])

L[0]() 'my name is fred'
L[1]()'my name is wilma'

That makes it easier to do unit testing too: you can test your makefoo
function independently of your makefoos function, if that's important.

If you absolutely have to have everything in one function:
... def foo():
... return "my name is " + name
... return foo
... L = []
... for name in names:
... L.append(makefoo(name))
... return L
...>>> L = makefoos(["betty", "barney"])

L[0]() 'my name is betty'
L[1]()'my name is barney'

Best of all, now I don't have to argue as to which binding behaviour is
more correct for closures!!! *wink*