M
Mark Lawrence
functory
Ooh, that's bad even by my standards.
functory
Ooh, that's bad even by my standards.
R
Robert Kern
I count 162 uses in the Python standard library, 69 uses in numpy, 108 in
matplotlib, and 238 uses in Biopython. Adding in the unit tests for each would
add significantly to those counts.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
I
Ian Kelly
You know, partial does handle keyword arguments correctly:
funcs = [partial(lambda x, i: x ** i, i=i) for i in range(5)]
This might be bad practice though if there are other arguments to the
right of the keywords that the eventual caller might want to specify,
as those arguments would then effectively be keyword-only. The error
message in that case is not particularly illuminating, either:
.... return (a, b, c)
....Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: f() got multiple values for keyword argument 'b'
It would be a bit clearer if it instead noted that 'c' were
keyword-only in the 'g' argspec.
Cheers,
Ian
R
Rotwang
I don't think anyone has suggested this yet:
funcs = [(lambda i: lambda x: x**i)(i) for i in range(5)]
It's a bit more opaque than other solutions, but it fits in one line and
avoids defining functions with an implicit second argument, if that's
desirable for some reason.
S
Steven D'Aprano
Oooh, nice! A factory function solution. Excellent!
R
rusi
Your expectations are reasonable. Heres the equivalent in Haskell
from which python has taken comprehensions.
---------------------------------
GHCi, version 7.4.1: http://www.haskell.org/ghc/ :? for help
Prelude> let funcs = [ (\ x -> x ^ i)| i <- [0..4]]
Prelude> (funcs !! 0)(2)
1
Prelude> (funcs !! 1)(2)
2
Prelude> (funcs !! 2)(2)
4
Prelude>
S
Steven D'Aprano
You forget to finish that sentence.
"Your expectations are reasonable, for languages that don't have
variables which can actually vary."
*wink*
For purely functional languages like Haskell, the behaviour you show
below makes sense. Since Haskell doesn't allow variables to change their
value, once a closure sees i=1 (say), then it must *always* see i=1.
But that's not the case in Python, where the Haskell behaviour would be
unfortunate. Imagine if you did this:
VERBOSE = True
def function(arg):
if VERBOSE:
print("calling function with arg %r" % arg)
process(arg)
You would expect the function to honour changes to the variable VERBOSE,
would you not? Using Python's scoping rules for closures, it does. Using
Haskell's rules, it wouldn't.
R
rusi
Heres a more appropriate analog
------------------------------------------
VERBOSE = True
def function(arg):
if VERBOSE:
print("calling function with arg %r" % arg)
process(arg)
def caller():
VERBOSE = False
function(1)
R
rusi
To come back to the OPs question.
Variables can be assigned. Or they can be bound.
[C++ programmers will be familiar with the difference between
initialization and assignment]
List comprehensions are defined in terms of assignment to the local
variable rather than binding.
Hence the issue.
Below are two functions that simulate mapping (lambda x: x**i) where
the i's come from some given list
def binding_version(lst):
if not lst: return []
i = lst[0]
return [(lambda x: x ** i)] + binding_version(lst[1:])
def assign_version(lst):
ret = []
for i in lst:
ret.append(lambda x: x**i)
return ret
----------------------
Variables can be assigned. Or they can be bound.
[C++ programmers will be familiar with the difference between
initialization and assignment]
List comprehensions are defined in terms of assignment to the local
variable rather than binding.
Hence the issue.
Below are two functions that simulate mapping (lambda x: x**i) where
the i's come from some given list
def binding_version(lst):
if not lst: return []
i = lst[0]
return [(lambda x: x ** i)] + binding_version(lst[1:])
def assign_version(lst):
ret = []
for i in lst:
ret.append(lambda x: x**i)
return ret
----------------------
4
R
rusi
Ok let me restate: if python were to work that way (without the
global) we could say either
a Python chooses to have dynamic scoping of variables
or
b There is a bug in python's scoping rules
I would guess that most younger folks (who've not seen lisp and apl)
would choose b
We can say the same analogously in the context of ZF expressions when
the i leaks as it does in the OPs example.
Another tack on the same question: python 3 cleaned up the variable
leakage from inside ZFs to outside. It missed cleaning up the leakage
from one step to next.
tl;dr version: Beware of mixing up functional and imperative
programming
Double-beware when you are a language designer
C
Chris Angelico
Or c, there's a declaration at the top:
from __future__ import variable_declarations
Like braces, it's a MASSIVE improvement to the language, and it's part
of a huge conspiracy that it isn't there by default. But all it takes
is a little future statement and you're on Python 4000!
</troll>
ChrisA
(I shouldn't post at half past three in the morning. I get stupid... er.)
D
Dennis Lee Bieber
Wrong... "function" still sees the module level VERBOSE with a value
of True, whereas "caller" created a local VERBOSE set to False. To get
your behavior needs:
def caller():
global VERBOSE
VERBOSE = False
function(1)
H
Hans Mulder
That would be quite different from what Rusi is after.
If you add `global VERBOSE` to `caller`, then there is only one
variable named `VERBOSE` and what `function` does, depends on
the most recent assignment to that variable.
If you remove your `global VERBOSE`, then there are two
variables by that name, one global and one local to `caller`.
In that case, there is the question of which one `function`
will use.
The function `function` refers to a variable `VERBOSE` that
isn't local. In some programming langauages, the interpreter
would then scan the call stack at run-time, looking for a scope
where that name is defined. It would find the local one in
`caller`. This is known as "dynamic binding".
Other interpreters use the `VERBOSE` that was in scope at
the point in the program text where `function` was defined.
In this case, that would be the global one. This is called
"lexical binding".
Some programming languages allow you to indicate on a per-
variable basis whether you want dynamic or lexical binding.
Python is firmly in the lexical camp. Dynamic binding is not
available in Python, and never will be.
Hope this helps,
-- HansM
I
Ian Kelly
I don't believe that dynamic vs. lexical binding is what rusi was
attempting to describe. If he was, then Python and Haskell would be a
bad comparison since both are lexical. Rather, I think what he was
trying to show was capture by reference vs. capture by value in the
context of closures. Python uses capture by reference, and so the
upvalue is the value of that reference at the time the closure is
called. Haskell uses capture by value, and the upvalue is the value
at the time of definition.
I've also seen the distinction described as "early" vs. "late" binding
on this list, but I'm not sure how precise that is -- I believe that
terminology more accurately describes whether method and attribute
names are looked up at compile-time or at run-time, late binding being
the feature that makes duck typing possible.
Cheers,
Ian
H
Hans Mulder
I think that these terms describe the problem at the start of
this thread: the OP was expecting early binding. However, Python
does late binding, or "acts funny" as it says in the subject line.
-- HansM
R
rusi
Good point. See below.
Thats a good intention. The question is whether it is uniformly
realized.
Consider the following
def foo(x):
i = 100
if x:
j = [i for i in range(10)]
return i
else:
return i
In python 2 two different 'i's could be returned. In python3 only one
i can be returned.
One could call it dynamic binding. Evidently Guido thinks it a bug
which is why he changed it.
The leakage of i in the OPs question is the same kind of bug.
tl;dr version:
This is 2012. Dynamic binding is considered a bug even by the lisp
community where it originated.
Lets at least call it a feature and a gotcha
S
Steven D'Aprano
A simpler example:
def foo():
i = 100
j = [i for i in range(10)]
return i
In Python 3, foo() returns 100; in Python 2, it returns 9.
One could also call it a tractor, but it isn't either of those things.
The difference is whether or not list comprehensions create their own
scope. In Python 2, they don't; in Python 3, they do. Personally I don't
have an opinion as to which is better, but in neither case does this have
any thing to do with lexical versus dynamic binding.
It was a case that either list comps be changed to *not* expose their
loop variable, or generator expressions be changed *to* expose their loop
variable. The Python 2 situation where list comps and gen expressions
have opposite behaviour was unfortunate.
Not at all. The OP was *deliberately* creating a closure using i -- under
those circumstances, it would be a bug if i *didn't* leak.
The OP's gotcha was:
1) he expected the closure to use early binding, where i in each function
was bound to the value of i in the enclosing scope at the moment the
function was defined;
2) but Python actually uses late binding for closures, where the value of
i in each function is set to the value of i in the enclosing scope when
the function is called.
As far as I can tell, Python always uses late binding for scopes; the
only time it does early binding is for default values of function
parameters.
R
rusi
You did not get the point.
Converting my example to your format:
def foo_steven(n):
i = 100
j = [i for i in range(n)]
return i
$ python3
Python 3.2.3 (default, Jun 26 2012, 00:38:09)
[GCC 4.7.1] on linux2
Type "help", "copyright", "credits" or "license" for more information..... i = 100
.... j = [i for i in range(n)]
.... return i
....$ python
Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information..... i = 100
.... j = [i for i in range(n)]
.... return i
....
Python 2:
When n>0 comprehension scope i is returned
When n=0 function scope i is returned
Python 3: The return statement is lexically outside the comprehension
and so that outside-scope's i is returned in all cases.
S
Steven D'Aprano
I got the point. I just thought it was unnecessarily complex and that it
doesn't demonstrate what you think it does.
Yes, we know that in Python 3, list comprehensions create their own
scope, and the loop variable does not leak.
Yes, we know that in Python 2, list comprehensions don't create their own
scope, and consequently the list variable does leak.
Incorrect.
In Python 2, *there is no comprehension scope*. There is only local
scope. In Python 2, regardless of the value of n, the local variable i is
ALWAYS returned. It just happens that sometimes the local variable i is
modified by the list comprehension, and sometimes it isn't. In Python 2,
this is no more mysterious than this piece of code:
def example(n):
i = 100
for i in range(n):
pass
return i
py> example(0)
100
py> example(4)
3
If the loop doesn't execute, the loop variable isn't modified. In Python
2, it doesn't matter whether you use a for-loop or a list comprehension,
the loop variable is local to the function.
Correct. In Python 3, list comprehensions now match generator expressions
and introduce their own scope which does not effect the local function
scope.
Some history:
http://bugs.python.org/issue510384
http://bugs.python.org/issue1110705
S
Steven D'Aprano
I don't remember whether it is Javascript or PHP that uses dynamic
binding, but whichever it is, it is generally considered to be a bad
idea, at least as the default or only behaviour.
Bash is another language with dynamic binding. Some very old versions of
Lisp use dynamic binding, because it was the easiest to implement. Most
modern languages use lexical (also known as static) binding, because it
is more sensible.
Here is an illustration of the difference: suppose we have two modules,
library.py and main.py:
# library.py
x = 23
def func(y):
return x + y
# main.py
import library
x = 1000
print func(1)
If main.py prints 24 (and it does), then Python is using lexical scoping.
But if it prints 1001 (which it doesn't), then it is using dynamic
scoping. The obvious problem with dynamic binding is that the behaviour
of a function may vary depending on where you call it.
I don't think "by reference" versus "by value" are good terms to use
here, since they risk conflating the issue with "call by reference"
versus "call by value" semantics. I prefer "late" versus "early", as you
suggest below.
Not necessarily *compile* time, but the distinction is between when the
function is defined (which may at compile time, or it may be at run time)
versus when the function is called. I think that gets to the heart of the
issue, not whether the capture copies a value or a reference. At some
point, the capture must make use of the value: whether it does so via a
direct C-style memory location copy, or an object pointer, or some other
mechanism, is irrelevant. What matters is whether that value is grabbed
at the time the closure is created, or when the closure is called.
That is conflating two entirely distinct subjects. Python 1.5 had duck-
typing but no closures, so the one certainly does not depend on the other.
Duck-typing is more a philosophy than a language feature: the language
must be typed, and the programmer must prefer to program to a protocol or
a specification rather than to membership of a type.
binding, but whichever it is, it is generally considered to be a bad
idea, at least as the default or only behaviour.
Bash is another language with dynamic binding. Some very old versions of
Lisp use dynamic binding, because it was the easiest to implement. Most
modern languages use lexical (also known as static) binding, because it
is more sensible.
Here is an illustration of the difference: suppose we have two modules,
library.py and main.py:
# library.py
x = 23
def func(y):
return x + y
# main.py
import library
x = 1000
print func(1)
If main.py prints 24 (and it does), then Python is using lexical scoping.
But if it prints 1001 (which it doesn't), then it is using dynamic
scoping. The obvious problem with dynamic binding is that the behaviour
of a function may vary depending on where you call it.
I don't think "by reference" versus "by value" are good terms to use
here, since they risk conflating the issue with "call by reference"
versus "call by value" semantics. I prefer "late" versus "early", as you
suggest below.
Not necessarily *compile* time, but the distinction is between when the
function is defined (which may at compile time, or it may be at run time)
versus when the function is called. I think that gets to the heart of the
issue, not whether the capture copies a value or a reference. At some
point, the capture must make use of the value: whether it does so via a
direct C-style memory location copy, or an object pointer, or some other
mechanism, is irrelevant. What matters is whether that value is grabbed
at the time the closure is created, or when the closure is called.
That is conflating two entirely distinct subjects. Python 1.5 had duck-
typing but no closures, so the one certainly does not depend on the other.
Duck-typing is more a philosophy than a language feature: the language
must be typed, and the programmer must prefer to program to a protocol or
a specification rather than to membership of a type.