Feature request: subclassing FunctionType [Was: Some language proposals]

[Michele Simionato]

I have read with interest the recent thread about closures. The funny
thing is that the authors are arguing one against the other but I
actually agree with all of them and I have a proposal that may
be of some interest. To refresh your memory, I report here some
quotation from that thread.

Jacek Generowicz:

I sumbit to you that read-only closures are rare in Python because
they are a recent addition to the language.

I submit to you that read-write closures are so much rarer still,
because they require an ugly hack to work.

Paul Prescod:

I disagree. Closures are rare in Python because Python is primarily an
OOP language.

Micheal Hudson:

Using closures to fake objects sucks.

Jacek Generowicz:

When I need "objects" I use objects, and when I need closures I want
to be able to use closures.
[...] I have some stateful methods of classes (which I create
dynamically and stick onto the class as the information about the
existence of the method becomes available). By implementing them as
closures I can just stick them on to the class. If I were to implement
them as instances then I'd have to reimplement all the descriptors
that take care of turning functions into bound or unbound methods.

[At this point I'd love someone to step up and show me how to re-use
the existing FunctionType descriptors in my own classes.]

Another example. I make quite heavy use of a trivial memoizer. The
closure verison is much shorter, clearer and faster.

Paul Prescod:

Before Python had nested scopes at all, it was VERY common to fake them
using a linguistic trick: default arguments. This idiom was very common
in Python's source base. Python didn't have a proper way to do the
nesting but people figured out how to do it anyway and did it a LOT.
(similarly people often emulate OO in C)

The "wrap your mutables" feature is by comparison _much less intrusive_
but I don't ever see it in real Python code. This suggests to me that
people don't need the feature that badly.

There is some merit on all those points of view. Speaking for myself,
I would certainly use more closures if they were better supported by
the language. The fact that closures are read-only don't bothers me too
much, since I also think that when you want write access to a closure
you are really trying to fake an object, and you are better off using
a real object. On the other hand, I am really bothered by the scoping
rules. My first post on this mailing list some time ago was caused by
problems I had with the surprising scope rules. Take for instance
this example:

def make_adders(n):
return [(lambda x: x+i) for i in range(n)]

add0,add1=make_adders(2)

print add0(0) #=> 1
print add1(0) #=> 1

This does not work as expected, and the solution is an ugly hack
with default arguments:

def make_adders(n):
return [(lambda x,i=i: x+i) for i in range(n)]

add0,add1=make_adders(2)

print add0(0) #=> 0
print add1(0) #=> 1

I don't like that, but I don't think that the current scope rules
will change any soon. Alternatively, I can use a class:

class adder(object):
def __init__(self,i):
self.i=i
def __call__(self,x):
return x+self.i

def make_adders(n):
return [adder(i) for i in range(n)]

add0,add1=make_adders(2)

print add0(0) #=> 0
print add1(0) #=> 1

However, this is not really a solution, since as Jacek pointed
out, to really fake a function I need to reimplement the
descriptor protocol and I cannot get it from FunctionType.
I have experience with the examples he cites, memoizing
functions and wrapping methods, and they are exactly the
examples where I have used closures instead of callable objects.
OTOH, Paul Prescod is right and Python is primarily an
OOP language, so I would prefer to use real objects over
closures.

What I think would help is the ability to subclass FunctionType.
I remember a thread of some time ago where Tim Peters said
that it is quite doable, simply nobody bothered to implement
it yet.

At the moment, I can fake the built-in FunctionType with something like
that:

class function(object):
def __init__(self,func):
self.__func__=func
def __call__(self,*args,**kw):
return self.__func__(*args,**kw)
def __get__(self,obj,cls=None):
return self.__func__.__get__(obj,cls)

Then I can subclass "function" to implement any kind of wrapped functions.
For instance, to trace functions:

class traced_function(function):
def __init__(self,func):
def traced_func(*args,**kw): # using an handy closure ...
print "Calling %s with arguments %s %s ..." % (func.__name__,args,kw)
result=func(*args,**kw)
print "Called %s with result %s" % (func.__name__,result)
return result
self.__func__=traced_func

Since "function" implements the descriptor protocol, I can wrap methods
for free:

class C(object):
add1=traced_function(lambda self,x: x+1)

C().add1(1)

My previous example with the adder would read

class adder(function):
def __init__(self,i):
self.__func__=lambda x: x+i

I think this reads quite well. At the present, however, I cannot subclass
FunctionType and I am a bit disturbed by that, also because I can subclass
the other built-ins types and there no reason why functions must be so
special: at they end they are just callable objects with a descriptor
protocol. So, I submit it as a feature request for Python 2.4.
Maybe, if enough people wants the feature, we may get it!

Michele Simionato

 

[Michael Hudson]

(e-mail address removed) (Michele Simionato) writes:

[...]

There is some merit on all those points of view. Speaking for myself,
I would certainly use more closures if they were better supported by
the language. The fact that closures are read-only don't bothers me too
much, since I also think that when you want write access to a closure
you are really trying to fake an object, and you are better off using
a real object. On the other hand, I am really bothered by the scoping
rules. My first post on this mailing list some time ago was caused by
problems I had with the surprising scope rules. Take for instance
this example:

def make_adders(n):
return [(lambda x: x+i) for i in range(n)]

add0,add1=make_adders(2)

print add0(0) #=> 1
print add1(0) #=> 1

This does not work as expected,

However, you would probably like this to work:

def make_mutual_rec():
def y1(x):
if x > 0: return y2(x/2)
else: return x
def y2(x):
if x > 0: return y2(x-2)
else: return x
return y1

make_mutual_rec()(3)

It seems to be difficult to reconcile the two points of view.

More reading:

http://www.python.org/~jeremy/weblog/040204.html
http://python.org/sf/872326

and the solution is an ugly hack with default arguments:

Or an object...

[...]

I don't like that, but I don't think that the current scope rules
will change any soon. Alternatively, I can use a class:
Ah...

]...]
I think this reads quite well. At the present, however, I cannot subclass
FunctionType and I am a bit disturbed by that, also because I can subclass
the other built-ins types and there no reason why functions must be so
special: at they end they are just callable objects with a descriptor
protocol. So, I submit it as a feature request for Python 2.4.
Maybe, if enough people wants the feature, we may get it!

I suspect that creating the patch wouldn't take much longer than
writing that article.

Cheers,
mwh

 

[Jacek Generowicz]

def make_adders(n):
return [(lambda x: x+i) for i in range(n)]

add0,add1=make_adders(2)

print add0(0) #=> 1
print add1(0) #=> 1

This does not work as expected,

Hmm, I think that what is expected is not an invariant between
different people and even between more careful or less careful
readings by the same person.

On careful reading, it does exatly what I expect it to do.

All the closures in the list share the same free-variable binding[*];
there is only one binding of i, so they all find that single binding.

I don't think that Python's behaviour differs from other languages, in
this respect. For example in Common Lisp:

* (defun make-adders (n)
(loop for i below n collect
(lambda (x) (+ x i))))
MAKE-ADDERS
* (destructuring-bind (add0 add1) (make-adders 2)
(list (funcall add0 0)
(funcall add1 0)))
(2 2)

[The important point is that the two resulting numbers are the same,
the fact that they are both twos rather than ones is a feature of the
looping construct and is irrelevant to the scoping issues.]

and the solution is an ugly hack
with default arguments:

def make_adders(n):
return [(lambda x,i=i: x+i) for i in range(n)]

add0,add1=make_adders(2)

print add0(0) #=> 0
print add1(0) #=> 1

Yes, if you want the free variables to be bound differently, then
you need to bind i in a seperate "more inner" scope for each closure:

(defun make-adders (n)
(loop for i below n collect
(let ((i i))
(lambda (x) (+ x i)))))
MAKE-ADDERS
*
(destructuring-bind (add0 add1) (make-adders 2)
(list (funcall add0 0)
(funcall add1 0)))
(0 1)

I don't like that, but I don't think that the current scope rules
will change any soon.

I'm not sure how you would like them to change.

OTOH, Paul Prescod is right and Python is primarily an
OOP language, so I would prefer to use real objects over
closures.

Bah! Humbug! It's multi-paradigm, I tell you

So, I submit it as a feature request for Python 2.4. Maybe, if
enough people wants the feature, we may get it!

In complete ignorance of the technical difficulties and second order
consequences of allowing this, is second your request.


[*] I have a feeling that "free variable binding" is not a correct way
of expressing this, but I can't think of anything better just now.

 

[Michele Simionato]

I don't think that Python's behaviour differs from other languages, in
this respect. For example in Common Lisp:

* (defun make-adders (n)
(loop for i below n collect
(lambda (x) (+ x i))))
MAKE-ADDERS
* (destructuring-bind (add0 add1) (make-adders 2)
(list (funcall add0 0)
(funcall add1 0)))
(2 2)

I don't know Common Lisp, but Scheme scope rules do what I expect:

(define (make-adders n)
(unfold (cut = <> n) (lambda (i) (lambda (x) (+ x i))) add1 0))

(define-values (add0 add1) (apply values (make-adders 2)))

(print (add0 0)) ;=> 0
(print (add1 0)) ;=> 1

My feeling about Python scope rules are mixed: in the beginning, I thought
they were just broken, since they were not following my expectations
(I don't know where those expectations were coming from, maybe from
Pascal? I didn't know any functional language at that time, mah!).
Later on, when I learned about the default argument tricks, I thought
there was a logic in there, i.e. I may choose if I want to use the value
of the "i" argument at the definition time or at the usage time
explicitly and "explicit is better than implicit". However, now that
I see that there are languages that implement the scope rule I
would expect, and since I don't see a real life case where Python
default of using the value of "i" as the usage time is useful, I am
starting questioning them again.
The only reason I see for the present state of the scope rules, is
to discourage people from using closures and using objects instead.
Which is not necessarely a bad thing, yes, but then I would need a better
support from the language, and the ability to subclass FunctionType
to be happy again

Michele Simionato

 

[Bengt Richter]

On 26 Feb 2004 21:58:41 -0800, (e-mail address removed) (Michele Simionato) wrote:
[...]

a real object. On the other hand, I am really bothered by the scoping
rules. My first post on this mailing list some time ago was caused by
problems I had with the surprising scope rules. Take for instance
this example:

def make_adders(n):
return [(lambda x: x+i) for i in range(n)]

add0,add1=make_adders(2)

print add0(0) #=> 1
print add1(0) #=> 1

This does not work as expected, and the solution is an ugly hack
with default arguments:

I got misled by those near-each-other i's too, but though they both mean use-the-x-binding-in-
a-particular-environment, the use is not at the same time. The code from the lambda
expression evaluation does not look for i until it gets executed at some future time that could be
arbitrarily long after the list comprehension is done with its i-value side effect in the local
namespace of make_adders.

I agree it is easy to make the wrong assumption, with those i's seemingly in the
same expression context, but what should be the expectation? I think the problem is that
lambda x: x+i is a lambda expression, that results in a function, but the i is a name,
not a value in the context of the lambda expression per se. Thus the result is
code that looks for the value of a name someplace, not code that refers to a
specific value. If you want the name lookup to find different values, you must
provide different name bindings, which you don't get in the single closure
reflecting the environment of the list comprehension in make_adders above.

But you can generate multiple closures with separate values
(which maybe your scheme code really is doing?):
... return [(lambda sepval: lambda x: x + sepval)(i) for i in range(n)]
... 1

Here the lambda being _executed_ as well as _evaluated_ in the list comprehension provides
the closure environment for the lambda expression only being _evaluated_ (inside the one
being executed, to generate code looking for sepval). UIAM ;-)

[...]

I think having function instances be first-class instances of a first-class function class would
be a unifying move. (First-class class sounds funny) I wonder if we can integrate the creation
of generators and coroutines (and even threads?) nicely in that, in some hierarchy. I'd like to
think about that some, but no time for now. Ciao ;-)

Regards,
Bengt Richter

 

[Michele Simionato]

But you can generate multiple closures with separate values
(which maybe your scheme code really is doing?):
... return [(lambda sepval: lambda x: x + sepval)(i) for i in range(n)]
...1

Here the lambda being _executed_ as well as _evaluated_ in the list comprehension provides
the closure environment for the lambda expression only being _evaluated_ (inside the one
being executed, to generate code looking for sepval). UIAM ;-)

Hi Bengt! You reduced your contribution to this list in the
last few months, it is nice to see you back

For what concerns the trick you are talking about, it is the workaround
I often use in this situation. Of course, I don't nest the two lambdas
together and I write something more like

def adder(sepval):
return lambda x : x + sepval

def make_separate_adders(n):
return [adder(i) for i in range(n)]

which does the right thing and is readable too. But still it is
easy to forget that the helper function is needed and that using
directly the lambda would not work. It is kind of error prone.
Scheme is less error prone since it uses local loop variables and
automatically generates separate closures:

(define list-of-adders '())

(let loop ((i 0))
(set! list-of-adders (cons (lambda (x) (+ x i)) list-of-adders))
(if (= i 1) 'exit (loop (add1 i))))

(define add1 (first list-of-adders))
(define add0 (second list-of-adders))

(print (add0 0)) ;=> 0
(print (add1 0)) ;=> 1

You can get Python behavior if you explicitely use a global variable:

(define list-of-adders '())
(define i 0)

(let loop ()
(set! list-of-adders (cons (lambda (x) (+ x i)) list-of-adders))
(if (= i 1) 'exit (begin (set! i (add1 i)) (loop))))

(define add1 (first list-of-adders))
(define add0 (second list-of-adders))

(print (add0 0)) ;=> 1
(print (add1 0)) ;=> 1

So, in a sense, Python and Scheme have the same scope
rules here, but in Scheme you don't see the issue
because you just use a local variable, whereas
Python by default use a global loop variable.


Michele Simionato

 

[Jacek Generowicz]

I don't know Common Lisp, but Scheme scope rules do what I expect:

(define (make-adders n)
(unfold (cut = <> n) (lambda (i) (lambda (x) (+ x i))) add1 0))

(define-values (add0 add1) (apply values (make-adders 2)))

(print (add0 0)) ;=> 0
(print (add1 0)) ;=> 1

I think that you are misleading yourself. You are comparing apples to
quinces.

Let's try to reproduce your Scheme code in Python. (I'm no Schemer,
but here goes ...). Googling for "scheme unfold" comes up with
something which I can fake in Python more or less like this:

def unfold(p, f, g, seed):
if p(seed):
return []
return [f(seed)] + unfold(p, f, g, g(seed))

I'm guessing that add1 does this:

def scheme_add1(n): # don't name-clash with the add1 we're about to
make
return n+1

Allow me to simplify cut to the following, which I hope you will agree
does what you want in this particular case:

def cut(limit): # [*]
def cut(x):
return not x<limit
return cut

Now our user code becomes

def make_adders(n):
return unfold(cut(n), lambda i: lambda x:x+i, scheme_add1, 0)

add0, add1 = make_adders(2)

print add0(0) # => 0
print add1(0) # => 1

Note: Python gives the same results as Scheme.

So, what's going on? The original problem was that all the closures
that you generated in your Python code shared a single binding for
their free variable. The solution is to create an inner binding unique
to each closure, which shadows the original shared binding. In your
Python solution you achieve this by creating a local variable i, which
shadows the outer i (so the i in the lambda is no longer free, and the
lambda is no longer a closure); in my CL solution I achieve this by
creating a new binding of i using let (which is still outside of the
scope of the lambda, so lambda's i remains free, and therefore it
remains a closure). In your Scheme solution, and my Python translation
of it, you'll see that we have:

lambda i: lambda x:x+i

which gets called by unfold to create the closure you want. Note that
you are rebinding i in the outer lambda, each time unfold calls it to
create a closure over the next number in the sequence, and that
binding is visible only to the single closure being returned from that
invocation of the outer lambda. So, to compare like to like, you
should do the same in your original Python attempt.

def make_adders(n):
return [ (lambda i: lambda x:x+i)(i) for i in range(n) ]

add0, add1 = make_adders(2)
print add0(0) # => 0
print add1(0) # => 1

So, you see, Python's scoping rules are just like Scheme's, in this
sense. Unfortunately, I don't think that it's so easy (or even
possible?) to demonstrate it the other way around (ie show a Scheme
snippet in which add0(0) == add1(0) == 1), because of Scheme's
insistence on using recursion to implement iteration. You see, in both
the Python and the Common Lisp versions of the original, there is a
single biding of i in the loop, which gets modified as the loop goes
about its business. In Scheme, the "loop variable" gets rebound each
time the looping function (eg unfold) calls itself, so for each and
every number in the sequence there is a seperate binding of seed, as
well as a separate binding of i.

Looking at it pictorially (variables with a * next to them are free,
those with a + are bound; where the value of the binding does not
change, the value is shown in parentheses; arrow up means binding
found by free variable, arrow down means binding by argument passing):

The Python case

list comprehension: i+
^
|
________ / \___
/ \
add0: i* add1: i*


Scheme style case

unfold: seed+ (0)
|
|
/ \
| ---(via scheme_add1)--
| \
V V
lambda: i+ (0) unfold: seed+ (1)
^ |
| V
add0: i* lambda: i+ (1)
^
|
add1: i*

The scoping rules in the two languages are pretty much the same, but
the scheme code throws in lots of extra scopes which are absent from
the (original) Python code.

If you can find a way of making a genuine loop in Scheme, you should
observe similar behaviour to that of the original Python example.

However, now that I see that there are languages that implement the
scope rule I would expect

I hope that you agree, by now, that Scheme is _not_ an example of such
a language.

If you need more convincing, how about the following?

guile> (define (make-adder i)
(list (lambda (x) (+ x i))
(lambda (n) (set! i (+ i n)))))
guile> (define tmp (make-adder 5))
guile> (define add5 (car tmp))
guile> (define hehe (cadr tmp))
guile> (add5 3)
8
guile> (hehe 1000)
guile> (add5 3)
1008

Conclusion: the value of i that add5 sees, is definitely _not_ fixed
at the time of creation of add5.

and since I don't see a real life case where Python default of using
the value of "i" as the usage time is useful,

You are effectively saying that you don't see the point of using the
run-time value of a variable, and that you'd rather be stuck with the
first value that was ever assigned to the variable. You'd never claim
this was the right thing for bound variables (including globals
referenced in a local scope), would you? And I don't believe that you
_really_ think it is appropriate for free variables either.

Try the following:

x = 3
def show(): global x; print x
show() # => 3
x += 1
show() # => 4

def outer():
x = 3
def show(): print x
show() # => 3
x += 1
show() # => 4
outer()

.... which prints out the numbers 3,4,3 and 4.

You don't really believe that the last number should be 3, do you ?

[In case anyone is in any doubt, the following is essentially
identical

def outer(x):
def show(): print x
show() # => 3
x += 1
show() # => 4
outer(3)
]

[*] def cut(limit):
def cut(x):
return not x<limit
return cut

Sorry, I should have written cut as follows, which is much clearer and
more Pythonic, of course

class cut:

def __init__(self,limit):
self.limit = limit

def __call__(self,x):
return not x<limit

 

[Michele Simionato]

I think that you are misleading yourself. You are comparing apples to
quinces.

I thank you for the long post and the time you spent in clarifying the
issue I had; however, you are speaking to a converted, since I already
came to the same conclusion on my own (see my reply to Bengt Richter).

So, you see, Python's scoping rules are just like Scheme's, in this
sense. Unfortunately, I don't think that it's so easy (or even
possible?) to demonstrate it the other way around (ie show a Scheme
snippet in which add0(0) == add1(0) == 1)

See my reply to Bengt.

Try the following:

x = 3
def show(): global x; print x
show() # => 3
x += 1
show() # => 4

def outer():
x = 3
def show(): print x
show() # => 3
x += 1
show() # => 4
outer()

... which prints out the numbers 3,4,3 and 4.

You don't really believe that the last number should be 3, do you ?

No, I agree with you on every respect. It is just that Scheme hides
the issue; but it hides it so well that in practice you never do
the mistake.
Still, I maintain that the ability to subclass FunctionType would be nice

Michele Simionato

 

[Michele Simionato]

Thinking a bit more, the issue is not about the scope rules, it is about
how the "for" loop is interpreted. Currently

for i in iterable:
<do_something i>

is interpreted as

try:
it=iter(iterable)
while True:
i=it.next()
<do_something i>
except StopIteration:
pass

The issue would disappear if the "for" loop included an hidden function
call and was interpreted as follows:

try:
it=iter(iterable)
def helper(i):
<do_something i>
while True:
helper(it.next())
except StopIteration:
pass

For instance, in the example I am talking about,

def make_adders(n):
return [lambda x: x+i for i in range(n)]

would be interpreted as

def make_adders(n):
try:
adders=[]
it=iter(range(2))
def helper(i):
adders.append(lambda x: x+i)
while True:
helper(it.next())
except StopIteration:
return adders

Essentially, the "i" variable would be passed via the helper function
call and at each iteration the lambda function would see a different
value of it.

I am proposing nothing here, just asking if it would make sense to
have a looping construct acting this way (my guess is that this
has already been proposed and discussed). This would have
the added benefit of avoiding a non-local loop variable (i.e. a
loop variable which exists even outside the loop) which is the
actual unfortunate behavior.

Michele Simionato

 

[Jacek Generowicz]

I thank you for the long post and the time you spent in clarifying the
issue I had; however, you are speaking to a converted, since I already
came to the same conclusion on my own (see my reply to Bengt Richter).

Ah, OK, sorry. Didn't see that one at the time I was writing.

Still, I maintain that the ability to subclass FunctionType would be nice

I agreed with you originally on that, and I still do

Just one thing bothers me about your conclusions in your reply to
Bengt ...

[As my discussion turned out longer than I originally intended, let me
pull the summary up here to the top:

In short, the reason Scheme hides the "feature" is that there is no
iteration in Scheme, only recursion.
]

But still it is easy to forget that the helper function is needed
and that using directly the lambda would not work. It is kind of
error prone.

Don't think about helper functions, think about the scopes that are
intoduced. It's not the function that matters, it's the scope (the
inner binding), which in Python can only be introduced by nesting a
function, while in Scheme and CL you can use let.

Scheme is less error prone since it uses local loop variables

The behaviour has nothing to do with globality or locality of the
variable which is used. It depends on whether the same binding is
being updated, or whether a new binding is being created each time
around the loop.

and automatically generates separate closures:

Python also greates separate closures. Scheme (and Python when you
instruct it to do so) creates separate bindings of the same name so
that the binding found by the free variables in the closure is
different for each closure.

(define list-of-adders '())

(let loop ((i 0))
(set! list-of-adders (cons (lambda (x) (+ x i)) list-of-adders))
(if (= i 1) 'exit (loop (add1 i))))

(define add1 (first list-of-adders))
(define add0 (second list-of-adders))
[...]

You can get Python behavior if you explicitely use a global variable:

(define list-of-adders '())
(define i 0)

(let loop ()
(set! list-of-adders (cons (lambda (x) (+ x i)) list-of-adders))
(if (= i 1) 'exit (begin (set! i (add1 i)) (loop))))

In the second case you are updating the _single_ (global as it
happens) binding [(set! i (add1 i))]. In the first case you do (loop
(add1 i)) which certainly looks like it's passing the new value to a
function which will create a new binding (though I suspect that
there's some macrology behind the scenes). If, in the first case, you
were to introduce a local variable to the loop and use (set! i (add1
i)) to update it, just like you do to the global, you'd see the
closures sharing the same value again.

As I said before, iteration in Scheme is usually (always?) implemented
via recursion, so it's difficult create a loop with a genuine looping
variable.

If you used recursion to implement iteration in Python, you'd get the
same as in Scheme:

def make_adders(i):
if i < 0: return []
return make_adders(i-1)+[lambda x:x+i]

add0, add1 = make_adders(1)

print add0(0) # => 0
print add1(0) # => 1



In short, the reason Scheme hides the "feature" is that there is no
iteration in Scheme, only recursion.


Disclaimer:

I am no Scheme expert. There may well be Schemes which support real
iteration, but I believe that the usual and even encouraged means of
iterating in Scheme is via recursion. (Note, this is _not_ the case
in Common Lisp.) Please correct me if you know this to be wrong.

 

[Jacek Generowicz]

Thinking a bit more, the issue is not about the scope rules, it is about
how the "for" loop is interpreted. Currently

Heh, this hadn't yet appeared on my server at the time I started
writing my last followup to you. It seems that you have spotted for
yourself, that which my article attempts to point out ... again

Essentially, the "i" variable would be passed via the helper function
call and at each iteration the lambda function would see a different
value of it.

The important point is that it would see a different _binding_ of that
name.

I am proposing nothing here, just asking if it would make sense to
have a looping construct acting this way

Scheme is over there =>

 

[Michele Simionato]

Heh, this hadn't yet appeared on my server at the time I started
writing my last followup to you. It seems that you have spotted for
yourself, that which my article attempts to point out ... again

Yes, but still your thoughtful explanations may be useful to others
who got catched by the same issues. Personally, I did understood
quite well how Python worked, but I didn't realized until this thread
that Scheme actually works the same, since the recursive loop hides
the effect. I also thank you for correcting my sloppy and/or
incorrect usage of the terminology. Nice discussion, anyway!


Michele

 

[Michele Simionato]

Thinking a bit more, the issue is not about the scope rules, it is about
how the "for" loop is interpreted. Currently

for i in iterable:
<do_something i>

is interpreted as

try:
it=iter(iterable)
while True:
i=it.next()
<do_something i>
except StopIteration:
pass

The issue would disappear if the "for" loop included an hidden function
call and was interpreted as follows:

try:
it=iter(iterable)
def helper(i):
<do_something i>
while True:
helper(it.next())
except StopIteration:
pass

For instance, in the example I am talking about,

def make_adders(n):
return [lambda x: x+i for i in range(n)]

would be interpreted as

def make_adders(n):
try:
adders=[]
it=iter(range(2))
def helper(i):
adders.append(lambda x: x+i)
while True:
helper(it.next())
except StopIteration:
return adders

Essentially, the "i" variable would be passed via the helper function
call and at each iteration the lambda function would see a different
value of it.

I am proposing nothing here, just asking if it would make sense to
have a looping construct acting this way (my guess is that this
has already been proposed and discussed). This would have
the added benefit of avoiding a non-local loop variable (i.e. a
loop variable which exists even outside the loop) which is the
actual unfortunate behavior.

Michele Simionato

Just to add another data point, I asked on comp.lang.functional and
discovered that Haskell list comprehension works as I would expect,
i.e. differently from Python:

Prelude> let make-adders n = [ \x -> x + i | i <- [0..n-1] ]
Prelude> let [add0,add1] = make_adders 2
Prelude> add0 0
0
Prelude> add1 0
1

So, it looks like as if Haskell implements the idea I exposed.
Still, it is to be discussed if the idea makes sense in Python.

1. Assuming all "for" loops and list comprehensions are replaced
with Haskell bindings rules, what about backward compatibility
problems? For instance: is there enough code relying on the loop
variable being available outside the loop?

2. If, due to compatibility issues, it is not possible to change the
binding rules of existing loops, would it make sense to propose
these binding rules for Python 2.4 generator comprehension?
This would not be backward incompatible, but would it generate
confusion?

3. Assuming the rules can be changed without real trouble, is the feature
worth enough? Do people care about it, and is there somebody willing
to take the job?


Michele Simionato

 

[Josiah Carlson]

1. Assuming all "for" loops and list comprehensions are replaced

with Haskell bindings rules, what about backward compatibility
problems? For instance: is there enough code relying on the loop
variable being available outside the loop?

Having a variable available outside of a loop can be both a convenience
and inconvenience, depending on the situation. I have used it before,
but acknowledge that it is poor form.

2. If, due to compatibility issues, it is not possible to change the
binding rules of existing loops, would it make sense to propose
these binding rules for Python 2.4 generator comprehension?
This would not be backward incompatible, but would it generate
confusion?

From what I can gather from the python-dev mailing list, generator
expressions seem to work the same as list comprehensions, without the
list. If I am incorrect, feel free to correct me.

3. Assuming the rules can be changed without real trouble, is the feature
worth enough? Do people care about it, and is there somebody willing
to take the job?

Changing the rules can cause real trouble. If loop variables were an
unintended consequence, were not desireable by a large portion of Python
users, and were used rarely, if at all, then I believe it is likely that
would have been removed already.

- Josiah

 

[Bob Ippolito]

(e-mail address removed) (Michele Simionato) wrote in message

Thinking a bit more, the issue is not about the scope rules, it is about
how the "for" loop is interpreted. Currently

for i in iterable:
<do_something i>

is interpreted as

try:
it=iter(iterable)
while True:
i=it.next()
<do_something i>
except StopIteration:
pass

The issue would disappear if the "for" loop included an hidden function
call and was interpreted as follows:

try: it=iter(iterable)
def helper(i):
<do_something i>
while True:
helper(it.next())
except StopIteration:
pass

For instance, in the example I am talking about,

def make_adders(n):
return [lambda x: x+i for i in range(n)]

would be interpreted as

def make_adders(n):
try:
adders=[]
it=iter(range(2))
def helper(i):
adders.append(lambda x: x+i)
while True:
helper(it.next())
except StopIteration:
return adders

Essentially, the "i" variable would be passed via the helper function
call and at each iteration the lambda function would see a different
value of it.

I am proposing nothing here, just asking if it would make sense to
have a looping construct acting this way (my guess is that this
has already been proposed and discussed). This would have
the added benefit of avoiding a non-local loop variable (i.e. a
loop variable which exists even outside the loop) which is the
actual unfortunate behavior.

Michele Simionato

Just to add another data point, I asked on comp.lang.functional and
discovered that Haskell list comprehension works as I would expect,
i.e. differently from Python:

Prelude> let make-adders n = [ \x -> x + i | i <- [0..n-1] ]
Prelude> let [add0,add1] = make_adders 2
Prelude> add0 0
0
Prelude> add1 0
1

So, it looks like as if Haskell implements the idea I exposed.
Still, it is to be discussed if the idea makes sense in Python.

1. Assuming all "for" loops and list comprehensions are replaced
with Haskell bindings rules, what about backward compatibility
problems? For instance: is there enough code relying on the loop
variable being available outside the loop?

2. If, due to compatibility issues, it is not possible to change the
binding rules of existing loops, would it make sense to propose
these binding rules for Python 2.4 generator comprehension?
This would not be backward incompatible, but would it generate
confusion?

It could always be a __future__

3. Assuming the rules can be changed without real trouble, is the feature
worth enough? Do people care about it, and is there somebody willing
to take the job?

I know that I've been bitten by this before, but after doing it once or
twice you learn how to get around it.. the biggest problem I have with
list comprehensions is that I think they should have the same scoping
rules as an inner function would, as opposed to a for loop (i.e.,
shouldn't pollute namespace).

-bob

 

[Aahz]

1. Assuming all "for" loops and list comprehensions are replaced
with Haskell bindings rules, what about backward compatibility
problems? For instance: is there enough code relying on the loop
variable being available outside the loop?

Yes. Guido has Pronounced that this will stay, but...

2. If, due to compatibility issues, it is not possible to change the
binding rules of existing loops, would it make sense to propose
these binding rules for Python 2.4 generator comprehension?
This would not be backward incompatible, but would it generate
confusion?

.....while the rules for generator expressions have not been finalized,
Guido has agreed that they will have their own scope and that listcomps
will be changed in Python 3 to have their own scope. You should read the
python-dev thread on generator expressions yourself to understand what's
being proposed.

 

[Terry Reedy]

1. Assuming all "for" loops and list comprehensions are replaced
with Haskell bindings rules, what about backward compatibility
problems? For instance: is there enough code relying on the loop
variable being available outside the loop?

Guido has given things like the following as an intentional reason for the
current rule (and for for...else):

for item in seq:
if acceptable(item): break
else:
item = None
# item is now either the first acceptible item in seq or None

If you do not like the iteration var remaining bound, you can explicitly
delete it. If it is automatically deleted, constructions like the above
are rendered more difficult.

A list comp encapulates a specialized for loop whose specific job is to
produce a list. It makes more sense to delete the binding automatically
there since a) there is no way, when seq is empty, to force a binding to
*something*, with else and b) the last item of the list is readily
available anyway as result[-1].

Terry J. Reedy