Twisted (or for loops ?) madness

[looping]

Hi,
Probably not the best group to post my question but I'm sure there is
some people here that use Twisted.
First here is the beginning of my source code:

from twisted.internet import reactor, defer, threads
import time

class CompilerThread(object):
def __init__(self, task, delay):
self.task = task
self.delay = delay

def _processing(self, delay):
print 'Start :', self.task
# Simulate delayed result, to fire immediately use
self.d.callback(self.task)
time.sleep(delay)
return self.task

def compile(self):
print 'Compile :', self.task
print self
# Create Deferred in another thread and add callback
self.d = threads.deferToThread(self._processing,
self.delay).addCallback(self.print_result)
# Return the deferred, this way you could add callback later
return self.d

def print_result(self, result):
# Print result
print 'Compiler result :', result, self.task
# MUST return result otherwise next callback receive None
return result

# Create Compiler objects
ct1 = CompilerThread('*OBJECT 1*', 2)
ct2 = CompilerThread('*OBJECT 2*', 3)
ct3 = CompilerThread('*OBJECT 3*', 5)

# Use succeed to create a deferred already fired
d = defer.succeed(None)

Now my problem:
With this code everything work fine:

d.addCallback(lambda result: ct1.compile())
d.addCallback(lambda result: ct2.compile())
d.addCallback(lambda result: ct3.compile())

reactor.callLater(20, reactor.stop)
reactor.run()

Output:

Compile : *OBJECT 1*
<__main__.CompilerThread object at 0x00BAD070>
Start : *OBJECT 1*
Compiler result : *OBJECT 1* *OBJECT 1*
Compile : *OBJECT 2*
<__main__.CompilerThread object at 0x00BAD050>
Start : *OBJECT 2*
Compiler result : *OBJECT 2* *OBJECT 2*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CDA4B0>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*


But when I try to replace this code with a for loops, something goes
wrong:

l = [ct1, ct2, ct3]
for c in l:
d.addCallback(lambda result: c.compile())

reactor.callLater(20, reactor.stop)
reactor.run()

Output:

Compile : *OBJECT 1*
<__main__.CompilerThread object at 0x00BAD030>
Start : *OBJECT 1*
Compiler result : *OBJECT 1* *OBJECT 1*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*

OBJECT 3 run 2 times and OBJECT 2 never ?!?

Any idea ? Maybe something related to Threads ?
Thanks for your help.

 

[looping]

l = [ct1, ct2, ct3]
for c in l:
d.addCallback(lambda result: c.compile())

reactor.callLater(20, reactor.stop)
reactor.run()

Output:

Compile : *OBJECT 1*
<__main__.CompilerThread object at 0x00BAD030>
Start : *OBJECT 1*
Compiler result : *OBJECT 1* *OBJECT 1*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*

OBJECT 3 run 2 times and OBJECT 2 never ?!?

Any idea ? Maybe something related to Threads ?
Thanks for your help.

After further tests, it look like it is the lambda that cause the
problem:
-Adding a parameter result to compile(self, result)
-Changing d.addCallback(lambda result: c.compile()) for
d.addCallback(c.compile)

And everything run fine.

Why lambda doesn't work ? (variable scope problem ?)

 

[Diez B. Roggisch]

l = [ct1, ct2, ct3]
for c in l:
d.addCallback(lambda result: c.compile())

reactor.callLater(20, reactor.stop)
reactor.run()

Output:

Compile : *OBJECT 1*
<__main__.CompilerThread object at 0x00BAD030>
Start : *OBJECT 1*
Compiler result : *OBJECT 1* *OBJECT 1*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*

OBJECT 3 run 2 times and OBJECT 2 never ?!?

Any idea ? Maybe something related to Threads ?
Thanks for your help.

After further tests, it look like it is the lambda that cause the
problem:
-Adding a parameter result to compile(self, result)
-Changing d.addCallback(lambda result: c.compile()) for
d.addCallback(c.compile)

And everything run fine.

Why lambda doesn't work ? (variable scope problem ?)

Yes. When defined, the lambda inherits the surrounding scope - which
contains the name c, that is bound to the last instance of the iteration.

What you need to do is to introduce a scoped variable inside the lambda at
creation time - e.g. like this:

for a in 1,2,3:
d.addCallback(lambda arg=a: work(a))

Diez

 

[Michele Simionato]

Why lambda doesn't work ? (variable scope problem ?)

This is a well known issue of for loops. Some believe it to be a bug
but Guido says it
is a design decision, in the sense that Python always do late binding.
If you
browse this list you will find many discussions. Basically if you do

funclist = []
for i in 1,2,3:
def f():
print i
funclist.append(f)

you will get funclist[0]() == funclist[1]() == funclist[2]() == 3 (you
get the latest
binding of "i"). As you see, it has nothing to do with lambdas.


Michele Simionato

 

[looping]

is a design decision, in the sense that Python always do late binding.
If you

you will get funclist[0]() == funclist[1]() == funclist[2]() == 3 (you
get the latest
binding of "i"). As you see, it has nothing to do with lambdas.

Thanks Diez, replacing my addCallback with d.addCallback(lambda
result, comp=c: comp.compile()) do the trick.

So if I understand what Michele wrote (thanks too), when a function is
defined (with def), no scope is saved and every variable value not
passed in parameter is lost ? It means that variable value come from
the outer scope when the function is called ?

 

[Michele Simionato]

So if I understand what Michele wrote (thanks too), when a function is
defined (with def), no scope is saved and every variable value not
passed in parameter is lost ? It means that variable value come from
the outer scope when the function is called ?

Yes, in my example you get the value of "i" at the function *calling*
time,
not at the function definition time. If you want to store the value at
the
definition time, you must use the default argument trick:

funclist = []
for i in 1,2,3:
def f(i=i):
print i
funclist.append(f)

Michele Simionato

 

[looping]

So if I understand what Michele wrote (thanks too), when a function is
defined (with def), no scope is saved and every variable value not
passed in parameter is lost ? It means that variable value come from
the outer scope when the function is called ?

Yes, in my example you get the value of "i" at the function *calling*
time,
not at the function definition time. If you want to store the value at
the
definition time, you must use the default argument trick:

funclist = []
for i in 1,2,3:
def f(i=i):
print i
funclist.append(f)

Michele Simionato

Thanks Michele, now I understand how it works and I learned something
new. Not a bad day...