send() to a generator in a "for" loop with continue(val)??

[Dale Roberts]

I've started using generators for some "real" work (love them!), and I
need to use send() to send values back into the yield inside the
generator. When I want to use the generator, though, I have to
essentially duplicate the machinery of a "for" loop, because the "for"
loop does not have a mechanism to send into the generator. Here is a
toy example:

def TestGen1():
for i in xrange(3):
sendval = yield i
print " got %s in TestGen()" % sendval

g = TestGen1()
sendval = None
try:
while True:
val = g.send(sendval)
print 'val in "while" loop %d' % val
sendval = val * 10
except StopIteration: pass

I have to explicitly create the generator with an assignment, send an
initial None to the generator on the first go, then have to catch the
StopIteration exception. In other words, replicate the "for"
mechanism, but use send() instead of next().

It would be nice if I could just do this instead:

for val in TestGen1():
print 'val in "for" loop %d' % val
continue(val*10)

....or something similar. Is this an old idea? Has it been shot down in
the past already? Or is it worth pursuing? I Googled around and saw
one hit here: http://mail.python.org/pipermail/python-ideas/2009-February/003111.html,
but not much follow-up.

I wonder if people want to keep the idea of an "iterator" style
generator (where send() is not used) separate from the idea of a "co-
routine" style generator (where send() is used). Maybe combining the
two idioms in this way would cause confusion?

What do folks think?

dale

 

[Aaron Brady]

I've started using generators for some "real" work (love them!), and I
need to use send() to send values back into the yield inside the
generator. When I want to use the generator, though, I have to
essentially duplicate the machinery of a "for" loop, because the "for"
loop does not have a mechanism to send into the generator. Here is a
toy example:

def TestGen1():
    for i in xrange(3):
        sendval = yield i
        print "   got %s in TestGen()" % sendval

g = TestGen1()
sendval = None
try:
    while True:
        val = g.send(sendval)
        print 'val in "while" loop %d' % val
        sendval = val * 10
except StopIteration: pass

I have to explicitly create the generator with an assignment, send an
initial None to the generator on the first go, then have to catch the
StopIteration exception. In other words, replicate the "for"
mechanism, but use send() instead of next().

It would be nice if I could just do this instead:

for val in TestGen1():
    print 'val in "for" loop %d' % val
    continue(val*10)

...or something similar. Is this an old idea? Has it been shot down in
the past already? Or is it worth pursuing? I Googled around and saw
one hit here:http://mail.python.org/pipermail/python-ideas/2009-February/003111.html,
but not much follow-up.

I wonder if people want to keep the idea of an "iterator" style
generator (where send() is not used) separate from the idea of a "co-
routine" style generator (where send() is used). Maybe combining the
two idioms in this way would cause confusion?

What do folks think?

dale

You can do it with a wrapping generator. I'm not sure if it
interferes with your needs. It calls 'next' the first time, then just
calls 'send' on the parameter with the value you send it.
.... x= True
.... for _ in range( 10 ):
.... i= yield( x )
.... if i is None:
.... i= not x
.... x= i
........ x= next( gen )
.... while 1:
.... x= yield( x )
.... x= gen.send( x )
........ print( i, x )
.... i+= 1
.... if not i% 3:
.... _= a.send( False )
....
0 True
1 False
2 True
3 True
4 False
5 True
6 True
7 False

As you can see, it skips a beat every third iteration as directed.

 

[Peter Otten]

I've started using generators for some "real" work (love them!), and I
need to use send() to send values back into the yield inside the
generator. When I want to use the generator, though, I have to
essentially duplicate the machinery of a "for" loop, because the "for"
loop does not have a mechanism to send into the generator. Here is a
toy example:

def TestGen1():
for i in xrange(3):
sendval = yield i
print " got %s in TestGen()" % sendval

g = TestGen1()
sendval = None
try:
while True:
val = g.send(sendval)
print 'val in "while" loop %d' % val
sendval = val * 10
except StopIteration: pass

I have to explicitly create the generator with an assignment, send an
initial None to the generator on the first go, then have to catch the
StopIteration exception. In other words, replicate the "for"
mechanism, but use send() instead of next().

It would be nice if I could just do this instead:

for val in TestGen1():
print 'val in "for" loop %d' % val
continue(val*10)

...or something similar. Is this an old idea? Has it been shot down in
the past already? Or is it worth pursuing? I Googled around and saw
one hit here:
http://mail.python.org/pipermail/python-ideas/2009-February/003111.html,
but not much follow-up.

I wonder if people want to keep the idea of an "iterator" style
generator (where send() is not used) separate from the idea of a "co-
routine" style generator (where send() is used). Maybe combining the
two idioms in this way would cause confusion?

What do folks think?

If it were up to me I'd rip out send() immediatly. At first I thought I
would see a compelling use case and be enlightened, but it never happened.

I just grepped the python 3 source for '= yield' and the only matches were
in the unit tests for generators and parser.

Let it die while it hasn't polluted the wider code base.

Peter

 

[Aahz]

If it were up to me I'd rip out send() immediatly. At first I thought I
would see a compelling use case and be enlightened, but it never happened.

Too late -- it's likely to get an upgrade for 3.1 and 2.7. Basically,
send() is useful for coroutines, and if you haven't yet read
http://dabeaz.com/coroutines/
you really should (assuming you want to continue arguing).

 

[Michele Simionato]

Too late -- it's likely to get an upgrade for 3.1 and 2.7.  Basically,
send() is useful for coroutines, and if you haven't yet readhttp://dabeaz..com/coroutines/
you really should (assuming you want to continue arguing).
--

I suspect Peter knows everything about coroutines and still he is not
convinced about
..send. FWIW, I am sympathic with him.

M. Simionato

 

[Aahz]

I suspect Peter knows everything about coroutines and still he is not
convinced about .send. FWIW, I am sympathic with him.

Okay, I'm curious, is the argument that you shouldn't use generators for
coroutines or something else?

 

[Michele Simionato]

Okay, I'm curious, is the argument that you shouldn't use generators for
coroutines or something else?

Yes, there is certainly the confusion between generators
and coroutines (the ones David Beazley warns about).

At first, I was a supported of yield
expressions. After they entered in Python 2.5 and I
had occasion to play with them in practice, I realized that we
did not gain much from them. In an ideal world (one
that would never happen) I would like to remove
yield expressions and keep generators simple. OTOH,
I would like a simple coroutine library in the standard
library, with a trampoline and a scheduler. Coroutines,
however, would be implemented as simple objects with
a .send and .recv method, not by abusing yield expression.
May be we would lose a tot percent of performance, but
that should not be a concern for a Pythonista.
I dunno if Peter Otten thinks the same or has other
ideas, but this is what I have in mind. The important
addition in Python 2.5 was the introduction of
GeneratorExit and the ability to use yield inside
try .. finally blocks. This was a needed language level change.
Coroutines instead could have been implemented as
a library, without requiring any language change.

But the point is moot, anyway, and certainly I did not
think of it at the time yield expressions were introduced.

Michele Simionato

 

[Michele Simionato]

Coroutines instead could have been implemented as
a library, without requiring any syntax change.

Here is a proof of principle, just to make clearer
what I have in mind. Suppose you have the following
library:

$ cat coroutine.py
from abc import abstractmethod

class Coroutine(object):
@abstractmethod
def main(self):
yield

def send(self, value):
if not hasattr(self, '_it'): # first call
self._it = self.main()
self._value = value
return self._it.next()

def recv(self):
return self._value

def close(self):
self._it.close()
del self._it
del self._value

Then you could write coroutines as follows:

$ cat coroutine_example.py
from coroutine import Coroutine

class C(Coroutine):
def main(self):
while True:
val = self.recv()
if not isinstance(val, int):
raise TypeError('Not an integer: %r' % val)
if val % 2 == 0:
yield 'even'
else:
yield 'odd'

if __name__ == '__main__':
c = C()
for val in (1, 2, 'x', 3):
try:
out = c.send(val)
except StopIteration:
break
except Exception, e:
print e
else:
print out

This is not really worse of what we do today with the yield
expression:

$ cat coroutine_today.py
def coroutine():
val = yield
while True:
if not isinstance(val, int):
raise TypeError('Not an integer: %r' % val)
if val % 2 == 0:
val = yield 'even'
else:
val = yield 'odd'

if __name__ == '__main__':
c = coroutine()
c.next()
for val in (1, 2, 'x', 3):
try:
out = c.send(val)
except StopIteration:
break
except Exception, e:
print e
else:
print out

Actually it is clearer, since it avoids common mistakes
such as forgetting the ``val = yield`` line in the coroutine
function and the ``c.next`` line right after instantiation
of the coroutine object, which are needed to initialize
the coroutine correctly.

 

[Peter Otten]

Could you give some details or a link?

I only just started reading Beazley's presentation, it looks interesting.
Thanks for the hint!

Are you currently using coroutines in Python? If so, what kind of practical
problems do they simplify for you?

Okay, I'm curious, is the argument that you shouldn't use generators for
coroutines or something else?

I don't know nearly as much about coroutines as Michele thinks; in
particular I have no practical experience with them.

What I've seen so far is that the once beautifully simple generator
mechanism has become very complex with the recent additions for ressource
management and coroutines.

Generators as filters and synthetic sequences are now ubiquitous, the with
statement is spreading like wildfire.

The send()/yield-expression duo on the other hand is limping along, and
someone like Michele who is definitely in the "intended audience" for the
more arcane features of Python says that you can do it with a library.
If that is possible shouldn't it have been the first step to put such a
library into the stdlib and see how it fares?

Generators at the moment seem to have turned into what in German we call
an "eierlegende Wollmilchsau"*.

Peter

(*)
http://www.pollux.franken.de/fileadmin/user_upload/images/eier-legende-wollmilchsau.jpg

 

[Aahz]

Could you give some details or a link?

http://mail.python.org/pipermail/python-ideas/2009-April/004189.html

(You'll need to backtrack considerably to see the full discussion.)

Are you currently using coroutines in Python? If so, what kind of practical
problems do they simplify for you?

I'm not; I avoid coroutines like the plague. ;-) I much prefer nice,
simple threading...

What I've seen so far is that the once beautifully simple generator
mechanism has become very complex with the recent additions for ressource
management and coroutines.

The send()/yield-expression duo on the other hand is limping along, and
someone like Michele who is definitely in the "intended audience" for the
more arcane features of Python says that you can do it with a library.
If that is possible shouldn't it have been the first step to put such a
library into the stdlib and see how it fares?

You have a point; I wasn't paying much attention when send() was first
added, so I can't argue for it. I'm just referring you to material I
happen to have handy that discusses use for it.

 

[Aahz]

Generators at the moment seem to have turned into what in German we call
an "eierlegende Wollmilchsau"*.

One more point: there has always been a tension within the Python
community between people pushing the bleeding edge and people who prefer
to keep things simple. Unfortunately, the former group tends to be much
more active. :-( There really needs to be a solid core of people who
are fundamentally conservative. I like to think that I'm one of them
(and probably so do many other people in the python-dev community), but
each of us has our own hobby horses that we like to push, and none of us
keeps completely on top of everything.

 

[Dale Roberts]

You can do it with a wrapping generator.  I'm not sure if it
interferes with your needs.  It calls 'next' the first time, then just
calls 'send' on the parameter with the value you send it.

Aaron,

Thanks for the hint. I'd made a modified version of my generator that
was "for loop aware" and had two yields in it, but this seemed very
fragile and hackish to me, and left my generator only usable inside a
"for" loop.

The wrapper method seems to be a much better way to go.

dale

 

[Dale Roberts]

...
I only just started reading Beazley's presentation, it looks interesting.
Thanks for the hint!

Are you currently using coroutines in Python? If so, what kind of practical
problems do they simplify for you?

I thought I'd chime in with an application too. I am using this
mechanism to implement a state machine. I read through Beazley's
presentation too - wow, lots of ideas in there.

For my simple state machine, I am using a very simple "trampoline"
function (see his slides starting at about #172). My "run" routine is
a bit different, but the idea is similar.

I'm using this to present images to a test subject (a person looking
at a computer screen), and the person's responses guide the state
machine. So I need to get data in (the subject responses) and out (the
next image to be presented).

So I have violated The Beazley Principle of slide #195:

Keeping it Straight
• If you are going to use coroutines, it is critically
important to not mix programming paradigms
together
• There are three main uses of yield
• Iteration (a producer of data)
• Receiving messages (a consumer)
• A trap (cooperative multitasking)
• Do NOT write generator functions that try to
do more than one of these at once

....whoops!

But I think this is a valid use of the mechanism, in that it is very
localized and self contained to just the few routines that make up the
state machine. It works very well, makes it easy to implement the
state machine clearly, and is easy to understand and maintain.

I can see where it could get very confusing to use this mechanism in a
more general way.

dale