What is executed when in a generator

[Jerzy Karczmarczuk]

I thought that the following sequence

gl=0
def gen(x):
global gl
gl=x
yield x

s=gen(1)

suspends the generator just before the yield, so after the
assignment of s gl becomes 1.

Well, no. It is still zero. If I put

print "something"

before the yield, this doesn't get executed either. *EVERYTHING*
from the beginning until the yield gets executed only upon s.next().

Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.
Thank you.


Jerzy Karczmarczuk

 

[Fredrik Lundh]

Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.

the behaviour is described in the language reference manual:

http://docs.python.org/ref/yield.html

"When a generator function is called, it returns an iterator known as a
generator iterator, or more commonly, a generator. The body of the
generator function is executed by calling the generator's next()
method repeatedly until it raises an exception."

the above page points to the design document, which contains the full
story:

http://www.python.org/peps/pep-0255.html

"When a generator function is called, the actual arguments are bound to
function-local formal argument names in the usual way, but no code in
the body of the function is executed. Instead a generator-iterator
object is returned; this conforms to the iterator protocol /.../

Each time the .next() method of a generator-iterator is invoked, the
code in the body of the generator-function is executed until a yield
or return statement (see below) is encountered, or until the end of
the body is reached."

</F>

 

[Matt Hammond]

before the yield, this doesn't get executed either. *EVERYTHING*
from the beginning until the yield gets executed only upon s.next().

Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.
Thank you.

This is probably what you want:

http://www.python.org/doc/2.4.2/ref/yield.html

But you've basically got the idea anyway. I think the important thing is
to think of the generator as a factory. When called, it returns you
something you can iterate over by calling the next() method. Think of it
almost like instantiation. In fact, you could write something functionally
equivalent like this:

gl=0

class GeneratorLikeBehaviour:
def __init__(self, x):
self.state = 0
self.x = x

def next(self):
global gl
if self.state == 0:
g1 = x
self.state = 1
return x
else:
raise StopIteration

s=gen(1)


Unless I've made a typo, you should get exactly the same behaviour.

The difference under the bonnet is that calling the generator has less
overheads, as it is not a true function call - stack frames etc are not
having to be set up fully. Instead they are (presumably) set aside between
calls to s.next()

Hope this helps


Matt
--

| Matt Hammond
| R&D Engineer, BBC Research & Development, Tadworth, Surrey, UK.
| http://kamaelia.sf.net/
| http://www.bbc.co.uk/rd/

 

[Steve Holden]

I thought that the following sequence

gl=0
def gen(x):
global gl
gl=x
yield x

s=gen(1)

suspends the generator just before the yield, so after the
assignment of s gl becomes 1.

Well, no. It is still zero. If I put

print "something"

before the yield, this doesn't get executed either. *EVERYTHING*
from the beginning until the yield gets executed only upon s.next().

Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.
Thank you.

The first hing you need to realise is that s is a generator, and it
needs to be used in an iterative context to see the (sequence of)
yielded results:
<generator object at 0x4e028c>

The easiest way to do this is to use the generator in a loop - though of
course in this case the sequence of yielded results is going to be of
length 1 ...
... print o
...
1
It's easier to see the power of this when the generator yields several
results, either from a loop or otherwise:
... for i in range(n):
... if i % 2:
... yield i
... ... print i
...
1
3
5
The reason for this is so you can create multiple (parameterised)
generators using different calls:

>>> s1 = shg(3)
>>> s2 = shg(7)
>>> print [i for i in s1] [1]
>>> print [i for i in s2] [1, 3, 5]
>>>

Fredrik Lundh has already pointed you to the full description, so I'll
content myself with adding that you can, if you want to you can call the
generator's next() method to access the next in its sequence of results:

>>> s1 = shg(3)
>>> s2 = shg(7)
>>> print [(i, s2.next()) for i in s1] [(1, 1)]
>>> print [i for i in s2]

[3, 5]

Hope this makes things a little clearer.

regards
Steve

 

[Jerzy Karczmarczuk]

Thank you all for some precisions about yield and generators.
But I would like to underline that I am *not* a complete newbie,
and I understand this stuff in general. I read the yield.html
material quite a time ago. But thank you for the PEP, I should
have scanned it more carefully.

My problem was the *context of suspension*. You see, when some text
speaks about the preservation of local state, etc., *a priori* the
following sequence

def gen()
a
b
c
yield x
d
e
f
yield y

*COULD BE* understood as follows. Upon the call s=gen(), a, b and c
*get* executed, change the global state, install some local, and then
the system makes the snapshot, and returns the generator with its
context. The call to next returns x to the caller, but the generator
works for some time, and executes d, e and f before the next suspension.

Now I know that this is not true. The CO- flag of the gen() procedure
inhibits the execution of the code altogether, and a, b, c are executed
upon the first s.next; d, e and f - upon the second next. Etc. OK,
that's it, I accept such behaviour, although the alternative could be
interesting as well.

====

The first hing you need to realise is that s is a generator, and it
needs to be used in an iterative context to see the (sequence of)
yielded results:

This is not exact, this is a *particular* consumer view of generators.
I don't want them for iterators, I use them as emulators of *lazy
programming*, I implement some co-recursive algorithms with them.
So i use next() when I wish, and never 'for'.

Thank you once more.

Jerzy Karczmarczuk

 

[Michael]

[ I've got no idea of your skill level, but since you say you feel a bit
lost I'll take baby steps here. Apologies if it's too low or high
I'm also taking baby steps because less people understand generators than
they might. After all, most new programmers often want something like
generators when they start programming, but life gets in their way.
]

Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.

Watch what happens if you take your example (I'm not convinced it's the
best example for this, but it is your example on the python shell:
.... global gl
.... gl=x
.... yield x
....
Clearly this has done something. What you haven't understood by the looks
of things is what and why. Let's take this session a bit further. Let's
find out the value of s:<generator object at 0x40397a8c>

OK, so the call to the generator function returned a generator. That
should make sense to you. This will hopefully make more sense if we make
a few more calls to 'gen' and look at their values.
<generator object at 0x40397bec>

As you'd expect the next call *also* creates a generator object. You'll
note that it has a different location ( "at 0x....." ) meaning they're
different objects.

Do that a few more times and we see the same:<generator object at 0x40397c0c>

What this is doing is creating a generator object which can then be
asked repeatedly to run, and yield values.

A more accurate term might be iterating over the generator object. However
the /idea/ is that you create the object wrapping a context of/for running
that can suspend its own control and return intemediate values. Clearly
the generator also needs a way of telling us its finished iterating. Let's
make that more concrete. You call the .next() method of the object, and
when it's done it raises a StopIteration exception.

The way we do this with your code is as follows:
1

Yay! That's the "1" that you were expecting to see. In your function you
also updated a global "gl". This is probably not a wise idea, but hey,
let's look at what happened to that value:
1

Again, this is what you should have expected - since it's what you were
after.

Again, looking at your function, "yield x" was the last statement, so you
should be wondering what happens next:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

As you can see we get told that it's dropped off the end. If we try calling
again:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

If we retry with the generator objects created above, we can see:

4

3

Hopefully that's a clearer explanation of what's actually going on with
your code. A more complex example is based on the example in the PEP:

def configurable_fib(base1=1, base2=1):
a, b = base1, base2
while 1:
yield a
a,b = b, a+b
.... a, b = base1, base2
.... while 1:
.... yield a
.... a,b = b, a+b3

We can then create another one and ask that for values:
(13, 2)

Or we can create a few, with unusual bases for the fibonacci sequence and
find their first few values. Let's take the bases of 1 to 10.

fibs = [ configurable_fib(x,x) for x in range(1,11) ]
print [ x.next() for x in fibs ] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print [ x.next() for x in fibs ] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print [ x.next() for x in fibs ] [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
print [ x.next() for x in fibs ] [3, 6, 9, 12, 15, 18, 21, 24, 27, 30]
print [ x.next() for x in fibs ]

[5, 10, 15, 20, 25, 30, 35, 40, 45, 50]

This shows that returning a generator *object* rather than just a value
when you call the generator function is a useful behaviour, even if
initially when learning generators it's slightly counter intuitive.

One way to really understand them though is to try building something
with a whole load of generators. For those purposes, I wrote a tutorial
for our project which is based very much around having lots of generators.
The tutorial is made up of a set of learning exercises:

* http://kamaelia.sourceforge.net/MiniAxon/

We've tested this tutorial on a couple of novices at work (they learn
python one week and get this tutorial the next), and they've found it
relatively simple. The first hadn't done any programming before, except
a small amount of VB - he was a pre-university trainee. The second was
a university vacation trainee who'd done 2 years, but had no experience
of the ideas in the tutorial or python before joining us.

It's specifically targeted at novices and might be of use when seeing the
possibilities of what you can actually use generators for.

{ Incidentally if anyone reading this, not just yourself, decides to give
it a go, I'd really appreciate hearing back from you what you thought
of it, easy/difficult, clear/unclear, what level of experience you have,
etc. People don't have to really, I'm posting this because I've noticed
a couple of people have tried this so far as a means to trying to
understand generators }

Best Regards and hope the above is useful,


Michael.