generator functions: why won't this work?

[zillow20]

Hi all,

I'm trying to understand generator functions and the yield keyword.
I'd like to understand why the following code isn't supposed to work.
(What I would have expected it to do is, for a variable number of
arguments composed of numbers, tuples of numbers, tuples of tuples,
etc., the function would give me the next number "in sequence")
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
getNextScalar(arg)
else:
yield arg
####################################

# here's an example that uses this function:
# creating a generator object:
g = getNextScalar(1, 2, (3,4))
g.next() # OK: returns 1
g.next() # OK: returns 2
g.next() # not OK: throws StopIteration error

####################################

I'm sure I'm making some unwarranted assumption somewhere, but I
haven't been able to figure it out yet (just started learning Python a
couple of days ago).

Any help will be appreciated

Akiel

 

[Mel]

I'm trying to understand generator functions and the yield keyword.
I'd like to understand why the following code isn't supposed to work.
(What I would have expected it to do is, for a variable number of
arguments composed of numbers, tuples of numbers, tuples of tuples,
etc., the function would give me the next number "in sequence")
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
getNextScalar(arg)
else:
yield arg
####################################

# here's an example that uses this function:
# creating a generator object:
g = getNextScalar(1, 2, (3,4))
g.next() # OK: returns 1
g.next() # OK: returns 2
g.next() # not OK: throws StopIteration error

####################################

I'm sure I'm making some unwarranted assumption somewhere, but I
haven't been able to figure it out yet (just started learning Python a
couple of days ago).

Any help will be appreciated

`getNextScalar(arg)` doesn't yield anything (it creates, but doesn't
use, a new generator object,) so nothing comes out. Look at
5


Maybe you want

if isinstance (arg, tuple):
for s in getNextScalar (*arg):
yield s




Mel.

 

[George Sakkis]

Hi all,

I'm trying to understand generator functions and the yield keyword.
I'd like to understand why the following code isn't supposed to work.
(What I would have expected it to do is, for a variable number of
arguments composed of numbers, tuples of numbers, tuples of tuples,
etc., the function would give me the next number "in sequence")
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
getNextScalar(arg)
else:
yield arg
####################################

# here's an example that uses this function:
# creating a generator object:
g = getNextScalar(1, 2, (3,4))
g.next() # OK: returns 1
g.next() # OK: returns 2
g.next() # not OK: throws StopIteration error

####################################

I'm sure I'm making some unwarranted assumption somewhere, but I
haven't been able to figure it out yet (just started learning Python a
couple of days ago).

Any help will be appreciated

You're pretty close, there's just one more thing left. The return
value of a generator function is an iterable, something you're
supposed to iterate over. In the 'if' clause you call recursively the
generator on arg but you don't use the result, you discard it as soon
as it is returned. What you have to do is iterate over the returned
iterable and yield its values:

# don't need parentheses around the if expression
if isinstance(arg, tuple):
for scalar in getNextScalar(arg):
yield scalar


Hope this helps,
George

 

[Steve Holden]

Hi all,

I'm trying to understand generator functions and the yield keyword.
I'd like to understand why the following code isn't supposed to work.
(What I would have expected it to do is, for a variable number of
arguments composed of numbers, tuples of numbers, tuples of tuples,
etc., the function would give me the next number "in sequence")
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
getNextScalar(arg)
else:
yield arg
####################################

# here's an example that uses this function:
# creating a generator object:
g = getNextScalar(1, 2, (3,4))
g.next() # OK: returns 1
g.next() # OK: returns 2
g.next() # not OK: throws StopIteration error

####################################

I'm sure I'm making some unwarranted assumption somewhere, but I
haven't been able to figure it out yet (just started learning Python a
couple of days ago).

Any help will be appreciated

In your recursive call you are passing a single argument, a tuple. You
should create it to multiple arguments with a star. Neither do you do
anything with the iterator after you create it.

Try (untested)

####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
for a in getNextScalar(*arg):
yield a
else:
yield arg
####################################

regards
Steve

 

[George Sakkis]

You're pretty close, there's just one more thing left. The return
value of a generator function is an iterable, something you're
supposed to iterate over. In the 'if' clause you call recursively the
generator on arg but you don't use the result, you discard it as soon
as it is returned. What you have to do is iterate over the returned
iterable and yield its values:

# don't need parentheses around the if expression
if isinstance(arg, tuple):
for scalar in getNextScalar(arg):
yield scalar

Hope this helps,
George

Just after hitting send I noticed a second unrelated issue that has to
do with the generator's signature. Since you define it to take any
positional arguments instead of a single argument (as I mistakenly
assumed), the right way to call it recursively is expand the tuple arg
into positional arguments: getNextScalar(*arg) instead of
getNextScalar(arg):

# don't need parentheses around the if expression
if isinstance(arg, tuple):
for scalar in getNextScalar(*arg):
yield scalar

George

 

[Gabriel Genellina]

I'm trying to understand generator functions and the yield keyword.
I'd like to understand why the following code isn't supposed to work.
(What I would have expected it to do is, for a variable number of
arguments composed of numbers, tuples of numbers, tuples of tuples,
etc., the function would give me the next number "in sequence")
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
getNextScalar(arg)
else:
yield arg
####################################

You're not the first one in getting confused. After all, this schema works
well for other recursive constructs.
Perhaps a progression of working code samples will help to understand what
happens here. The simplest thing would be to just print the items as
they're encountered, in a recursive call:

py> data = (1, 2, (3,4,(5,6),7))
py>
py> print "1) using print"
1) using print
py>
py> def getNextScalar(args):
.... for arg in args:
.... if isinstance(arg, tuple):
.... getNextScalar(arg)
.... else:
.... print arg
....
py> getNextScalar(data)
1
2
3
4
5
6
7

Now one could try to collect the numbers in a list:

py> print "2) using extend"
2) using extend
py>
py> def getNextScalar(args):
.... result = []
.... for arg in args:
.... if isinstance(arg, tuple):
.... result.extend(getNextScalar(arg))
.... else:
.... result.append(arg)
.... return result
....
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]

Note that we use two different list methods: for individual items, we use
"append", but for tuples we use "extend" in the recursive call. If extend
weren't available, we could emulate it with append:

py> print "3) using append"
3) using append
py>
py> def getNextScalar(args):
.... result = []
.... for arg in args:
.... if isinstance(arg, tuple):
.... for item in getNextScalar(arg):
.... result.append(item)
.... else:
.... result.append(arg)
.... return result
....
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]

See how we need an additional loop to iterate over the results that we get
from the recursive call.
Now instead of building an intermediate result list, we delegate such task
over the caller, and we use a generator that just yields items; this way,
we remove all references to the result list and all result.append calls
become yield statements. The inner loop has to remain the same. The yield
statement acts somewhat as an "append" over an outer list created by the
generator's caller.

py> print "4) using yield"
4) using yield
py>
py> def getNextScalar(args):
.... for arg in args:
.... if isinstance(arg, tuple):
.... for item in getNextScalar(arg):
.... yield item
.... else:
.... yield arg
....
py> getNextScalar(data)
<generator object at 0x00A3AE68>
py> list(getNextScalar(data))
[1, 2, 3, 4, 5, 6, 7]

I hope it's more clear now why you have to use yield on the recursive call
too.

<idea mode="raw">
Perhaps this:

yield *iterable

could be used as a shortcut for this:

for __temp in iterable: yield __temp

</idea>

 

[zillow10]

I'm trying to understand generator functions and the yield keyword.
I'd like to understand why the following code isn't supposed to work.
(What I would have expected it to do is, for a variable number of
arguments composed of numbers, tuples of numbers, tuples of tuples,
etc., the function would give me the next number "in sequence")
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
getNextScalar(arg)
else:
yield arg
####################################

You're not the first one in getting confused. After all, this schema works
well for other recursive constructs.
Perhaps a progression of working code samples will help to understand what
happens here. The simplest thing would be to just print the items as
they're encountered, in a recursive call:

py> data = (1, 2, (3,4,(5,6),7))
py>
py> print "1) using print"
1) using print
py>
py> def getNextScalar(args):
... for arg in args:
... if isinstance(arg, tuple):
... getNextScalar(arg)
... else:
... print arg
...
py> getNextScalar(data)
1
2
3
4
5
6
7

Now one could try to collect the numbers in a list:

py> print "2) using extend"
2) using extend
py>
py> def getNextScalar(args):
... result = []
... for arg in args:
... if isinstance(arg, tuple):
... result.extend(getNextScalar(arg))
... else:
... result.append(arg)
... return result
...
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]

Note that we use two different list methods: for individual items, we use
"append", but for tuples we use "extend" in the recursive call. If extend
weren't available, we could emulate it with append:

py> print "3) using append"
3) using append
py>
py> def getNextScalar(args):
... result = []
... for arg in args:
... if isinstance(arg, tuple):
... for item in getNextScalar(arg):
... result.append(item)
... else:
... result.append(arg)
... return result
...
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]

See how we need an additional loop to iterate over the results that we get
from the recursive call.
Now instead of building an intermediate result list, we delegate such task
over the caller, and we use a generator that just yields items; this way,
we remove all references to the result list and all result.append calls
become yield statements. The inner loop has to remain the same. The yield
statement acts somewhat as an "append" over an outer list created by the
generator's caller.

py> print "4) using yield"
4) using yield
py>
py> def getNextScalar(args):
... for arg in args:
... if isinstance(arg, tuple):
... for item in getNextScalar(arg):
... yield item
... else:
... yield arg
...
py> getNextScalar(data)
<generator object at 0x00A3AE68>
py> list(getNextScalar(data))
[1, 2, 3, 4, 5, 6, 7]

I hope it's more clear now why you have to use yield on the recursive call
too.

<idea mode="raw">
Perhaps this:

yield *iterable

could be used as a shortcut for this:

for __temp in iterable: yield __temp

</idea>

Thanks to everyone for your very helpful replies. I think I was trying
to use generator functions without first having really read about
iterators (something I still haven't done, although I've managed to
extrapolate some details based on your comments), and therefore really
"putting the cart before the horse". I was interpreting
getNextScalar(arg) on line 3 as a function call in the usual sense
rather than an object that was being created but never used.

I have a question about the other issue that was pointed out. With
reference to the following (corrected) code:

def getNextScalar(*args):
for arg in args:
if(isinstance(arg, tuple)):
for f in getNextScalar(*arg): # why not getNextScalar(arg)?
yield f
else:
yield arg

although I've verified that the line 4 needs to be "for f in
getNextScalar(*arg)" rather than "for f in getNextScalar(arg)" when
passing multiple arguments to the function, I'd just like to test my
understanding of this. Suppose I create the following generator
object:

g = getNextScalar(1, 2, (3, 4), 5)

when the iterator reaches the tuple argument (3, 4) then, according to
Steve and George, the * in *arg causes this tuple to be expanded into
positional arguments, and it makes sense to do it this way. But what
happens when getNextScalar(arg) is used instead? I presume that (3,4)
is passed as a single tuple argument, if(isinstance(arg, tuple)) is
true, and the tuple is passed again as an the argument to
getNextScalar()... ad infinitum. The alternative is to define a
function that takes a tuple, as Gabriel has done.

Regards,
AK

 

[Steve Holden]

I'm trying to understand generator functions and the yield keyword.
I'd like to understand why the following code isn't supposed to work.
(What I would have expected it to do is, for a variable number of
arguments composed of numbers, tuples of numbers, tuples of tuples,
etc., the function would give me the next number "in sequence")
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
getNextScalar(arg)
else:
yield arg
####################################

You're not the first one in getting confused. After all, this schema works
well for other recursive constructs.
Perhaps a progression of working code samples will help to understand what
happens here. The simplest thing would be to just print the items as
they're encountered, in a recursive call:

py> data = (1, 2, (3,4,(5,6),7))
py>
py> print "1) using print"
1) using print
py>
py> def getNextScalar(args):
... for arg in args:
... if isinstance(arg, tuple):
... getNextScalar(arg)
... else:
... print arg
...
py> getNextScalar(data)
1
2
3
4
5
6
7

Now one could try to collect the numbers in a list:

py> print "2) using extend"
2) using extend
py>
py> def getNextScalar(args):
... result = []
... for arg in args:
... if isinstance(arg, tuple):
... result.extend(getNextScalar(arg))
... else:
... result.append(arg)
... return result
...
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]

Note that we use two different list methods: for individual items, we use
"append", but for tuples we use "extend" in the recursive call. If extend
weren't available, we could emulate it with append:

py> print "3) using append"
3) using append
py>
py> def getNextScalar(args):
... result = []
... for arg in args:
... if isinstance(arg, tuple):
... for item in getNextScalar(arg):
... result.append(item)
... else:
... result.append(arg)
... return result
...
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]

See how we need an additional loop to iterate over the results that we get
from the recursive call.
Now instead of building an intermediate result list, we delegate such task
over the caller, and we use a generator that just yields items; this way,
we remove all references to the result list and all result.append calls
become yield statements. The inner loop has to remain the same. The yield
statement acts somewhat as an "append" over an outer list created by the
generator's caller.

py> print "4) using yield"
4) using yield
py>
py> def getNextScalar(args):
... for arg in args:
... if isinstance(arg, tuple):
... for item in getNextScalar(arg):
... yield item
... else:
... yield arg
...
py> getNextScalar(data)
<generator object at 0x00A3AE68>
py> list(getNextScalar(data))
[1, 2, 3, 4, 5, 6, 7]

I hope it's more clear now why you have to use yield on the recursive call
too.

<idea mode="raw">
Perhaps this:

yield *iterable

could be used as a shortcut for this:

for __temp in iterable: yield __temp

</idea>

Thanks to everyone for your very helpful replies. I think I was trying
to use generator functions without first having really read about
iterators (something I still haven't done, although I've managed to
extrapolate some details based on your comments), and therefore really
"putting the cart before the horse". I was interpreting
getNextScalar(arg) on line 3 as a function call in the usual sense
rather than an object that was being created but never used.

I have a question about the other issue that was pointed out. With
reference to the following (corrected) code:

def getNextScalar(*args):
for arg in args:
if(isinstance(arg, tuple)):
for f in getNextScalar(*arg): # why not getNextScalar(arg)?
yield f
else:
yield arg

although I've verified that the line 4 needs to be "for f in
getNextScalar(*arg)" rather than "for f in getNextScalar(arg)" when
passing multiple arguments to the function, I'd just like to test my
understanding of this. Suppose I create the following generator
object:

g = getNextScalar(1, 2, (3, 4), 5)

when the iterator reaches the tuple argument (3, 4) then, according to
Steve and George, the * in *arg causes this tuple to be expanded into
positional arguments, and it makes sense to do it this way. But what
happens when getNextScalar(arg) is used instead? I presume that (3,4)
is passed as a single tuple argument, if(isinstance(arg, tuple)) is
true, and the tuple is passed again as an the argument to
getNextScalar()... ad infinitum. The alternative is to define a
function that takes a tuple, as Gabriel has done.

Either approach would work - it just seemed to make more sense to use
existing functionality and recurse (at least to me). You should try
removing the * from the recursive call. I expect you would find that you
(eventually) experience a stack overflow due to the infinite nature of
the recursion.

regards
Steve

 

[Mel]

understanding of this. Suppose I create the following generator
object:

g = getNextScalar(1, 2, (3, 4), 5)

when the iterator reaches the tuple argument (3, 4) then, according to
Steve and George, the * in *arg causes this tuple to be expanded into
positional arguments, and it makes sense to do it this way. But what
happens when getNextScalar(arg) is used instead?

Try it:

Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information..... print arg
........ print arg
....Traceback (most recent call last):




Mel.

 

[castironpi]

En Tue, 01 Apr 2008 23:56:50 -0300, <[email protected]> escribió:

   yield *iterable

could be used as a shortcut for this:

   for __temp in iterable: yield __temp

How serious were you about that?

 

[zillow10]

(e-mail address removed) wrote:

I'd just like to test my

understanding of this. Suppose I create the following generator
object:

g = getNextScalar(1, 2, (3, 4), 5)

when the iterator reaches the tuple argument (3, 4) then, according to
Steve and George, the * in *arg causes this tuple to be expanded into
positional arguments, and it makes sense to do it this way. But what
happens when getNextScalar(arg) is used instead?

Try it:

Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.... print arg
...... print arg
...Traceback (most recent call last):

Mel.

Well, I understand that (unless I missed the point you're trying to
make). But with respect to the example I quoted:

def getNextScalar(*args):
for arg in args:
if(isinstance(arg, tuple)):
for f in getNextScalar(arg): # should use *arg
yield f
else:
yield arg
where the function is declared as def getNextScalar(*arg), but is
called using getNextScalar(arg), with arg being a tuple: here the
generator is being passed a single argument, so there's no TypeError
as in your example. However, it fails - as I understand it - because
the function keeps passing the same tuple (being unable to access the
elements inside it) and goes into an infinite loop: print i


1
2
3
4

# third argument is a tuple: print i


1
2

Traceback (most recent call last):
File "<pyshell#108>", line 1, in <module>
for i in g:
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
....

AK

 

[Gabriel Genellina]

How serious were you about that?

Not so much, I haven't thougth enough on it. Looks fine in principle, but
yield expressions may be a problem.

 

[Mel]

(e-mail address removed) wrote:

I'd just like to test my

understanding of this. Suppose I create the following generator
object:
g = getNextScalar(1, 2, (3, 4), 5)
when the iterator reaches the tuple argument (3, 4) then, according to
Steve and George, the * in *arg causes this tuple to be expanded into
positional arguments, and it makes sense to do it this way. But what
happens when getNextScalar(arg) is used instead?

Try it:

Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

def a (arg):

... print arg
...

def astar (*arg):

... print arg
...

a(3,4)

Traceback (most recent call last):

astar(3,4) (3, 4)
a((3,4)) (3, 4)
astar((3,4)) ((3, 4),)

Mel.

Well, I understand that (unless I missed the point you're trying to
make). But with respect to the example I quoted:

def getNextScalar(*args):
for arg in args:
if(isinstance(arg, tuple)):
for f in getNextScalar(arg): # should use *arg
yield f
else:
yield arg
where the function is declared as def getNextScalar(*arg), but is
called using getNextScalar(arg), with arg being a tuple: here the
generator is being passed a single argument, so there's no TypeError
as in your example. However, it fails - as I understand it - because
the function keeps passing the same tuple (being unable to access the
elements inside it) and goes into an infinite loop: print i


1
2
3
4

# third argument is a tuple: print i


1
2

Traceback (most recent call last):
File "<pyshell#108>", line 1, in <module>
for i in g:
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
...

Yeah. I feel as though I haven't put the idea clearly. When a
function is defined as `def astar (a*)`, then the parameters that it's
called with, however many they are, are packed up into one tuple.
Illustrated by those print statements.

On the calling side, `a (*(3,4))` will unpack the (3,4) tuple and
treat the contents as ordinary positional parameters of a. The call
`a(*(3,4))` and the call `a(3,4)` are completely equivalent as far as
the function a can see.

This explanation is going nowhere fast. Anyway, if you define
getNextScalar to pack its positional arguments into a tuple that you
can iterate over, THEN if you want to feed it a tuple of values to
iterate over you have to star-unpack the tuple when you call. The
buggy behaviour is continually calling getNextScalar with a single
positional parameter (3,4), packing that single parameter into a
1-tuple ((3,4),) , iterating over the 1-tuple to extract (3,4) and
calling getNextScalar again -- ad dump.

Mel.

 

[Robert Lehmann]

Not so much, I haven't thougth enough on it. Looks fine in principle,
but yield expressions may be a problem.

Issue 2292: "Missing *-unpacking generalizations"
http://bugs.python.org/issue2292

Discussion on python-3000.devel:
http://thread.gmane.org/gmane.comp.python.python-3000.devel/12131

 

[Arnaud Delobelle]

Not so much, I haven't thougth enough on it. Looks fine in principle, but  
yield expressions may be a problem.

Funnily, there's a patch for py3k to make this work as in your idea.
It's currently being discussed on python-3000