What makes an iterator an iterator?

[Steven D'Aprano]

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.

But I'm having problems writing a class that acts as an iterator. I have:

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self._next()
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

But this is what I get:
.... print word
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: iter() returned non-iterator of type 'Parrot'

Why is my instance not an iterator?

But I can do this:
.... print word
....
Norwegian
Blue's
have
beautiful
plumage!

I find myself perplexed as to this behaviour.

 

[I V]

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method. ....
class Parrot(object): ....
def __init__(self):
self.next = self._next()

self.next isn't a method here, it's a generator. You could do:

def __init__(self):
self.next = self._next().next

But, having tested, that doesn't appear to work either - I think
this is because, again, self.next is not strictly a method, it's an
attribute that happens to be callable.

The python manual gives you a possible solution:

---QUOTE http://docs.python.org/lib/typeiter.html ---
Python's generators provide a convenient way to implement the iterator
protocol. If a container object's __iter__() method is implemented as a
generator, it will automatically return an iterator object (technically, a
generator object) supplying the __iter__() and next() methods.
---END QUOTE---

i.e., just rename your _next function to __iter__ . Your class won't
itself be an iterator, but it will be usable in for statements and so one,
and convertable to an iterator with the iter builtin.

 

[Dennis Lee Bieber]

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.

But I'm having problems writing a class that acts as an iterator. I have:

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self._next()
def _next(self):

.... def __iter__(self):
.... return self.next
.... def __init__(self):
.... self.next = self._next()
.... def _next(self):
.... for word in "Norwegian Blue's have beautiful
plumage!".split():
.... yield word
.... .... print word
....
Norwegian
Blue's
have
beautiful
plumage!--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Ben Finney]

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self._next()
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

Clearly the problem is you've misused an apostrophe. Python doesn't
like the plural getting an apostrophe.

<URL:http://www.angryflower.com/bobsqu.gif>

 

[Stefan Rank]

I thought that an iterator was any object that follows the iterator

replace object with "instance of a class", i.e. the relevant methods are
looked up in the __class__ not in the instance (I think).
I had the same troubles trying to dynamically reassign a __call__ method...

protocol, that is, it has a next() method and an __iter__() method.

But I'm having problems writing a class that acts as an iterator. I have:

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self._next()
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

Try this::

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.__class__.next = self._next().next # see post by I V
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

This works but practically forces the class to be used as a singleton...
not very helpful

Better:

* use the '__iter__ returns/is a generator' way,
* or if you need the object to be the iterator, implement the next
method directly on the class::

class Parrot(object):
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word
def __iter__(self):
self.generator = self._next()
return self
def next(self):
return self.generator.next()

cheers,
stefan

 

[Steven D'Aprano]

Clearly the problem is you've misused an apostrophe. Python doesn't
like the plural getting an apostrophe.

<URL:http://www.angryflower.com/bobsqu.gif>

I thought the rule wa's that any time you 'see an 'S, you put an
apo'strophe before it. If that's wrong, 'shouldn't it rai'se an exception?

 

[Steven D'Aprano]

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.

[snip]

i.e., just rename your _next function to __iter__ . Your class won't
itself be an iterator, but it will be usable in for statements and so one,
and convertable to an iterator with the iter builtin.

Thanks to all those who helped, this fixed my problem.

For the record, this is what I actually wanted: a four-line self-sorting
dictionary:

class SortedDict(dict):
def __iter__(self):
for key in sorted(self.keys()):
yield key

Note that using sorted(self) does not work.

Iterating over a SortedDictionary returns the keys in sorted order. This
minimalist implementation doesn't sort the values, items or string
representation of the dict, but they should be easy to implement.

 

[Paul McGuire]

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.

[snip]

i.e., just rename your _next function to __iter__ . Your class won't
itself be an iterator, but it will be usable in for statements and so one,
and convertable to an iterator with the iter builtin.

Thanks to all those who helped, this fixed my problem.

For the record, this is what I actually wanted: a four-line self-sorting
dictionary:

class SortedDict(dict):
def __iter__(self):
for key in sorted(self.keys()):
yield key

Note that using sorted(self) does not work.

Iterating over a SortedDictionary returns the keys in sorted order. This
minimalist implementation doesn't sort the values, items or string
representation of the dict, but they should be easy to implement.

Very neat. Why not this?

class SortedDict(dict):
def __iter__(self):
return iter(sorted(self.keys()))

-- Paul

 

[Peter Otten]

class SortedDict(dict):
    def __iter__(self):
        for key in sorted(self.keys()):
            yield key

Note that using sorted(self) does not work.

That's because sorted() invokes __iter__() if present. To prevent the
recursion you can explicitly invoke dict.__iter__():
.... def __iter__(self):
.... return iter(sorted(super(SortedDict, self).__iter__()))
....['a', 'b', 'c']

Note that a list of keys is still built before the first key is yielded,
and, unlike dict, you can modify your SortedDict while iterating over it:
.... if k == "b": sd["x"] = 42
....{'a': 1, 'x': 42, 'c': 3, 'b': 2}

whereas:
.... if k == "b": d["x"] = 42
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration

By the way, I think it would be worthwile to change super() to allow
e. g. super(SomeClass, self)[...] as an alternate spelling for
super(SomeClass, self).__getitem__(...) etc. With such an enhancement
SortedDict would become

class SortedDict(dict):
def __iter__(self):
# doesn't work in current Python
iter(sorted(super(SortedDict, self)))


Peter

 

[Georg Brandl]

replace object with "instance of a class", i.e. the relevant methods are
looked up in the __class__ not in the instance (I think).
I had the same troubles trying to dynamically reassign a __call__ method...

This is correct.

It's not properly documented though, and not applied consistently, e.g.
__enter__ and __exit__ are looked up in the instance itself.

Georg

 

[Isaac Rodriguez]

class Parrot(object):
def __iter__(self):
return self
def __init__(self):

Typo right here!!!!

self.next = self._next()

write:
self.next = self._next

no parenthesis.

def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

See previous explanation.

thanks,

- Isaac.

 

[Isaac Rodriguez]

Sorry, my previous post was incomplete. I didn't realized that you
implemented _next() as a generator funcition. Besides changing
__init__() from

self.next = self._next()

to
self.next = self._next

you need to implement __iter__() as:

return self.next()

 

[Alex Martelli]

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.

The special methods need to be on the type -- having attributes of those
names on the instance doesn't help (applies to all special methods in
the normal, aka newstyle, object model; legacy, aka classic, classes,
work by slightly different and not entirely self-consistent semantics).


Alex

 

[boggom]

I find myself perplexed as to this behaviour.

You can not iterate over a dead object!

 

[mensanator]

You can not iterate over a dead object!

It's not dead, it's restin'. All shagged out over a long squak.

 

[7stud]

The special methods need to be on the type -- having attributes of those
names on the instance doesn't help (applies to all special methods in
the normal, aka newstyle, object model; legacy, aka classic, classes,
work by slightly different and not entirely self-consistent semantics).

Alex

Can you explain some of the details of why this code fails:

---
class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self.next().next
def next(self):
for word in "Norwegian Blue's have beautiful
plumage!".split():
yield word

P = Parrot()
for word in P:
print word
------

It causes an infinite loop that just prints out the iterator object
returned when you call the generator function.

If I try this:

-----
class Parrot(object):
def __iter__(self):
return self
def __init__(self):
print self.next()
print self.next().next
#self.next = self.next().next
def next(self):
for word in "Norwegian Blue's have beautiful
plumage!".split():
yield word

P = Parrot()

'''
for word in P:
print word
'''
----------

the output is:

<generator object at 0x5b080>
<method-wrapper object at 0x55d30>

Based on that output, self.next() is the iterator object that wraps
the generator function, and it has a next() method. If 'i' is the
iterator object, then the statement:

self.next = self.next().next

is equivalent to:

self.next = i.next

and therefor calling P.next() is equivalent to i.next(), which appears
to be exactly what you want. As I understand it, this is how the for
loop works:

1) The for loop causes the built in, global iter() function to be
called with P as an argument: iter(P). That calls P's __iter__()
method, which just returns P.

2) The for loop repeatedly calls next() on whatever object is returned
by 1), so that results in calls to P.next().

3) P.next() is equivalent to i.next()

I don't understand at which step does the code fail.

 

[Terry Reedy]

An iterator is an object with a .__iter__ method that returns self and a
..next method that either returns an object or raises StopIteration.

One very good way to get an iterator from an iterable is for .__iter__ to
be a generator function. When called, it returns an generator with
..__iter__ and .next methods that work as specified.

words = "Norwegian Blues have beautiful plumage!".split()
print words

prints
['Norwegian', 'Blues', 'have', 'beautiful', 'plumage!']

class Parrot(object):
def __init__(self, words):
self.words = words
def __iter__(self):
for word in words:
yield word

for word in Parrot(words):
print word

prints

Norwegian
Blues
have
beautiful
plumage!

One can also make an iterable an (self) iterator by correctly writing the
two needed methods.

class Parrot2(object):
def __init__(self, words):
self.words = words
def __iter__(self):
self.it = -1
return self
def next(self):
self.it += 1
try:
return self.words[self.it]
except IndexError:
raise StopIteration

for word in Parrot2(words):
print word

which prints the same 5 lines. However, when an object is its own
iterator, it can only be one iterator at a time (here, pointing at one
place in the word list), whereas the first method allows multiple iterators
(here, possibly pointing to different places in the list.

| Can you explain some of the details of why this code fails:

[snip all examples of bad code that violate the iterator rule by improperly
writing .next as a generator function]

I think most people should learn how to write iterators correctly, as I
gave examples of above, rather than worry about the details of how and why
mis-written code fails with a particular version of a particular
implementation. So I won't go down that road.

Terry Jan Reedy

 

[Alex Martelli]

Can you explain some of the details of why this code fails: ...
def next(self):
for word in "Norwegian Blue's have beautiful
plumage!".split():
yield word

Sure, easily: a loop like "for x in y:" binds an unnamed temporary
variable (say _t) to iter(y) and then repeatedly calls _t.next() [or to
be pedantic type(_t).next(t)] until that raises StopIteration.

Calling a generator, such as this next method, returns an iterator
object; calling it repeatedly returns many such iterator objects, and
never raises StopIteration, thus obviously producing an unending loop.


Alex

 

[7stud]

Hi,

Thanks for the responses.

Can you explain some of the details of why this code fails:

---
class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self.next().next
def next(self):
for word in "Norwegian Blue's have beautiful
plumage!".split():
yield word

P = Parrot()
for word in P:
print word
------


...a loop like "for x in y:" binds an unnamed temporary
variable (say _t) to iter(y) and then repeatedly calls _t.next() [or to
be pedantic type(_t).next(t)] until that raises StopIteration.

Aiiii. Isn't this the crux:

repeatedly calls....[type(_t).next(t)]

As far as I can tell, if the call was actually _t.next(), the code I
asked about would work. However, the name look up for 'next' when you
call:

P.next()

is entirely different from the name look up for 'next' when you call:

type(P).next().

In the first lookup, the instance attribute "next" hides the class
attribute "next". In the second lookup, the "next" attribute of the
class is returned, i.e. the next() method. Yet, a generator-function-
call returns an iterator object that wraps the generator function, so
when the for loop makes repeated calls to type(P).next(), an iterator
object is repeatedly returned--what you want is i.next() to be
returned.

I suspected next() might be called through the class, but after
carefully parsing the section on iterators (Python in a Nutshell, p.
65) where it says iter() returns an object i, and then the for loop
repeatedly calls i.next(), I dismissed that idea. I have to admit I'm
still a little confused by why you only parenthetically noted that
information and called it pedantic.

One very good way to get an iterator from an iterable is for .__iter__ to
be a generator function.

Ahhh. That eliminates having to deal with next().next constructs.
Nice.

snip all examples of bad code that violate the iterator rule
by improperly writing .next as a generator function

What iterator rule states that .next can't be a generator function?
My book says an iterator is any object with a .next method that is
callable without arguments (Python in a Nutshell(p.65) says the same
thing). Also, my boos says an iterable is any object with an
__iter__ method. As a result, next() and __iter__() don't have to
be in the same object:

lass MyIterator(object):
def __init__(self, obj):
self.toIterateOver = obj
def next(self):
for x in range(self.toIterateOver.age):
print x
raise StopIteration

class Dog(object):
def __init__(self, age):
self.age = age
def __iter__(self):
return MyIterator(self)

d = Dog(5)
for year in d:
print year


I've read recommendations that an iterator should additionally contain
an __iter__() method, but I'm not sure why that is. In particular PEP
234 says:

----------
Classes can define how they are iterated over by defining an
__iter__() method; this should take no additional arguments and
return a valid iterator object. A class that wants to be an
iterator should implement two methods: a next() method that
behaves
as described above, and an __iter__() method that returns self.

The two methods correspond to two distinct protocols:

1. An object can be iterated over with "for" if it implements
__iter__() or __getitem__().

2. An object can function as an iterator if it implements next().

Container-like objects usually support protocol 1. Iterators are
currently required to support both protocols. The semantics of
iteration come only from protocol 2; protocol 1 is present to make
iterators behave like sequences; in particular so that code
receiving an iterator can use a for-loop over the iterator.

Classes can define how they are iterated over by defining an
__iter__() method; this should take no additional arguments and
return a valid iterator object. A class that wants to be an
iterator should implement two methods: a next() method that
behaves
as described above, and an __iter__() method that returns self.

The two methods correspond to two distinct protocols:

1. An object can be iterated over with "for" if it implements
__iter__() or __getitem__().

2. An object can function as an iterator if it implements next().

Container-like objects usually support protocol 1. Iterators are
currently required to support both protocols. The semantics of
iteration come only from protocol 2; protocol 1 is present to make
iterators behave like sequences; in particular so that code
receiving an iterator can use a for-loop over the iterator.
--------

The semantics of
iteration come only from protocol 2

My example demonstrates that.

protocol 1 is present to make
iterators behave like sequences; in particular so that code
receiving an iterator can use a for-loop over the iterator.

I don't understand that part--it looks like my example is using a for
loop over the iterator.

 

[Steven D'Aprano]

For the record, this is what I actually wanted: a four-line self-sorting
dictionary:

class SortedDict(dict):
def __iter__(self):
for key in sorted(self.keys()):
yield key

[snip]

Very neat. Why not this?

class SortedDict(dict):
def __iter__(self):
return iter(sorted(self.keys()))

Good question. I don't have a good answer except for "because I didn't
think of it".