Comment on PEP-0322: Reverse Iteration Methods

[David Eppstein]

How about

from itertools import ireverse
for i in ireverse(xrange(n)):
# suite

ireverse(), like imap(), izip(), etc., suggests that the operation is
iterative, and that no modification of the original sequence will be
performed. Others have suggested riter() (right iteration), in order to form
an association with the iter() builtin. As a matter of taste, I prefer
ireverse().

ireverse, like imap(), izip(), etc., suggests that the operation happens
without the memory overhead of copying the whole sequence into a list
before reversing it. Do you have some plan for how to do that e.g. with
simple generators? Or easy to understand explanation for which things
can be ireversed and which can't?

 

[Lulu of the Lotus-Eaters]

|ireverse, like imap(), izip(), etc., suggests that the operation happens
|without the memory overhead of copying the whole sequence into a list
|before reversing it. Do you have some plan for how to do that e.g. with
|simple generators? Or easy to understand explanation for which things
|can be ireversed and which can't?

Or even just a DECIDABLE explanation for which things can be 'ireversed'
and which can't[*].

Yours, Lulu...

[*] And if you -can- produce such an explanation, I can almost guarantee
you a Fields Medal, and probably a Nobel Prize also.

 

[Lulu of the Lotus-Eaters]

|ireverse, like imap(), izip(), etc., suggests that the operation happens
|without the memory overhead of copying the whole sequence into a list
|before reversing it. Do you have some plan for how to do that e.g. with
|simple generators? Or easy to understand explanation for which things
|can be ireversed and which can't?

Or even just a DECIDABLE explanation for which things can be 'ireversed'
and which can't[*].

Yours, Lulu...

[*] And if you -can- produce such an explanation, I can almost guarantee
you a Fields Medal, and probably a Nobel Prize also.

 

[Alex Martelli]

David Abrahams wrote:
...

Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.

Sure, but so is denying them, e.g., non-mutating methods such as .index()
and .count(). At least we're _consistently_ arbitrary and capricious!-)


Alex

 

[Sean Ross]

[snip]

ireverse, like imap(), izip(), etc., suggests that the operation happens
without the memory overhead of copying the whole sequence into a list
before reversing it. Do you have some plan for how to do that e.g. with
simple generators? Or easy to understand explanation for which things
can be ireversed and which can't?

Nope. Nor have I intimated I had such, nor that any would be forthcoming.
This is a naming suggestion. I am neither the originator of the proposal nor
a proponent of it, so I see no reason to formulate such a plan nor to give
an explanation of how it should be accomplished. I saw a name, and offered
another. That is all.

P1: "If I make this thing, I propose to call it 'ShomozzleBoB'."
P2: "Really? How about just BoB?"
P3: "Do you have a plan for how to do BoB?"
P2: "Huh?"

 

[David Eppstein]

Nope. Nor have I intimated I had such, nor that any would be forthcoming.
This is a naming suggestion. I am neither the originator of the proposal nor
a proponent of it, so I see no reason to formulate such a plan nor to give
an explanation of how it should be accomplished. I saw a name, and offered
another. That is all.

P1: "If I make this thing, I propose to call it 'ShomozzleBoB'."
P2: "Really? How about just BoB?"
P3: "Do you have a plan for how to do BoB?"
P2: "Huh?"

Your proposed name has an implication for the semantics. I was
questioning the feasibility of the implied semantics. I don't think
you're let off the hook by the excuse that "it's only a name".

 

[Stephen Horne]

ireverse, like imap(), izip(), etc., suggests that the operation happens
without the memory overhead of copying the whole sequence into a list
before reversing it. Do you have some plan for how to do that e.g. with
simple generators? Or easy to understand explanation for which things
can be ireversed and which can't?

How about this - only support ireverse for sequences (identified using
len, and perhaps with a check for the keys method in case of mapping
types) - not for iterators at all.

Add an xrange_backward function to complement xrange rather than
trying to modify the result of xrange, and where an object can
logically support reverse iteration have a method or property that
returns an alternative generator/iterator.

At its simplest, something like...

class newlist (list) :
def ibackward (self) :
i = len(self)
while i > 0 :
i -= 1
yield i

....is no great burden for most built-in sequence types, and would
define a protocol for other types to follow where appropriate. It
doesn't need a big memory overhead.

Rather than define a separate xrange_backwards, it is worth noticing
that the built-in 'functions' xrange and enumerate actually seem to be
classes. We could add factory methods, set up to effectively give
alternate constructors, to allow syntax such as...

for i in xrange.backward (10) :
...

for i, j in enumerate.backward (seq) :
...

 

[Sean Ross]

Your proposed name has an implication for the semantics.

Do the implied semantics of ireverse() differ from those implied by the
names iter_backwards(), reverse_view(), riter(), etc.?
Because, to me, they' re just different words for the same idea.

I was questioning the feasibility of the implied semantics.

Fine. Take it up with the person who proposed the idea.

I don't think you're let off the hook by the excuse that "it's only a

name".

I'm let off the "hook" because I was never on it. The responsibility for
answering your questions re: the feasability of this enterprise do not fall
to me, but to the proponent of the enterprise. It's like someone said they
where going to build a bridge from the mainland to Hawaii, and that they
would paint it green. And I said, "What about painting it red?" And you
asked me, "And how to you propose to build this bridge?". My answer is, "Uh,
why don't you ask the fellow over there who brought up the thing in the
first place. I'm just talking paint colours over here." I think your
questions are mis-directed. And, even if they are not, I don't have your
answers, I'm not going to try to come up with them, and I feel no
responsibility to do so.

 

[David Abrahams]

David Abrahams wrote:
...

Sure, but so is denying them, e.g., non-mutating methods such as
.index() and .count().

Not IMO. Immutability is a very useful trait.

At least we're _consistently_ arbitrary and capricious!-)

Not in this case.

 

[Sean Ross]

I'll put it another way. I'm not interested in the semantics, nor in the
implementation. My interest is in the syntax. If this feature can be added
to the language (and I don't care whether it can or cannot be), then I would
prefer to read code like

for i in ireverse(xrange(n)):

than

for i in xrange(n).iter_backwards(n):

And there ends my superficial concerns in this matter. If my proposed syntax
implies more than the other syntax proposals, then I withdraw the
suggestion. I've no interest in arguing about whether those, or any other,
implications can be met. I'm talking sugar. You're talking substance. And I
don't care. If it can be done, let it be done, and give it pretty name. If
it can't be done, then it doesn't much matter what name you give it.

I hope were done here,
Sean

 

[Chad Netzer]

Fine. Take it up with the person who proposed the idea.

You proposed it (hear me out). You are missing David's point. In
addition to whatever semantics a general reverse iterator might have, he
is saying that the itertools iterators have an implied additional
semantic of not (necessarily) needing to fully expand an iterable in
memory to operate on it, as a general ireverse() would probably have to
do. So, having ireverse potentially use up VAST quantities of memory,
in order to work, doesn't quite fit into the itertools philosophy.

That is the point he is making, and it is a specific comment on your (I
believe; others have probably proposed it as well) suggestion of an
ireverse() in itertools.

Now, my response to David's point is that currently, cycle() may also
require enough extra storage to remember an entire iterated sequence, so
the itertools philosophy is not a hard rule. Still, ireverse() would
imply some real devilry under the covers...

 

[Stephen Horne]

Not IMO. Immutability is a very useful trait.

Yes - and perfectly consistent with having *NON*-mutating methods such
as .index() and .count() ;-)

I always assumed that these were considered inconsistent with normal
use of tuples (which certainly I rarely need to get 'index' or
'count'-like results from).

Actually, they even seem a little odd in list, to be honest. I'd have
functions, not necessarily even in __builtins__, which work on any
sequence. They just don't seem like everyday operations that should be
built into the object.

 

[Stephen Horne]

Do the implied semantics of ireverse() differ from those implied by the
names iter_backwards(), reverse_view(), riter(), etc.?
Because, to me, they' re just different words for the same idea.

So far as I can see, it is having the function rather than the method
that creates the issues - that originated with David Abrahams post (at
least tracing back through this thread - it had been suggested
before).

However, your reply was implicitly supporting the use of a function
rather than a method. If 'ireverse' is imported from 'itertools' then
it cannot be a method of the object being reversed.

I think some wires have been slightly crossed - David Eppstein raised
a good point, but should perhaps have aimed it at David Abrahams.

 

[Alex Martelli]

Not IMO. Immutability is a very useful trait.

Sure, very useful indeed -- and why does YO suggest that add such
NON-mutating methods would damage immutability in the LEAST...?

Not in this case.

Some amplification would be welcome, because the above comment
on immutability's usefulness is totally obscure to me in context.


Alex

 

[Sean Ross]

You proposed it (hear me out). You are missing David's point. In
addition to whatever semantics a general reverse iterator might have, he
is saying that the itertools iterators have an implied additional
semantic of not (necessarily) needing to fully expand an iterable in
memory to operate on it, as a general ireverse() would probably have to
do. So, having ireverse potentially use up VAST quantities of memory,
in order to work, doesn't quite fit into the itertools philosophy.

That is the point he is making, and it is a specific comment on your (I
believe; others have probably proposed it as well) suggestion of an
ireverse() in itertools.

Now, my response to David's point is that currently, cycle() may also
require enough extra storage to remember an entire iterated sequence, so
the itertools philosophy is not a hard rule. Still, ireverse() would
imply some real devilry under the covers...

Hi.

Thanks for clearing that up.

Yes, "others have ... proposed it as well". I was merely interested in the
appearance of the code, not the underlying implementation. I was not aware
that the name would imply more than the other suggestions simply because it
was housed in itertools. But then that's not really the point, I suppose:
The main point is the "implied additional semantic", regardless of the
function's location. Yes? OK. Fine. I was not concerned with semantics when
I made the suggestion, only sugar, so I couldn't see where David was coming
from. Sorry for the confusion.

Sean

 

[David Abrahams]

Sure, very useful indeed -- and why does YO suggest that add such
NON-mutating methods would damage immutability in the LEAST...?

Sorry, I misread your reply. I didn't know those were missing.

Some amplification would be welcome, because the above comment
on immutability's usefulness is totally obscure to me in context.

My mistake; you were right as usual.

 

[David Abrahams]

Yes - and perfectly consistent with having *NON*-mutating methods such
as .index() and .count() ;-)

Right. Oops, I misread.

I always assumed that these were considered inconsistent with normal
use of tuples (which certainly I rarely need to get 'index' or
'count'-like results from).

I don't see why not.

Actually, they even seem a little odd in list, to be honest. I'd have
functions, not necessarily even in __builtins__, which work on any
sequence. They just don't seem like everyday operations that should be
built into the object.

It's probably just for optimization purposes.

 

[David Abrahams]

Do the implied semantics of ireverse() differ from those implied by the
names iter_backwards(), reverse_view(), riter(), etc.?
Because, to me, they' re just different words for the same idea.

Well, the only other place we have an 'i' prefix on a function name,
AFAIK, it means "inplace". I don't mind riter(); it has a nice
resonance with rbegin() in C++.

 

[Raymond Hettinger]

[Raymond]
[Steve Holden]

Hmm. My little brain is having difficulty imagining anything that won't.
What am I missing?

The PEP proposes adding a method only to a few objects
that don't have degenerate cases: list, str, unicode, xrange
and possibly tuple.


Raymond Hettinger

 

[Raymond Hettinger]

[Andrew Dalke]

verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith("slack-version-")]

I like your version better and recommend you submit it as a patch.
I'll take out the ifilter() comment.

But that goes against the pretty good motto of "if it ain't broke,
don't touch it."

Between major releases, refactoring is fair game. Using newer
constructs like list comprehensions can make this code more
readable and maintainable.

Granted, XP-style tests are supposed to
help out with that, but I don't have a /usr/lib/setup where I
can test this.

Submit a patch, assign to me, and I'll test it.

Ahh, right. Didn't think about that. You should include that
wording in the PEP.

Okay, done!

I agree. When I do need to go backwards I often forgot to get
both -1s in place. Ie, I'll do (n, 0) instead of (n-1, -1).
Well, don't forgot much and more, but there's some tension in
my mind as I worry if I got it correct or not.

That is a key motivation for the pep!

The backup which uses __getitem__ and negative offsets
does work for all objects, behaves exactly as badly as the
existing iter, and doesn't crash with an infinite iterator, no?

1. Sure, all objects that define __getitem__ in a non-mapping
sense or that don't ignore the index (that was at one time a
common python practice).

2. With infinite iterators, running them to completion causes
the crash. They can be used in other generators, iterators,
or loops that have a break. This is not true with reverse
iterators where the very first call will cause the crash:

itertools.count().next() # this always works
riter(itertools.count()).next() # this always fails

That suggests a UserList.ListMixin might be useful, but it
isn't obvious to me that it would be.

You're the fourth person to suggest it.
If you want to know the issues involved, see my post at:


http://groups.google.com/[email protected]

Should dicts support riteritems, ritervalues?

Heck no! I only want reverse iteration for lists, strings, and xrange
where forward and reverse orders make some sense.

I also think that with these changes,
the PEP is pretty complete and solid.

Thanks for the detailed help getting it to that point.


Raymond