On PEP 322 (ireverse)

[Gonçalo Rodrigues]

I've lost the original thread on my news reader so I'm opening a new
one.

First, i like the name ireverse, much better then inreverse (eek!).

Just a simple comment/question:

- Why a builtin function? Why not just stuff it in the itertools
module? The builtins is already fat as it is, making it fatter is not
the way to go, IMHO. I'd like to remeber the copy module as a case
where a fundamental protocol (__copy__ and __deepcop__) is not in the
builtins but in a standard module.

If it were to be put in itertools i'm definitely +1. On the builtins
-0 and lowering...

With my best regards,
G. Rodrigues

 

[Alex Martelli]

I've lost the original thread on my news reader so I'm opening a new
one.

First, i like the name ireverse, much better then inreverse (eek!).

I still prefer Werner Schiendl's idea of iter.reversed, IF we find a
way to have builtin function iter grow a 'staticmethod' of sorts...!

Just a simple comment/question:

- Why a builtin function? Why not just stuff it in the itertools

It's inappropriate for itertools according to detailed criteria that
Raymond has recently posted here.

module? The builtins is already fat as it is, making it fatter is not

Yes, builtins should indeed be reserved strictly for the indispensable
cases (and trimmed a LOT in 3.0 -- but that's years away). Which is
why, e.g., list.sorted was made a classmethod of type list, rather
than a built-in. Similarly, IF iter was a type, iter.reversed would
be the "one obvious way to do it"...


Alex

 

[Raymond Hettinger]

First, i like the name ireverse, much better then inreverse (eek!).

Thanks.

[Alex]
I still prefer Werner Schiendl's idea of iter.reversed, IF we find a
way to have builtin function iter grow a 'staticmethod' of sorts...!

No thanks. Yuck. Please do not twist this simple idea into knots.
This is supposed to be something you can teach in the first half-hour
and then use forever. I would sooner teach negative xrange()
indices than immediately launch into dotted references to a static
method attached to something that doesn't look like it should have
a method. This is a simplification, not an exercise in being cute
or clever just to avoid a builtin.


Raymond Hettinger

 

[Terry Reedy]

(eek!).

I still prefer Werner Schiendl's idea of iter.reversed, IF we find a
way to have builtin function iter grow a 'staticmethod' of sorts...!

As I suggested elsewhere, make iter() the type object for iterator.
Then iter.reversed is closely parallel to list.sorted.

Which is
why, e.g., list.sorted was made a classmethod of type list, rather
than a built-in. Similarly, IF iter was a type, iter.reversed would
be the "one obvious way to do it"...

Any good reason not to make iter so? int, str, and float were once
bifs also. I'm +1 on iter.reversed, along with list.sorted.

Terry J. Reedy

 

[Roy Smith]

As I suggested elsewhere, make iter() the type object for iterator.
Then iter.reversed is closely parallel to list.sorted.

I've got a question about list.sorted. I assume it's just a list that's
initialized to have its elements sorted when it's created? We're not
talking about some new magic container type which keeps its elements
sorted all the time?

In other words, I'm assuming that

l = list.sorted ((1, 2, 5, 3))
l.append (4)
print l

will print [1, 2, 3, 5, 4], right?

 

[Michael Hudson]

As I suggested elsewhere, make iter() the type object for iterator.

The what?

Cheers,
mwh

 

[Terry Reedy]

I've got a question about list.sorted. I assume it's just a list that's
initialized to have its elements sorted when it's created?

Correct: As I understand it,
l=list.sorted(iterable) == l=list(iterable); l.sort

tjr

 

[Raymond Hettinger]

[Terry Reedy]

Any good reason not to make iter so? int, str, and float were once
bifs also.

The C code for it has a split path, calling __iter__ for objects that
have it, building iterator wrappers for sequences that don't have
__iter__, or building another wrapper when the inputs are a
function and sentinel. IOW, it is a factory function capable
of constructor any of several different types.

I'm +1 on iter.reversed, along with list.sorted.

Please no.

list.sorted() is entirely unrelated. It is attached to lists because that
is what it generally returns and because its functionality is closely
tied to list.sort(). It is a class method because that gave it
richer functionality by taking any iterable as an argument:

list.sorted('a string').

When I posted the PEP for comment, I was looking for confirmation
that the underlying problem is real and for folks to look through their
own code to see if the proposed solution truly offered improvements
in terms of clarity and performance.

Grafting this onto iter is not an option; I would rather lose the
functionality than have the experts here twist it into something
I can't easily explain to a client.


Raymond Hettinger

 

[Terry Reedy]

[Terry Reedy]

Any good reason not to make iter so? int, str, and float were once
bifs also.

The C code for it has a split path, calling __iter__ for objects that
have it, building iterator wrappers for sequences that don't have
__iter__, or building another wrapper when the inputs are a
function and sentinel. IOW, it is a factory function capable
of constructor any of several different types.

Good reason acknowledged.

rescinded.

tjr

 

[Paul Moore]

[Alex]

I still prefer Werner Schiendl's idea of iter.reversed, IF we find a
way to have builtin function iter grow a 'staticmethod' of sorts...!

No thanks. Yuck. Please do not twist this simple idea into knots.
[...]

This is a simplification, not an exercise in being cute
or clever just to avoid a builtin.

Agreed. Keep it simple. For a name, I'd go for ireverse() if reverse()
is not possible. But not inreverse()...

Paul

 

[Fredrik Lundh]

When I posted the PEP for comment, I was looking for confirmation
that the underlying problem is real

well, I've never felt any need for a function that makes it impossible
to reverse a sequence...

(I think the correct form is "irreversible", by the way. I'm not sure
"irreverse" is a real word...)

</F>

 

[Alex Martelli]

As I suggested elsewhere, make iter() the type object for iterator.
Then iter.reversed is closely parallel to list.sorted.

I've got a question about list.sorted. I assume it's just a list that's
initialized to have its elements sorted when it's created? We're not
talking about some new magic container type which keeps its elements
sorted all the time?

In other words, I'm assuming that

l = list.sorted ((1, 2, 5, 3))
l.append (4)
print l

will print [1, 2, 3, 5, 4], right?

Perfectly right. If you DO want a listoid container that keeps its
elements sorted, that's not hard to make for yourself. There are
basically two obvious implementation strategies:
-- ensure the underlying list is re-sorted at each insertion,
append &c, and/or
-- let the underlying list grow as it will on insertion &c, but
make sure it's sorted at any method that accesses it (you can
use a flag -- 'have there being any append/insert/... since
the last sort?' -- to re-sort only when needed).

I'll choose the first for general use because:
-- accesses are much more frequent than modifications,
-- list.sort is VERY fast when a list is "nearly sorted"
-- the second approach would require sorting also on many
kind of changes (e.g. __setitem__ -- to ensure the items
being replaced ARE those the user expects...)
&c.

So, we clearly start with:

class alwayssortedlist(list):

def __init__(self, *args):
list.__init__(self, *args)
self.sort()

We need to ensure sorting at creation. We _might_ make the sort
method in this class a noop and call list.sort(self) instead,
but, naah -- list.sort is SO fast when called on an already
sorted list that it ain't even funny, so, choose simplicity.
Onwards to simple mods:

def append(self, *args):
list.append(self, *args)
self.sort()

def __setitem__(self, *args):
list.__setitem__(self, *args)
self.sort()

....starting to see a pattern, by any chance...?-) Yep, ALL
list mutators we _need_ to override are exactly like this!
(We MAY override e.g. index and remove to take advantage
of the sortedness, but, that's optional.... extend and
insert too (now, reverse IS an interesting issue...!-), and
__iadd__ and __imul__ (methods creating new lists, such as
__add__, __mul__ and __rmul__, are slightly different). Oh,
__setslice__, too.

Well, doing it in longhand is fine, of course, but why not
have a bit more fun with (not showing the __add__ &c):

class solist(list): pass

def solist_method(methname):
list_method = getattr(list, methname)
def method(self, *args):
list_method(self, *args)
self.sort()
setattr(solist, methname, method)
for n in 'init setitem iadd imul'.split:
solist_method('__%s__' % n)
for n in 'append extend insert'.split:
solist_method('%s' % n)

Not much shorter than the boilerplate/longhand, and not
perfect (we should make a new function object to give it
the right func_name -- as is, all the methods' names
will be "method" in tracebacks &c..., but sorta fun.

Testing & completion left as an exercise to the student...


Alex

 

[Raymond Hettinger]

[Alex Martelli]

Perfectly right. If you DO want a listoid container that keeps its
elements sorted, that's not hard to make for yourself. There are
basically two obvious implementation strategies:
-- ensure the underlying list is re-sorted at each insertion,
append &c, and/or
-- let the underlying list grow as it will on insertion &c, but
make sure it's sorted at any method that accesses it (you can
use a flag -- 'have there being any append/insert/... since
the last sort?' -- to re-sort only when needed).

I'll choose the first for general use because:
-- accesses are much more frequent than modifications,
-- list.sort is VERY fast when a list is "nearly sorted"

That could also be an argument for the second approach.
The timsort excels at finding the unsorted part, sorting it,
and merging it with rest. And, if there were any partial
ordering of the unsorted part, it tends to take advantage
of that.

For the second approach, I would keep a sorted flag.
Whenever a new element is added the flag would be
set to False. Upon data access, a False flag indicates
a need to run sort() and then the flag is set to True.
IOW, sort only when needed and sort as many items
at a time as you can.


Raymond Hettinger

 

[Jeremy Fincher]

Is there any reason this PEP is of so limited scope? Why not a more
comprehensive PEP defining a reverse iteration protocol similar the
protocol we have now for forward iteration?

It just seems that if this is implemented as-is, it'll be one of those
warts left around when Python gets a reverse (or even more general)
iteration protocol that will almost certainly operate in a different
manner. It just seems to be fodder for supercession.

Jeremy

 

[Bengt Richter]

As I suggested elsewhere, make iter() the type object for iterator.
Then iter.reversed is closely parallel to list.sorted.

I've got a question about list.sorted. I assume it's just a list that's
initialized to have its elements sorted when it's created? We're not
talking about some new magic container type which keeps its elements
sorted all the time?

In other words, I'm assuming that

l = list.sorted ((1, 2, 5, 3))
l.append (4)
print l

will print [1, 2, 3, 5, 4], right?

Perfectly right. If you DO want a listoid container that keeps its
elements sorted, that's not hard to make for yourself. There are
basically two obvious implementation strategies:
-- ensure the underlying list is re-sorted at each insertion,
append &c, and/or
-- let the underlying list grow as it will on insertion &c, but
make sure it's sorted at any method that accesses it (you can
use a flag -- 'have there being any append/insert/... since
the last sort?' -- to re-sort only when needed).

I'll choose the first for general use because:
-- accesses are much more frequent than modifications,
-- list.sort is VERY fast when a list is "nearly sorted"
-- the second approach would require sorting also on many
kind of changes (e.g. __setitem__ -- to ensure the items
being replaced ARE those the user expects...)
&c.

[...]

No comments on the heapq module?

Regards,
Bengt Richter

 

[Raymond Hettinger]

[Jeremy Fincher]

Why not a more
comprehensive PEP defining a reverse iteration protocol similar the
protocol we have now for forward iteration?

It's in there under the Custom Reverse section. The approach is so
simple it doesn't take much elaboration: Objects may define a
__ireverse__ method that returns a reverse iterator -- this allows
objects without __getitem__ and __len__ to participate in
reverse iteration.


Raymond Hettinger

 

[Alex Martelli]

Bengt Richter wrote:
...

...
No comments on the heapq module?

It can often offer a good _alternative_ to "a listoid container that keeps
its elements sorted" -- keeping them as a heap is cheaper than keeping them
sorted. But "if you DO want" them sorted, so that at any time accessing
mycontainer[x] gives me the x-th smallest item in mycontainer, heapq isn't
gonna be very effective.

There are many good alternatives to "keeping the elements sorted", and
heapq may well help implement some of them, but I wasn't discussing the
alternatives -- just the implementation strategies under the assumption
that the specs DO call for "always sorted".


Alex

 

[Anton Vredegoor]

class alwayssortedlist(list):

def __init__(self, *args):
list.__init__(self, *args)
self.sort()

We need to ensure sorting at creation. We _might_ make the sort
method in this class a noop and call list.sort(self) instead,
but, naah -- list.sort is SO fast when called on an already
sorted list that it ain't even funny, so, choose simplicity.
Onwards to simple mods:

def append(self, *args):
list.append(self, *args)
self.sort()

def __setitem__(self, *args):
list.__setitem__(self, *args)
self.sort()

How about this code:

a = alwaysssortedlist(range(10))
a[5] = 20

What's the use of assigning to a specific list position if the object
could end up being in some other place?

Anton

 

[Alex Martelli]

Anton Vredegoor wrote:
...

How about this code:

a = alwaysssortedlist(range(10))
a[5] = 20

What's the use of assigning to a specific list position if the object
could end up being in some other place?

For an always-sorted list, this clearly means: replace the currently
sixth-lowest item with the value 20. More generally, a always
means "the (i+1)-th lowest item" -- referencing it no doubt going
to be useful much more often than replacing it, and value of i
different from the extremes (say 0, 1, -1) are going to be rare.
But, so what? One can surely imagine use cases (if one can for any
use of an "always-sorted list" rather than sorting on request).

For example: "if there is any occurrence of 20 in the list, then
replace the immediately-smaller item (if any, i.e., unless 20 is
the smallest) with another copy of the value 20". I.e.:

try: i = a.index(20)
except ValueError:
print "value 20 is not in the list"
else:
if i==0:
print "value 20 is in the list, but it's the smallest"
else:
a[i-1] = 20

I don't think I've ever needed to code this kind of thing in
production-code, but it doesn't seem particularly far-fetched
to me. Why else, except for such kinds of uses, would one want
a listoid that's _always_ sorted, rather than a cheaper heap
(see module heapq) or a list that's sorted on request...?


Alex

 

[Michael Hudson]

For an always-sorted list, this clearly means: replace the currently
sixth-lowest item with the value 20. More generally, a always
means "the (i+1)-th lowest item" -- referencing it no doubt going
to be useful much more often than replacing it, and value of i
different from the extremes (say 0, 1, -1) are going to be rare.
But, so what?

Oh, yow. Typing

a[5] = 20

and then having

a[5] != 20

is just icky. Find another notation.

Cheers,
mwh