On PEP 322 (ireverse)

[David Mertz]

|> I'm +1 on iter.reversed, along with list.sorted.

|list.sorted() is entirely unrelated. It is attached to lists because that
|is what it generally returns and because its functionality is closely
|tied to list.sort().

Regardless of the connection to list.sorted(), I'm -1 on a built-in
'reversed()' (or whatever name).

But I'd be +1 on a function or method that lived in either the itertools
modules, or as a member of a classified iter().

There's been WAY too much pollution of the builtin namespace lately.
'reversed()' is nice enough to have not too far away, but not nearly
common enough to need as a builtin.

In fact, I find at least the following anathema in the same way:
enumerate(), sum(), zip(), xrange(). I'm probably forgetting some more.
I'd love to have these in itertools (or maybe sum() could be in math).
But teaching extra builtins is a pain... as is explaining arbitrary
distinctions between what's builtin and what's in itertools. I think if
I had my druthers, I might even put iter() in itertools.

|Grafting this onto iter is not an option; I would rather lose the
|functionality than have the experts here twist it into something
|I can't easily explain to a client.

Hmmm... that's EXACTLY the reason that I DON'T want it as a builtin.

Yours, Lulu...

--
mertz@ | The specter of free information is haunting the `Net! All the
gnosis | powers of IP- and crypto-tyranny have entered into an unholy
..cx | alliance...ideas have nothing to lose but their chains. Unite
| against "intellectual property" and anti-privacy regimes!
-------------------------------------------------------------------------

 

[Jeremy Fincher]

Perfectly right. If you DO want a listoid container that keeps its
elements sorted, that's not hard to make for yourself. There are
basically two obvious implementation strategies:
-- ensure the underlying list is re-sorted at each insertion,
append &c, and/or
-- let the underlying list grow as it will on insertion &c, but
make sure it's sorted at any method that accesses it (you can
use a flag -- 'have there being any append/insert/... since
the last sort?' -- to re-sort only when needed).

I'll choose the first for general use because:
-- accesses are much more frequent than modifications,
-- list.sort is VERY fast when a list is "nearly sorted"
-- the second approach would require sorting also on many
kind of changes (e.g. __setitem__ -- to ensure the items
being replaced ARE those the user expects...)
&c.

Any reason he shouldn't just use bisect.insort?

Jeremy

 

[Alex Martelli]

...
Any reason he shouldn't just use bisect.insort?

What about measuring the performance of the two approaches?

I've shown how incredibly simple the sort-after-whatever-addition-
or-replacement approach is -- the insort one will require some more
coding, but IF it's a lot faster, sure. But don't discount timsort's
performance -- it's often incredibly good.


Alex

 

[Ian McMeans]

Not only is it not effective, but heaps don't keep their elements in
sorted order... so if you wanted a list to stay sorted, heaps aren't
what you want. They're great for finding the smallest element
repeatedly, but not for i'th order statistics in general.

import heapq
h = [1,3,2,4]
heapq.heapify(h)
h [1, 3, 2, 4]
h = [4,3,1,2]
heapq.heapify(h)
h [1, 2, 4, 3]
h.sort()
h

[1, 2, 3, 4]

Bengt Richter wrote:
...

...
No comments on the heapq module?

It can often offer a good _alternative_ to "a listoid container that keeps
its elements sorted" -- keeping them as a heap is cheaper than keeping them
sorted. But "if you DO want" them sorted, so that at any time accessing
mycontainer[x] gives me the x-th smallest item in mycontainer, heapq isn't
gonna be very effective.

There are many good alternatives to "keeping the elements sorted", and
heapq may well help implement some of them, but I wasn't discussing the
alternatives -- just the implementation strategies under the assumption
that the specs DO call for "always sorted".


Alex

 

[Alex Martelli]

Not only is it not effective, but heaps don't keep their elements in
sorted order... so if you wanted a list to stay sorted, heaps aren't

Please don't "top-post": somebody reading your reply can't see what
you ARE responding to. Putting your comments AFTER the phrase you're
commenting on is much kinder to readers.

I said a heap is "not effective" (as a way to sort, in Python) because
the sort method of lists is so much faster. You CAN use a heap to
sort, it's just not an _effective_ way to perform sorting in Python.


Alex

 

[Jeremy Fincher]

What about measuring the performance of the two approaches?

I'll do that when I get back to my own computer. It's interesting, of
course, because although the point of bisect.insort is to use O(log n)
binary search to find the place to insert an element, it's still O(n)
because list.insert is O(n). So if timsort is close to O(n) for
nearly-sorted lists, I wonder if it would actually be faster than
bisect.insort because it's coded in C.

I've shown how incredibly simple the sort-after-whatever-addition-
or-replacement approach is -- the insort one will require some more
coding, but IF it's a lot faster, sure. But don't discount timsort's
performance -- it's often incredibly good.

If timsort is actually faster than bisect.insort in all cases, perhaps
it's time for bisect.insort to be deprecated? It's what I would
naturally use for this case because "that's what it's made for," and
it'd be unfortunate to point Python programmers to the *less*
efficient way to do something.

Jeremy

 

[Alex Martelli]

Jeremy Fincher wrote:
...

If timsort is actually faster than bisect.insort in all cases, perhaps
it's time for bisect.insort to be deprecated? It's what I would
naturally use for this case because "that's what it's made for," and
it'd be unfortunate to point Python programmers to the *less*
efficient way to do something.

operations that modify data are a tad harder to time with timeit.py,
but not impossible with some precautions...:

[alex@lancelot bo]$ timeit.py -c -s'import bisect' -s'LL=range(99,0,3)'
'L=LL[:]'

[alex@lancelot bo]$ timeit.py -c -s'import bisect' -s'LL=range(99,0,3)'
'L=LL[:]; L.append(33); L.sort()'
100000 loops, best of 3: 2.3 usec per loop

alex@lancelot bo]$ timeit.py -c -s'import bisect' -s'LL=range(99,0,3)'
'L=LL[:]; bisect.insort(L, 33)'
100000 loops, best of 3: 4.4 usec per loop

....so, yes -- insort can't hold a candle to a list's sort method when
what you're asking them is to do the very same job. Of course,
bisect.insort_right, aka insort, has different specs -- e.g., if you
DO need to supply its lo and hi parameters, then you're considering
a very different problem from what the sort method is solving.

But yes, bisect is basically an EXAMPLE -- a working example of the
peculiarly-hard-to-code-RIGHT bisection algorithm; maybe we should
mention that in the docs. Lemme see if I can borrow the time machine
for the purpose... done. Now, the docs do clearly say:

"""
The source code may be most useful as a working example of the algorithm
"""

Better, hm?-)


Alex

 

[Aahz]

No thanks. Yuck. Please do not twist this simple idea into knots.
This is supposed to be something you can teach in the first half-hour
and then use forever. I would sooner teach negative xrange()
indices than immediately launch into dotted references to a static
method attached to something that doesn't look like it should have
a method. This is a simplification, not an exercise in being cute
or clever just to avoid a builtin.

Counter-yuck. ;-) I just can't see reverse() by any name being one of
the first things taught about Python.

 

[Aahz]

I still prefer Werner Schiendl's idea of iter.reversed, IF we find a
way to have builtin function iter grow a 'staticmethod' of sorts...!

Raymond has vetoed this, but I don't understand why iter() needs to grow
a staticmethod. After all, it's just a function that can be decorated
with attributes....

 

[Alex Martelli]

Raymond has vetoed this, but I don't understand why iter() needs to grow
a staticmethod. After all, it's just a function that can be decorated
with attributes....

built-in functions normally can't, and in that way they differ from
C-coded functions. So, iter would have to be changed into a different
type that _does_ support some attributes.


Alex

 

[Aahz]

built-in functions normally can't, and in that way they differ from
C-coded functions. So, iter would have to be changed into a different
type that _does_ support some attributes.

Did you mean "Python-coded"? Anyway, I doubt that adding an attribute
to iter() would cause problems the way changing it to a type constructor
would. I'm not suggesting this, mind, just curious why nobody else
mentioned the possibility.

 

[Alex Martelli]

Aahz wrote:
...

Did you mean "Python-coded"? Anyway, I doubt that adding an attribute
Yep.

to iter() would cause problems the way changing it to a type constructor
would. I'm not suggesting this, mind, just curious why nobody else
mentioned the possibility.

I think one needs *more* work to add an attribute to a C-coded function:
it needs to be made into a different type. To add a classmethod to a type
one only needs to supply the classmethod flag in one's C sources -- hard
to see how that might be more troublesome...?


Alex

 

[Aahz]

I think one needs *more* work to add an attribute to a C-coded function:
it needs to be made into a different type. To add a classmethod to a type
one only needs to supply the classmethod flag in one's C sources -- hard
to see how that might be more troublesome...?

Someone claimed that making iter() a type constructor would not be
backward-compatible. This certainly would be.

 

[Alex Martelli]

Someone claimed that making iter() a type constructor would not be
backward-compatible. This certainly would be.

Only if we propagated the [expletive deleted] hack whereby, these days,
int(...) is NOT guaranteed something that's an instance of int. We do
need that hack as a prop on the way to int/long unification, I guess
(sigh), but getting MORE such cases is an icky prospect.

If type iter is an abstract baseclass, calling iter should fail -- that
is what calling basestring does, for example, and AFAIK it's the only
precedent in Python, but it's the common meaning of abstract baseclasses.

If type iter is instantiable, right after "x=iter(...)" I'd really
want to be able to "assert type(x) is iter" -- and of course I can't.

I can't even assert the weaker isinstance(x, iter), not in a backwards
compatible manner, since e.g. iter(x) will gladly return x any time x
is an iterator (def __iter__(self): return self), and iterators need not
subclass a hypothetical type iter.

So, "makint iter() a type constructor" while remaining backwards
compatible would mean "makint iter() a type constructor which hardly
ever returns an instance of iter" -- barf.

Nope -- even though I'd STILL like "iter.reversed", we can't make
iter a type, backwards compatibly, in a DECENT way; we'd really need
a new subtype of "builtin function" which can have attributes.


Alex

 

[Dave Benjamin]

Nope -- even though I'd STILL like "iter.reversed", we can't make
iter a type, backwards compatibly, in a DECENT way; we'd really need
a new subtype of "builtin function" which can have attributes.

Pardon me for missing this, but what was the rationale for not just calling
it "reverse" and putting it in builtins? And likewise for "sort"?

 

[Anna]

Pardon me for missing this, but what was the rationale for not just calling
it "reverse" and putting it in builtins? And likewise for "sort"?

reverse and sort already exist in Python, as list methods.

L.reverse() reverses, in-place, the items of list L.
L.sort([f]) Sorts, in-place, the items of L, comparing items by f;
if f is ommitted, cmp is used as comparison function.
(pg 49, _Python in a Nutshell_)

Anna

 

[Alex Martelli]

Pardon me for missing this, but what was the rationale for not just
calling it "reverse" and putting it in builtins? And likewise for "sort"?

reverse and sort already exist in Python, as list methods.

L.reverse() reverses, in-place, the items of list L.
L.sort([f]) Sorts, in-place, the items of L, comparing items by f;
if f is ommitted, cmp is used as comparison function.
(pg 49, _Python in a Nutshell_)

Yes, the risk of confusion between the existing list methods and the
new functions does play a role in wanting to differentiate their names.

As to whether the new functions (by whatever names) must become
built-ins, that's a separate controversy. Raymond is keen to have
them as built-ins because he views them as "fundamental looping
constructs". Guido has decided that list.sorted will be a
classmethod of list instead (more general, one less built-in),
but has essentially approved some reverse-iterating function as a
built-in (and Raymond has vetoed having it as _anything BUT_ a
built-in) so the only issue where debate is allowed (at least
according to Raymond) is on the exact name -- almost.

Almost, because one alternative may still be open: having a
"reversed range" iterator. I think a built-in 'irange' with an
option to reverse would end up as more generally useful than a
"reverse iterator". But if the choice is only between either
the reverse iterator, or a revrange iterator builtin, I would
be roughly neutral -- having _three_ built-ins for arithmetic
progressions, with range returning a list, xrange a "neither
fish nor fowl" ``almost but not quite a list // almost but not
quite an iterator'', revrange an iterator, is just too messy to
feel comfortable. If we introduced a general irange instead
we'd be ready to deprecate xrange soon, _and_ eventually change
range into a simple list(irange(... -- ah BLISS. Therefore,
to me, introducing irange sounds like a general, clean, wise,
far-seeing approach, while revrange feels too specialized to
"pull its weight" as a built-in. Oh well.


Alex