reduce() anomaly?

[Ville Vainio]

[reduce]

If you don't want to learn a cool concept that will only take you 60
seconds to learn, then you shouldn't be programming! Or you can stick
to loops.

As far as reduce goes, ppl will undoubtedly take a look at the
description, understand it in well under 60 seconds, can't think of
any use for the feature during the next 60 seconds (that wouldn't be
clearer with explicit iteration), and forget it soon after turning the
page. I didn't forget it, just wondered why such an oddball feature
was a builtin. Obviously reduce can rock someone's world, but life is
too short to bother if it doesn't rock yours.

and powerful feature with a slew of specific, tailored features. If
reduce() can be relegated to a library or for the user to implement
for himself, then so can sum(). If the language is to only have one,
it should be reduce().

I also think that reduce, sum, map and filter (and lots of others,
__builtins__ has *too much stuff*) should be removed from builtins,
but that will probably take some time (1997 years?). LC's and genexpes
will take care of most of that stuff. And people can always do:

from funtional import *
# map, filter, reduce, curry, ... (I want lots of these

There are also tons of functions that should be in sys, math or
whatever:

reload, repr, divmod, max, min, hash, id, compile, hex...

What's your pet deprecation candidate? I have always thought
`backticks` as repr has got to be the most useless feature around.

 

[Erik Max Francis]

Ah, that reminds me -- both sum() and reduce() can be removed from
Python by extending operator.add so that it will take any number of
arguments.

reduce can't, since reduce doesn't require the function passed to be
operator.add.

--
Erik Max Francis && (e-mail address removed) && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \
\__/ The golden rule is that there are no golden rules. -- George
Bernard Shaw

 

[BW Glitch]

No, my mantra plainly states to keep general and powerful features
over specific, tailored features. reduce() is more general and
powerful than sum(), and would thus clearly be preferred by my
mantra.

The mantra "there should be only one obvious way to do it" apparently
implies that one should remove powerful, general features like
reduce() from the language, and clutter it up instead with lots of
specific, tailored features like overloaded sum() and max().

That's not what _I_ understand. There's room for powerful features, but
when these features gives trouble to the people who just want something
done, then another approach should be taken.

And I'm not talking about stupid people. I'm talking about the
microbiolgist/chemist/physics/etc who is programming because of need. CS
people would do a better job, but they are more costly to bring up in
any proyect that requires especific knowledge in one area.

If so,
clearly this mantra is harmful, and will ultimately result in Python
becoming a bloated language filled up with "one obvious way" to solve
every particular idiom. This would be very bad, and make it less like
Python and more like Perl.

You feel ok? Perl's mantra is "More Than One Way To Do It"...

I can already see what's going to happen with sum(): Ultimately,
people will realize that they may want to perform more general types
of sums, using alternate addition operations. (For intance, there may
be a number of different ways that you might add together vectors --
e.g, city block geometry vs. normal geometry. Or you may want to add
together numbers using modular arithmetic, without worrying about
overflowing into bignums.) So, a new feature will be added to sum()
to allow an alternate summing function to be passed into sum(). Then
reduce() will have effectively been put back into the language, only
its name will have been changed, and its interface will have been
changed so that everyone who has taken CS-101 and knows off the top of
their head what reduce() is and does, won't easily be able to find it.

I don't get you. There's a reason why special functions can be
overloaded (__init__, __cmp__, etc.; I don't use others very often).
That would allow for this kind of special treatments. Besides GvR would
not accept your scenario.

Also, whytf do you mention so much CS101? Maybe you took the course with
LISP, assembly and Scheme, but AFAIK, not everyone has/had access to
this course. Many people learned to program way before taking CS101.
IMHO, you think that the only people that should make software is a CS
major.

Yes, there are other parts of The Zen of Python that point to the
powerful and general, rather than the clutter of specific and
tailored, but nobody seems to quote them these days, and they surely
are ignoring them when they want to bloat up the language with
unneccessary features like overloaded sum() and max() functions,
rather than to rely on trusty, powerful, and elegant reduce(), which
can easily and lightweightedly do everything that overloaded sum() and
max() can do and quite a bit more.

GvR (or BDFL, as most people know him) has been very careful with his
design decisions. I've been only for about 2 years 'round here, but I've
seen why list compressions came by, why there's no ternary operator and
why Python uses indentation as block separations. These are design
decisions that GvR took.

And there's a good reason why they _are_ design decisions. (Try to guess
why. )

Or b + a. Perhaps we should prevent that, since that makes two
obviously right ways to do it!

Even without any algebra, any kid can tell you that 1 + 2 is the same as
2 + 1. Replace 1 and 2 by a and b and you get the same result.

[snip]

C'mon -- all reduce() is is a generalized sum or product. What's
there to think about? It's as intuitive as can be. And taught in
every CS curiculum. What more does one want out of a function?

|>oug

It wasn't obvious for me until later. reduce() is more likely to be used
for optimization. IIRC, some said that optimization is the root of all evil.

Just because it's _obvious_ to you, it doesn't mean it's obvious to
people who self taught programming.

--
Andres Rosado

-----BEGIN TF FAN CODE BLOCK-----
G+++ G1 G2+ BW++++ MW++ BM+ Rid+ Arm-- FR+ FW-
#3 D+ ADA N++ W OQP MUSH- BC- CN++ OM P75
-----END TF FAN CODE BLOCK-----

"Well, That's Just Prime"
"Shut up, Rattrap."
-- Rattrap and Optimus Primal, innumerable occasions

 

[Terry Reedy]

And then there are the corner cases, e.g. sum([]) vs.
reduce(operator.add,[]) (which throws an exception).

The proper comparison is to reduce(operator.add, [], 0), which does
not throw an exception either. sum(seq, start=0) is equivalent to
reduce(operator.add, seq, start=0) except that sum excludes seq of
string. (The doc specifically says equivalent for number (int) seqs,
so there might be something funny with non-number, non-string seqs.)
In other words, sum is more-or-less a special case of reduce with the
within-context constants operator.add and default start=0 built in
(and the special special case of seq of strings excluded).

I think it a big mistake (that should be repaired in 3.0) that the
result start value was made optional, leading to unnecessary empty-seq
exceptions. I also think the order is wrong and should also be fixed.
I believe it was placed last, after the seq of update values, so that
it could be made optional (which it should not be). But it should
instead come before the seq, to match the update function (result,
seq-item) arg order. This reversal has confused people, including
Guido.

(Having said that, I never
personally argued that reduce() should be removed from the language,
but I do agree that it does not have to be part of "core" Python,
and could easily be relegated to a module.)

If the builtins are reduced in 3.0, as I generally would like, I would
be fine with moving apply, map, filter, and a repaired version of
reduce to a 'fun'ctional or hof module. But the argument of some
seems to be that this batteries-included language should specifically
exclude even that.

Terry J. Reedy

 

[Douglas Alan]

Douglas Alan wrote:

reduce can't, since reduce doesn't require the function passed to be
operator.add.

Well, as I said, for this to be true, *all* binary operators (that it
makes sense to) would have to be upgraded to take an arbitrary number
of arguments, like they do in Lisp.

|>oug

 

[Douglas Alan]

That's not what _I_ understand. There's room for powerful features,
but when these features gives trouble to the people who just want
something done, then another approach should be taken.

If there's a problem with people not understaning how to sum numbers
with reduce(), then the problem is with the documentation, not with
reduce() and the documentation should be fixed. It is quite easy to
make this fix. Here it is:

FAQ
---
Q: How do I sum a sequence of numbers?

A: from operator import add
reduce(add, seq)

Problem solved.

And I'm not talking about stupid people. I'm talking about the
microbiolgist/chemist/physics/etc who is programming because of
need.

If a microbiologist cannot understand the above, they have no business
being a microbiologist. I learned reduce() in high school, and it
didn't take me any longer than the 60 seconds I claim it will take
anyone with a modicum of intelligence.

You feel ok? Perl's mantra is "More Than One Way To Do It"...

If both the mantras cause a language to have general features
replaced with a larger number of specialized features that accomplish
less, then both mantras are bad.

I don't get you. There's a reason why special functions can be
overloaded (__init__, __cmp__, etc.; I don't use others very
often). That would allow for this kind of special
treatments. Besides GvR would not accept your scenario.

There are often multiple different ways to add together the same data
types, and you wouldn't want to have to define a new class for each
way of adding. For instance, you wouldn't want to have to define a
new integer class to support modular arithmetic -- you just want to
use a different addition operation.

Also, whytf do you mention so much CS101?

Because anyone who has studied CS, which should include a significant
percentage of programmers, will know instantly where to look for the
summing function if it is called reduce(), but they won't necessarily
know to look for sum(), since languages don't generally have a
function called sum(). And everyone else will not know to look for
either, so they might as well learn a more powerful concept in the
extra 30 seconds it will take them.

Maybe you took the course with LISP, assembly and Scheme, but AFAIK,
not everyone has/had access to this course. Many people learned to
program way before taking CS101.

As did, I, and I had no problem with reduce() when I learned it long
before I took CS-101.

IMHO, you think that the only people that should make software
is a CS major.

Did I say that?

Even without any algebra, any kid can tell you that 1 + 2 is the same
as 2 + 1. Replace 1 and 2 by a and b and you get the same result.

Yes, but they are still two *different* ways to to get to that result.
Starting with a and adding b to it, is not the same thing as starting
with b and adding a to it. It is only the commutative law of
arithmetic, as any good second grade student can tell you, that
guarantees that the result will be the same. On the other hand, not
all mathematical groups are albelian, and consequently, a + b != b + a
for all mathematical groups.

It wasn't obvious for me until later. reduce() is more likely to be
used for optimization. IIRC, some said that optimization is the root
of all evil.

I don't know what you are getting at about "optimization". Reduce()
exists for notational convenience--i.e., for certain tasks it is easer
to read, write, and understand code written using reduce() than it
would be for the corresponding loop--and understanding it is no more
difficult than understanding that a summing function might let you
specify the addition operation that you'd like to use, since that's
all that reduce() is!

Just because it's _obvious_ to you, it doesn't mean it's obvious to
people who self taught programming.

It was obvious to me when I was self-taught and I taught myself APL in
high-school. It also seemed obvious enough to all the physicists who
used APL at the lab where I was allowed to hang out to teach myself
APL.

|>oug

 

[Alex Martelli]

Terry Reedy wrote:
...

If the builtins are reduced in 3.0, as I generally would like, I would
be fine with moving apply, map, filter, and a repaired version of
reduce to a 'fun'ctional or hof module. But the argument of some
seems to be that this batteries-included language should specifically
exclude even that.

A functional module would be neat. A great way to enhance the chance
that there will be one would be starting one today (e.g. on sourceforge),
ideally with both pure-Python and C-helped (or pyrex, etc) implementations,
and get it reasonably-widely used, debugged, optimized. There's plenty
of such ideas around, but gathering a group of people particularly keen
and knowledgeable about functional programming and hashing out the "best"
design for such a module would seem to be a productive endeavour.


Also, I advocate that 3.0 should have a module or package (call it
"legacy" for example) such that, if you started your program with
some statement such as:

from legacy import *

compatibility with Python 2.4 or thereabouts would be repaired as
much as feasible, to ease running legacy code, and to the expense
of performance, 'neatness' and all such other great things if needed
(e.g., no "repaired" versions or anything -- just compatibility).

One reasonably popular counterproposal to that was to have it as

from __past__ import *

by analogy with today's "from __future__". I'd also like to make it
easy to get this functionality with a commandline switch, like is
the case today with -Q specifically for _division_ legacy issues.


But mostly, each time I mention that on python-dev, I'm rebuffed with
remarks about such issues being way premature today. Somebody's
proposed starting a new list specifically about 3.0, to make sure
remarks and suggestions for it made today are not lost about more
day-to-day python-dev traffic, but I don't think anything's been
done about that yet.


Alex

 

[Alex Martelli]

Well, as I said, for this to be true, *all* binary operators (that it
makes sense to) would have to be upgraded to take an arbitrary number
of arguments, like they do in Lisp.

Your definition of "operator" appears to be widely at variance with
the normally used one; I've also noticed that in your comparisons of
reduce with APL's / , which DOES require specifically an OPERATOR (of
the binary persuasion) on its left. reduce has no such helpful
constraints: not only does it allow any (callable-as-binary) FUNCTION
as its first argument, but any other CALLABLE at all. Many of the
craziest, most obscure, and worst-performing examples of use of
reduce are indeed based on passing as the first argument some callable
whose behaviour is anything but "operator-like" except with respect to
the detail that it IS callable with two arguments and returns something
that may be passed back in again as the first argument on the next call.
[see note later].


Anyway, to remove 'reduce' by the trick of "upgrading to take an
arbitrary number of arguments", that "upgrade" should be applied to
EVERY callable that's currently subsceptible to being called with
exactly two arguments, AND the semantics of "calling with N arguments"
for N != 2 would have to be patterned on what 'reduce' would do
them -- this may be totally incompatible with what the callable does
now when called with N != 2 arguments, of course. For example,
pow(2, 10, 100)
now returns 24, equal to (2**10) % 100; would you like it to return
10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376
instead...?-)

I doubt there would be any objection to upgrading the functions in
module operator in the way you request -- offer a patch, or make a
PEP for it first, I would hope it would be taken well (I can't speak
for Guido, of course). But I don't think it would make much more of
a dent in the tiny set of reduce's current use cases.


[note on often-seen abuses of FP built-ins in Python]

Such a typical abuse, for example, is connected with the common idiom:

for item in seq: acc.umul(item)

which simply calls the same one-argument callable on each item of seq.
Clearly the idiom must rely on some side effect, since it ignores the
return values, and therefore it's totally unsuitable for shoehorning
into "functional-programming" idioms -- functional programming is
based on an ABSENCE of side effects.

Of course, that something is totally inappropriate never stops fanatics
of functional programming, that have totally misunderstood what FP is
all about, from doing their favourite shoehorning exercises. So you see:

map(acc.umul, seq)

based on ignoring the len(seq)-long list of results; or, relying on the
fact that acc.umul in fact returns None (which evaluates as false),

filter(acc.umul, seq)

which in this case just ignores an _empty_ list of results (I guess
that's not as bad as ignoring a long list of them...?); or, of course:

reduce(lambda x, y: x.umul(y) or x, seq, acc)

which does depend strictly on acc.umul returning a false result so
that the 'or' will let x (i.e., always acc) be returned; or just to
cover all bases

reduce(lambda x, y: (x.umul(y) or x) and x, seq, acc)


Out of all of these blood-curling abuses, it seems to me that the
ones abusing 'reduce' are the very worst, because of the more
complicated signature of reduce's first argument, compared to the
first argument taken by filter, or map with just one sequence.

To be sure, one also sees abuses of list comprehensions here:

[acc.umul(item) for item in seq]

which basically takes us right back to the "abuse of map" case.
List comprehension is also a rather FP-ish construct, in fact,
or we wouldn't have found it in Haskell to steal/copy it...;-).


Alex

 

[Alex Martelli]

BW Glitch wrote:
...

It wasn't obvious for me until later. reduce() is more likely to be used
for optimization. IIRC, some said that optimization is the root of all
evil.

That's *PREMATURE* optimization (and "of all evil IN PROGRAMMING", but
of the two qualifications this one may be less crucial here) -- just like
misquoting "The LOVE OF money is the root of all evil" as "MONEY is the
root of all evil", so does this particular misquote trigger me;-).

Optimization is just fine IN ITS PROPER PLACE -- after "make it work"
and "make it right", there MAY come a time where "make it fast" applies.

It's extremely unlikely that 'reduce' has a good place in an optimization
phase, of course -- even when some operator.foo can be found:

[alex@lancelot tmp]$ timeit.py -c -s'import operator' -s'xs=range(1,321)'
'r=reduce(operator.mul, xs)'
1000 loops, best of 3: 450 usec per loop

[alex@lancelot tmp]$ timeit.py -c -s'import operator' -s'xs=range(1,321)'
'r=1' 'for x in xs: r*=x'
1000 loops, best of 3: 440 usec per loop

reduce shows no advantage compared with a perfectly plain loop, and
when no operator.bar is around and one must use lambda:

[alex@lancelot tmp]$ timeit.py -c -s'import operator' -s'xs=range(1,321)'
'r=reduce(lambda x, y: pow(x, y, 100), xs)'
1000 loops, best of 3: 650 usec per loop

[alex@lancelot tmp]$ timeit.py -c -s'import operator' -s'xs=range(1,321)'
'r=1' 'for x in xs: r = pow(r, x, 100)'
1000 loops, best of 3: 480 usec per loop

reduce gets progressively slower and slower than the pian loop. It's
just no bloody good, except maybe for people short-sighted enough to
believe that it's "more concise" than the loop (check out the lengths
of the timed statements above to dispel THAT myth) and who'd rather
slow things down by (say) 35% than stoop to writing a shorter, plainer
loop that every mere mortal would have no trouble understanding.

Just because it's _obvious_ to you, it doesn't mean it's obvious to
people who self taught programming.

That may be the real motivation for the last-ditch defenders of reduce:
it's one of the few (uselessly) "clever" spots in Python (language and
built-ins) where they may show off their superiority to mere mortals,
happily putting down as sub-human anybody who doesn't "get" higher-order
functions in 10 seconds flat (or less)...;-)


Alex

 

[Patrick Maupin]

And then there are the corner cases, e.g. sum([]) vs.
reduce(operator.add,[]) (which throws an exception).

The proper comparison is to reduce(operator.add, [], 0), which does
not throw an exception either. sum(seq, start=0) is equivalent to
reduce(operator.add, seq, start=0) except that sum excludes seq of
string. (The doc specifically says equivalent for number (int) seqs,
so there might be something funny with non-number, non-string seqs.)
In other words, sum is more-or-less a special case of reduce with the
within-context constants operator.add and default start=0 built in
(and the special special case of seq of strings excluded).

I agree. Please remember that my post was arguing that it would
take more than 30 seconds to teach someone reduce() instead of sum().
This is precisely because sum() was deliberately chosen to special-case
the most common uses of reduce, including not only the add operator,
but also the default initial value.

I think it a big mistake (that should be repaired in 3.0) that the
result start value was made optional, leading to unnecessary empty-seq
exceptions. I also think the order is wrong and should also be fixed.
I believe it was placed last, after the seq of update values, so that
it could be made optional (which it should not be). But it should
instead come before the seq, to match the update function (result,
seq-item) arg order. This reversal has confused people, including
Guido.

That makes some sense, but you'd _certainly_ have to move and/or rename
the function in that case. There's breaking some backward compatibility,
and then there's taking a sledgehammer to things which used to work
perfectly.

As an aside, if sum() grew an optional _third_ parameter, which
was the desired operator, sum() would FULLY recreate the functionality
of reduce(), but with the same default behavior that was deemed
desirable enough to create the sum() function. This is similar to
your preferred form in that the starting value is not optional
when you specify the operator (simply because you have to use
three parameters to specify the operator), differing only in
the order of the first two parameters.

Although this is not quite your preferred form, perhaps those in such
a rush to remove reduce() should consider this slight enhancement to
sum(), to mollify those who don't want to see the functionality disappear,
but who could care less about the name of the function which provides
the functionality (e.g. sum(mylist,1,operator.mul) is slightly
counterintuitive).

If the builtins are reduced in 3.0, as I generally would like, I would
be fine with moving apply, map, filter, and a repaired version of
reduce to a 'fun'ctional or hof module. But the argument of some
seems to be that this batteries-included language should specifically
exclude even that.

I'm taking a wait-and-see attitude on this. I don't think any of the
implementers have such a vested interest in being "right" that these
functions will be removed at all cost. As soon as the implementers
start porting their _own_ code to 3.0, I believe we'll starting getting
useful feedback, either of the form "itertools et al. has everything
I need", or "boy, this SUCKS! I really need map!"

(I'm curious, though, why you included "apply" in this list. I've
personally never needed it since the * enhancement to the call syntax.)

Regards,
Pat

 

[BW Glitch]

BW Glitch wrote:
...



That's *PREMATURE* optimization (and "of all evil IN PROGRAMMING", but
of the two qualifications this one may be less crucial here) -- just like
misquoting "The LOVE OF money is the root of all evil" as "MONEY is the
root of all evil", so does this particular misquote trigger me;-).

Sorry, I didn't remember the quote completely. :S But that was the point
I wanted to make.

Andres Rosado

 

[Alex Martelli]

Ville Vainio wrote:
...

I also think that reduce, sum, map and filter (and lots of others,
__builtins__ has *too much stuff*) should be removed from builtins,
but that will probably take some time (1997 years?). LC's and genexpes
will take care of most of that stuff. And people can always do:

There are no way in which LC's and genexp can "take care of" CONSUMING
iterables and iterators -- they PRODUCE them (and itertools also do
some production, and, mostly, composable transformation). map and
filter, sure; sum, reduce, min, max, can never be replaced by any
LC nor any genexp. I think we need a separate module for _consumers_
of iterables/iterators, and I have repeatedly posted about it. But,
sure, I'd gladly agree that there's no real call for any of these to
be a built-in.

from funtional import *
# map, filter, reduce, curry, ... (I want lots of these

I'd rather put map/filter/reduce in a 'legacy' module -- warts and
all, e.g. map's horrible habit of None-padding shorter sequences,
EEK -- and have a designed-from-scratch functional module without
the warts. What about starting one as a sourceforge project, as I
mentioned elsewhere?

There are also tons of functions that should be in sys, math or
whatever:

reload, repr, divmod, max, min, hash, id, compile, hex...

Probably, yes. What functions, as opposed to types, do you
think should absolutely be in the builtins rather than in a separate
module, and (unless it's obvious) why? Of course __import__ had
better stay or we won't be able to get anything imported, for
example. The attribute-related functions hasattr, getattr,
setattr, and delattr, seem very fundamental (but I'd want the
same status for the item-functions currently over in operator),
as well as (at least some of them -- delattr isn't -- but I do
think that trying to discriminate too finely would make things
too hard to learn) very frequently used in code. What else?
iter, len, pow [for the crucial 3-arguments case], range (or some
preferable variant that returns an iterator), and zip seem pretty
fundamental; chr and ord might be suitable as str methods (in
the case of chr, a staticmethod, no doubt). Many functions that
are quite handy for interactive use, such as locals, globals,
dir, vars, ..., are not all that frequently used in programs --
so they might live in a module that the interactive mode gets
automatically, rather than being built-ins.

Having beginners learn 'import' before they do raw_input (or
output, which should also be a function, not a statement) may
not agree with somebody's didactics, so, we should consider how
to deal with those.

All of this would be perfect for the mailing list on Python 3.0
if the latter existed. Posting it to c.l.py makes it unlikely
Guido will ever consider the discussion's resuts, anyway. The
problem is that, with 3.0 at least 2 years away, there seems to
be little incentive to actually go and make such a list, so that
its archives may come in handy in the future.

What's your pet deprecation candidate? I have always thought
`backticks` as repr has got to be the most useless feature around.

Pretty bad, yes. 'apply' at least, while useless, doesn't make
Python's syntax any more complicated, while `...` does.


Alex

 

[Alex Martelli]

Douglas Alan wrote:
...

The argument that some programmers might be too lazy to want to learn
powerful, simple, and elegant features that can be taught in seconds,

Programmers may be quite ready to learn anything whose claimed "power"
IS actually there. But this complex of thread started with somebody
asking for good use cases for reduce, and NONE has been shown.

Programmers need not have any interest in spending what may well be
more than 'seconds' in learning something that will never given them
any *practical* return. Computer "scientists", maybe, have to be, by
some definition of "scientist". Fortunately, *programmers* are still
allowed to be very pragmatical types instead.

Besides, if you weren't exposed at all to LISP (or a LISP-like
language) while getting a CS degree, it wasn't a very good CS
program! They're going to teach you AI techniques in a different

Aha, showing your true colors: I see you have now moved from a
"CS 101 course" (which is given to majors in MANY disciplines)
to a "CS degree". Quite a different thing. Around here (for the
same length of course, say 3 years), graduates in "informatical
engineering", for example, may well not know what a Turing machine
is, nor be able to name any "AI technique" of practical use (they
may have learned, e.g. alpha-beta pruning, which historically did
originate within AI, but may be more practical today to learn in
completely different contexts) -- but they're likely to know more
than graduates in "informatics" (computer science) about, e.g.,
statistics, and/or how to organize a security-audit. (I have to
phrase it in terms of likelihood because most majors do offer quite
a bit of choice in terms of what exact courses you can take).

I wouldn't be surprised to find an "ingegnere informatico" (3-years
degree, i.e. BS-equivalent) who doesn't understand 'reduce' at
first; and once he or she has grasped it, I _would_ be surprised
not to get challenged with an "OK, now, what IS it GOOD for?".

When Plato was asked the same question about Geometry, he had a
slave give the querant a gold coin then throw him out of the school:
caring about USEFULNESS was OH so icky to Greek philosophers! I
would not be surprised if a similar stance (maybe not quite as
overt) lingers in several academic environments. Fortunately,
though, it seems to me that Engineering faculties have managed
to stay pretty free of it (yep, I'm an engineer, and proud of it,
and proud that my daughter has chosen an engineering major too --
my son chose Economics, a lovely calling to be sure, and is having
a much harder time selecting the courses that concentrate on the
_useful_ stuff ["how do I make a cool million before I'm 30" kind
of stuff] from those which are the Economics equivalent of
"Suppose cows were spheres of uniform density"....


Alex

 

[Terry Reedy]

That makes some sense, but you'd _certainly_ have to move and/or rename
the function in that case.

I agree on moving. I might rename reduce as induce simultaneously.

Although this is not quite your preferred form, perhaps those in such
a rush to remove reduce() should consider this slight enhancement to
sum(), to mollify those who don't want to see the functionality disappear,
but who could care less about the name of the function which provides
the functionality (e.g. sum(mylist,1,operator.mul) is slightly
counterintuitive).

The problem with sum(seq, [base=0, [op = add]]) is that it *still* has
the result base case and the seq params in what I consider to be the
wrong order (unless the optional update function has the order of its
parameters reverses from reduce) -- and for the same reason of
optionality.

I need to write a coherent summary of my views on the ++s and --s of
reduce.

functions will be removed at all cost. As soon as the implementers
start porting their _own_ code to 3.0, I believe we'll starting getting
useful feedback, either of the form "itertools et al. has everything
I need", or "boy, this SUCKS! I really need map!"

I presume that part of why Guido wants a year to do 3.0 is just so he
can test alternatives with ports of his own code.

(I'm curious, though, why you included "apply" in this list. I've
personally never needed it since the * enhancement to the call

syntax.)

I presume you mean as opposed to delete it altogether. Because apply
can be used with map while * and ** cannot and at least one person has
said he uses that.

Terry J. Reedy

 

[Alex Martelli]

Douglas Alan wrote:
...

I find this truly hard to believe. APL was a favorite among
physicists who worked at John's Hopkins Applied Physics Laboratory
where I lived for a year when I was in high school, and you wouldn't

Interesting. I worked for many years in an environment where physicists
doing research could freely choose between APL and Fortran (IBM Research),
and while there was a hard-core of maybe 10%-15% of them who'd _never_
leave APL for any reason whatsoever, an overwhelmingly larger number
of physicist was at least as keen on Fortran. I have no hard data on
the subject, but it appears to me that Fortran has always been way more
popular than APL among physicists as a whole.

thing. In fact, people seemed to like reduce() and friends -- people
seemed to think it was a much more fun way to program, rather than
using boring ol' loops.

....while most physicists I worked with were adamant that they wanted
to continue coding loops and have the _compiler_ vectorize them or
parallelize them or whatever. Even getting them to switch to Linpack
etc from SOME of those loops was a battle at first, though as I recall
the many advantages did eventually win them over.

Anyway, computational scientists using Python should be using Numeric
(if they aren't, they're sadly misguided). Numeric's ufuncs ARE the
right way to do the equivalent of APL's +/ (which is quite a different
beast than ANYusercodedfunction/ would be...), and show clear and
obvious advantages in so doing:

[alex@lancelot tmp]$ timeit.py -c -s'import operator; xs=range(999)'
'x=reduce(operator.add, xs)'
1000 loops, best of 3: 290 usec per loop

[alex@lancelot tmp]$ timeit.py -c -s'xs=range(999)' 's=sum(xs)'
10000 loops, best of 3: 114 usec per loop

[alex@lancelot tmp]$ timeit.py -c -s'import Numeric as N;
xs=N.arrayrange(999)' 'x=N.add.reduce(xs)'
100000 loops, best of 3: 9.3 usec per loop

Now *THAT* is more like it: 10+ times FASTER than sum, rather than
2+ times SLOWER! Of course, you do have to use it right: in this
snippet, if you initialize xs wrongly...:

[alex@lancelot tmp]$ timeit.py -c -s'import Numeric as N; xs=range(999)'
'x=N.add.reduce(xs)'
100 loops, best of 3: 2.1e+03 usec per loop

....then you can say goodbye to performance, as you see. But when
used skilfully, Numeric (or its successor numarray, I'm sure -- I
just don't have real experience with the latter yet) is just what
numerically heavy computations in Python require.


Alex

 

[Alex Martelli]

Dave Brueck wrote:
...

Why, pray-tell, would you want an OO program to do:

results = [ func(x) for x in sequence ]

... instead of ...

results = sequence.map(func) ??

Because I find the first much more readable (and IMO the "an OO program to
do" bit is irrelevent from a practical point of view).

I entirely agree with both points. They're even clearer when the
contrast is between, e.g.:

results = [ x+23 for x in sequence ]

and:

results = sequence.map(lambda x: x+23)

where using the HOF approach forces you to understand (and read) lambda too.


Alex

 

[Alex Martelli]

that kind of array handling. But the point here is that "reduce" is
fundamental: x/i5 (where x is multiplication-sign and i is iota) is
a lot like reduce(int.__mul__, range(1,6)), it's just "readable" if

I disagree about "is a lot like" -- I think Numeric Python's

mul.reduce(arrayrange(1, 6))

is much closer to APL (sure, spelling "iota 5" as "arrayrange(1, 6)"
may be alien, but the point is that .reduce in Numeric, like / in APL,
is a property of OPERATORS [ufuncs, in Numeric], NOT a generic FP
tool like Python's built-in reduce).

of poking around.) On the other hand, that readability does assume
you're thinking in terms of throwing arrays around, which can be
an... *odd* way of looking at things, though of course when it fits,
it's very nice.

That's the reason I'm happier to have it in a separate package, be
it Numeric or its successor numarray, than built into the language --
just like, e.g., regular expressions are a separate module in Python
but built into languages such as Perl or Ruby. Keeping the core
language simpler and other modules/packages rich is a strategy which
I think Python uses to excellent effect.


Alex

 

[Alex Martelli]

Robin Becker wrote:
...

on oop in this thread nobody has pointed out that we could have

sequence.sum()
sequence.reduce(operator.add[,init])

or even

sequence.filter(func) etc etc

and similar. That would make these frighteningly incomprehensible
concepts seem less like functional programming. Personally I wouldn't
like that to happen.

Maybe nobody "pointed it out" because it's utterly absurd to even
conceive of somehow magically adding all of those methods to *EVERY*
iterator under heavens?! E.g., today, this works just fine:
.... def __init__(self): self.seq = [1, 5, 3, 7, 19]
.... def __iter__(self): return self
.... def next(self):
.... try: return self.seq.pop()
.... except IndexError: raise StopIteration
....35

having instances of class BAH automagically spout such methods as
..sum(), etc, would be an absurdity.

Numeric's design -- considering that *functions* able to do .reduce
and the like are very special and should expose those as methods
(they're known as "ufuncs" in Numeric), taking the array as argument,
is a design which looks quite sound to me. So, for example,
Numeric.sum(x, axis) boils down (after some error checks &c) to
Numeric.add.reduce(x, axis) [summing, of course, IS by far important
enough that having to spell it out as add.reduce would be silly].


Alex

 

[Alex Martelli]

Douglas Alan wrote:
...

How's that? I've never used a programming language that has sum() in
it. (Or at least not that I was aware of.) In fact, the *Python* I
use doesn't even have sum() in it! I've used a number of languages

Never used a modern Fortran? Or never bothered to _learn_ it properly
"because it's a language for peasants"?-)

that have reduce(). If I didn't already know that it exists, I
wouldn't even think to look in the manual for a function that adds up
a sequence of numbers, since such a function is so uncommon and
special-purpose.

Right -- that's why the Fortran standard committee bothered to include
it, the Numeric Python designers bothered to define sum specifically
as an alias for add.reduce, and so on -- because it's so uncommon.

Indeed, that's why 'sum' is a commonly used word in English -- exactly
because nobody's every DOING anything like that -- while, of course,
'reduce' is totally obvious -- why, _anybody_ who's ever taken
Chemistry 101 knows it means "to remove oxygen"!


Alex

 

[David Eppstein]

How's that? I've never used a programming language that has sum() in
it. (Or at least not that I was aware of.) In fact, the *Python* I
use doesn't even have sum() in it! I've used a number of languages

Never used a modern Fortran? Or never bothered to _learn_ it properly
"because it's a language for peasants"?-)[/QUOTE]

For that matter, Haskell, the third thing I get from googling
"functional programming language", has had sum since it was first
created (version 1.0 report 1990), despite its being easily replaceable
with reduce (or I guess in Haskell terms foldl).