map/filter/reduce/lambda opinions and background unscientificmini-survey

[Robert Kern]

Peter Hansen wrote:


just inductive reasoning. i've been wrong before (like anyone who makes
that claim), and i'm a former python enthusiast, so my judgement must
be colored to some extent by bitterness. maybe they have solid reasons
for scrapping the functional constructs. but to me it seems like
they're eliminating them just because they offend the sensibilities of
C-programmers.

This is incorrect.

(i mean those stereotypical C-programmers, baffled by
recursion and the like, who don't want to be reproached with the fact
of their mathematical illiteracy.) if that's the case then list
comprehensions and/or "first class functions" are likely to be the next
target.

map and filter are being removed *because of* list comprehensions. Did
you even read Guido's articles about this issue? Your understanding of
why these changes are planned is incorrect; consequently your projection
based on that understanding is not on firm footing.

even if they're not, it's pretty clear that python is leaving
its multiparadigmatic origins behind. "do it our way," the pundits are
effectively saying, "or get out". for my part, i'm getting out.

If that's what you want to do, no one is going to stop you. But please
do it quietly.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Mike Meyer]

Peter Hansen wrote:

if that's the case then list comprehensions and/or "first class
functions" are likely to be the next target.

The existence of list comprehensions are the reason that these
functions are going away, so they aren't likely to be next. It's all
part of "There should be one-- and preferably only one --obvious way
to do it."

Personally, I hope they wind up in a "functional" module, so you can add
"from functional import *" to the top of your scripts, and keep doing
exactly what you've been doing.

<mike

 

[Ron Adam]

Seems like some new idioms would have to be coined, like:

def my_function(a1, a2):
def _(a,b): return a+b
call_a_lib_w_callback(callback=_)

doesn't seem too bad, and defeats the "wasting time thinking of names"
argument.

Usually the time is regained later when you need to go back and figure
out what it is that the lambda is doing. Not so easy for beginners.

A hard to understand process that is easy to do, is not easier than an
easy to understand process that is a bit harder to do.

So what would be a good example of a lambda that couldn't be replaced?

I suspect the hardest would be building a list of functions. Something
like:

powers = [lambda a, i=i: a**i for i in range(10)]

which you might be able to make like this:

powers = []
for i in range(10):
def _(a,i=i): return a**i
powers.append(_)

which works and is understandable, but a bit less concise.

This would be a more direct translation I think:

def put_func(i):
def power_of_i(a):
return a**i
return power_of_i
power = [put_func(i) for i in range(10)]

I think it's also clearer what it does. I had to type the lambda
version into the shell to be sure I understood it. I think that's what
we want to avoid.

The main obstacle to the lambda style here is that def statements
are not expressions. I think that's been proposed as an alternative,
too -- make def return a value so you could say:

powers = [def _(a,i=i): return a**i for i in range(10)]

Wouldn't it be:

powers = [(def _(a): return a**i) for i in range(10)]

The parens around the def make it clearer I think.

That would be pretty much just renaming lambda and changing a syntax a
tad. I get the feeling that the continual desire to change the syntax
of lambda is one of the reasons for getting rid of it. I'm not sure any
of the suggestions will fix that. Although I prefer this version over
the current lambda.

Instead of reusing 'def', resurrecting 'let' as a keyword might be an
option.

powers = [ (let a return a**i) for i in range(10) ]


I just now thought this up, but I like it a lot as an alternative syntax
to lambda.

Personally, I think this is understandable, and given that lambda
is to be pulled, a nice substitute (I would say it is easier to read
than the current lambda syntax, and easier for a newbie to
understand).

But it would probably encourage some bad habits, such as:

myfunc = def _(a,b):
print a,b
return a+b

which looks too much like Javascript, to me, where there are
about three different common idioms for defining a
function (IIRC). :-/

I don't think it would be used that way... Very often anyway.

Still none of these examples make for good use cases for keeping lambda.
And I think that's whats needed first, then the new syntax can be decided.

Ron

 

[Dennis Lee Bieber]

I predict, though, that one of the first 3rd party modules to come out
for Python 3000 will be such a library.

Should be a breeze, considering the work that had to go into
making a ComeFrom module...

--

 

[Donn Cave]

Quoth Tom Anderson <[email protected]>:
....
| I disagree strongly with Guido's proposals, and i am not an ex-Lisp,
| -Scheme or -any-other-functional-language programmer; my only other real
| language is Java. I wonder if i'm an outlier.
|
| So, if you're a pythonista who loves map and lambda, and disagrees with
| Guido, what's your background? Functional or not?

Dysfunctional, I reckon.

I think I disagree with the question more than the answer.

First, map and lambda are two different things, and it's reasonable
to approve of one and abhor the other. Especially if you have a
background in a functional language where lambda works like it should.
On the other hand, the list comprehension gimmick that replaces some
of the "higher order functions" is borrowed from Haskell, as you probably
know, so it isn't exactly alien to functional programming. Prelude.hs
defines map: map f xs = [ f x | x <- xs ]

Secondly, if there's anything I detest about the Python development
model, it is the tendency to focus on gimmicks. For 2.X, elimination
of these features would be an atrocity, a gratuitous change that would
break programs - but I don't think anyone who counts has seriously
proposed to do that. With 3.X, we are talking about a different language.
May not ever even get off the ground, but if it does, it's supposed to
be distinctly different, and we need to know a lot more about it before
we can reasonably worry about trivial details like whether map is going
to be there.

I personally think real FP is seriously hot stuff, but I think Python
is a lousy way to do it, with or without map. I suppose there's a
remote possibility that 3.X will change all that. Or more likely,
there will by then be a really attractive FP language, maybe out
of the "links" initiative by Wadler et al.

Donn Cave, (e-mail address removed)

 

[John Roth]

map and filter are being removed *because of* list comprehensions. Did you
even read Guido's articles about this issue? Your understanding of why
these changes are planned is incorrect; consequently your projection based
on that understanding is not on firm footing.

May I most respectfully point out that you've got it backwards.
Part of the justification for list comprehensions was that they could
be used to replace map and filter.

The jihad against the functional constructs has been going on for a
long time, and list comprehensions are only one piece of it.

John Roth

 

[Tom Anderson]

Personally, I find that Lisp & its derivatives put your head in a very
weird place. Even weirder than PostScript/Forth/RPN, when you come
right down to it.

+1 QOTW!

tom

 

[Robert Kern]

May I most respectfully point out that you've got it backwards.
Part of the justification for list comprehensions was that they could
be used to replace map and filter.

That is essentially what I said. List comprehensions were made to
replace map and filter. Now that they are here, Guido wants to remove
map and filter to keep OOWTDI.

My response was incomplete, not incorrect.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Jamey Cribbs]

So, if you're a pythonista who loves map and lambda, and disagrees with
Guido, what's your background? Functional or not?

I have no functional language background. Until recently, I had no use
for programming "expression to be evaluated later" or "deferred
expressions" or whatever else they are being called.

Where I came to see the awesomeness of "deferred expressions" was a few
months ago when I started a major rewrite of KirbyBase for Ruby. I
wanted to make the Ruby version of KirbyBase take advantage of the
strengths of the language. Another Ruby programmer, Hal Fulton, was
helping me by constantly pushing me to make KirbyBase more Ruby-ish.
One thing he kept pushing was to be able to specify select querys using
Ruby's "deferred expression" mechanism, code blocks (before anyone
starts yelling, I know that Ruby code blocks are *much* more than just
"deferred expressions"; I'm just using that descriptor here for the sake
of this discussion).

Code blocks allow you to wrap up any Ruby code and pass it to a method
and have it executed within that method. It is more powerful than
lambda, because you can have multiple statements in the code block and
you can do assignment within the code block. This allowed me to rewrite
KirbyBase so that you can do a select like this:

plane_tbl.select { |r| r.country == 'USA' and r.speed > 350 }

Now, this is cool, but you can do this using lambda in Python. Where
Ruby code blocks really shine is that you can also do this:

plane_tbl.update {|r| r.name == 'P-51'}.set {|r|
r.speed = 405
r.range = 1210
}

I have one code block that I pass to the update method which says
"Select all planes with name equal to P-51". Then, I pass a code block
to the set method which assigns new values to the speed and range fields
for those records (i.e. P-51) that were selected in the update method.
This is something you can't do with lambda.

Now, I think I can duplicate the same functionality of Ruby code blocks
by using Python functions, but it is not going to be as pretty.

So, even though lambda is not as powerful as Ruby code blocks, I was
still bummed to read that it is going away, because it is better than
nothing.

Hopefully, Guido will reconsider or, even better, give us something even
more powerful.

Jamey Cribbs

 

[Mike Meyer]

Code blocks allow you to wrap up any Ruby code and pass it to a method
and have it executed within that method. It is more powerful than
lambda, because you can have multiple statements in the code block and
you can do assignment within the code block.

Just FYI, the inability to have statements - even multiple statements
- in a lambda is what people are talking about when they talk about
the limitations of lambda. It's a common thing for someone with a
background that includes a proper lambda to trip over when they first
start programming in Python. It's not that uncommon for newcommers to
trip over it with "print".

<mike

 

[Steven D'Aprano]

Lambda serves a very specific purpose: declaring small, in-place
functions which are no bigger than a single expression. I do this often
enough that I DO want special syntax for it. But I'll admit that I
wish "lambda" were about 5 or 6 characters shorter

As in an empty string?

and didn't have such an obscure name.

Lambda is no more an obscure name than "function", "decorator", "closure",
"class", or "module". The first time you come across it, you don't know
what it means. Then you learn what it means, and then you know.

 

[Steven D'Aprano]

No, he wants Python to be Pythonic. TMTOWTDI is not Pythonic.

Too Many T--- Only Way To Do It?

There Might Tangle One Way To Do It?

T--- M--- Two Obvious Ways To Do It?

Nope, sorry, still not getting it.

If you'd rather spend your time figuring out which of multiple ways to
do things is the best for the job at hand than producing code, there's
a language that makes TMTOWTDI a way of life.

Figuring out which of multiple ways to do things is the best for the job
at hand _is_ part of producing code. There will always be multiple ways to
do the job. For starters, there is the choice, which language should I
use? Top-Down or Bottom-Up design? Test-driven or not? For loop or list
comprehension or generator? Procedural programming or object-oriented or a
mixture of both? Singleton or Borg design pattern? Test your data first or
deal with the exceptions when they happen? And so on.

Only One Obvious Way makes a nice slogan, but it is easy to turn a
flexible language like Python into a straight-jacket where there is Only
One Way To Do It Regards Of Whether It Is The Best For The Job On Hand Or
Not. Not such a short and concise slogan.

Now that Python has list comps, should for loops be removed from the
language? Why did Python bother introducing list comps when there is
nothing they can do that a for loop can't?

Functional programming using map etc does require a slightly different way
of thinking about programming than does procedural programming, just as
object-oriented needs a different way of thinking than spaghetti-coding
using GOTOs. Different ways of thinking about programming should be
encouraged, not discouraged. Even the much-maligned GOTO has its modern
usage case: the Exception.

If map/filter/reduce have to be removed from the built-ins, and I don't
think they should, I'd prefer for them to be moved into a module rather
than dropped altogether. Provided list comps are made as fast as map and
filter, then at the cost of readability they can be replaced by list
comps. But reduce can't be written as a list comp, only as a relatively
complex for loop at a HUGE loss of readability -- and I've never used
Lisp or Scheme in my life. I'm surely not the only one.

 

[Steven D'Aprano]

Slippery slope arguments are logical fallacies, you know.

Not if you are actually standing on a slippery slope. But seriously, no,
they aren't. The slippery slope argument is _not_ "X is happening now, so
Y will happen no matter what we do". That would be a fallacy.

The argument is actually "X is happening now. If X continues to happen
into the future, Y is the logical consequence of that process. If we wish
to avoid Y, we must stop X". And that is not a fallacy in general
(although of course it could be, if there is no causal relationship
between X and Y).

In this particular case, I suspect Sean is wrong. Guido seems to like list
comprehensions. Unless I'm mistaken (not for the first time) I think he
actually introduced them to the language. They won't be going anywhere
anytime soon.

 

[Devan L]

Claiming that sum etc. do the same job is the whimper of
someone who doesn't want to openly disagree with Guido.

Could you give an example where sum cannot do the job(besides the
previously mentioned product situation?

Also, map is easily replaced.
map(f1, sequence) == [f1(element) for element in sequence]

 

[Peter Hansen]

Too Many T--- Only Way To Do It?

There Might Tangle One Way To Do It?

T--- M--- Two Obvious Ways To Do It?

Nope, sorry, still not getting it.

If you were serious, Google would be a real good friend here, since the
answer is in its first search result... without even having to click on
the link! Heck, it even points you to the web site: http://tmtowtdi.com

-Peter

 

[Steven D'Aprano]

Claiming that sum etc. do the same job is the whimper of
someone who doesn't want to openly disagree with Guido.

Could you give an example where sum cannot do the job(besides the
previously mentioned product situation?

There is an infinite number of potential lambdas, and therefore an
infinite number of uses for reduce.



sum only handles a single case, lambda x,y: x+y

product adds a second case: lambda x,y: x*y

So sum and product together cover precisely 2/infinity, or zero percent,
of all possible uses of reduce.

Also, map is easily replaced.
map(f1, sequence) == [f1(element) for element in sequence]

Three mental tokens ( map, f1, sequence ) versus seven ( [], f1, element,
for, element, in, sequence ).

Also, map can take any number of sequences:

map(f1, seq1, seq2, seq3, seq4, ...)

 

[Christopher Subich]

comps. But reduce can't be written as a list comp, only as a relatively
complex for loop at a HUGE loss of readability -- and I've never used
Lisp or Scheme in my life. I'm surely not the only one.

See my reply to your other post for a more detailed explanation, but I
don't think that the for-loop solution is much less readable at all, and
the additional complexity involved is simply setting the initial value
and result for the accumulator. The for-loop solution is even more
flexible, because it can include anonymous code blocks and not just
expressions.

One caevat that I just noticed, though -- with the for-solution, you do
need to be careful about whether you're using a generator or list if you
do not set an explicit initial value (and instead use the first value of
'sequence' as the start). The difference is:
_accum = g.next()
for i in g: _accum = stuff(_accum,i)

versus
_accum = g[0]
for i in g[1:]: _accum = stuff(_accum,i)

The difference is because generators don't support subscripts, while
lists don't support .next() iteration. Unless I'm missing something in
the language (entirely possible), this suggests a missing feature for
same-syntax iteration over the two types.

 

[Ron Adam]

There is an infinite number of potential lambdas, and therefore an
infinite number of uses for reduce.



sum only handles a single case, lambda x,y: x+y

product adds a second case: lambda x,y: x*y

So sum and product together cover precisely 2/infinity, or zero percent,
of all possible uses of reduce.

But together, sum and product, probably cover about 90% of situations in
which you would use reduce. Getting a total (sum) from a list probably
covers 80% of the situations reduce would be used on it's own. (I can't
think of any real uses of product at the moment. It's late.)

I'm just estimating, but I think that is the gist of adding those two in
exchange for reduce. Not that they will replace all of reduce use
cases, but that sum and product cover most situations and can be
implemented more efficiently than using reduce or a for loop to do the
same thing. The other situations can easily be done using for loops, so
it's really not much of a loss.

Ron

 

[Erik Max Francis]

But together, sum and product, probably cover about 90% of situations in
which you would use reduce. Getting a total (sum) from a list probably
covers 80% of the situations reduce would be used on it's own. (I can't
think of any real uses of product at the moment. It's late.)

It's not uncommon in mathematics to do repeated products. If you're
familiar with the capital Greek letter sigma notation, if it's replaced
with a capital Greek letter pi, then it's an iterated product, rather
than an iterated sum:

http://mathworld.wolfram.com/Product.html

In general, pretty much _any_ operator can be replaced in this symbol to
indicate a repeated operation. Function composition, set unions and
intersections, logical conjunctions and disjunctions, direct sums and
products, the list goes on and on.

I'm just estimating, but I think that is the gist of adding those two in
exchange for reduce. Not that they will replace all of reduce use
cases, but that sum and product cover most situations and can be
implemented more efficiently than using reduce or a for loop to do the
same thing. The other situations can easily be done using for loops, so
it's really not much of a loss.

I really don't understand this reasoning. You essentially grant the
position that reduce has a purpose, but you still seem to approve
removing it. Let's grant your whole point and say that 90% of the use
cases for reduce are covered by sum and product, and the other 10% are
used by eggheads and are of almost no interest to programmers. But it
still serves a purpose, and a useful one. That it's not of immediate
use to anyone is an argument for moving it into a functional module
(something I would have no serious objection to, though I don't see its
necessity), not for removing it altogether! Why would you remove the
functionality that already exists _and is being used_ just because?
What harm does it do, vs. the benefit of leaving it in?

I'm not myself a huge functional programming guy, but I'm certainly not
in favor of the proposal to remove map, filter, reduce, and lambda. For
map and filter, I can at least see the argument, because they truly are
expressible with list comprehensions (which I do use myself, of course).
lambda I also don't buy, but at least there, yes, you can just define
a local function and use it locally, although I think expressively that
it skirts the line between expressivity and verbosity (if you know what
lambda is, straightforward use of lambda is not at all unclear, in fact
it's quite clear). So at least there's something to that, but I don't
follow it the whole way. But removing reduce is just removing
functionality for no other reason, it seems, than spite.

 

[egbert]

Also, map is easily replaced.
map(f1, sequence) == [f1(element) for element in sequence]

How do you replace
map(f1,sequence1, sequence2)
especially if the sequences are of unequal length ?

I didn't see it mentioned yet as a candidate for limbo,
but the same question goes for:
zip(sequence1,sequence2)