Lisp mentality vs. Python mentality

[Carl Banks]

In answering the recent question by Mark Tarver, I think I finally hit
on why Lisp programmers are the way they are (in particular, why they
are often so hostile to the "There should only be one obvious way to
do it" Zen).

Say you put this task to a Lisp and a Python programmer: Come up with
a good, generic, reusable way to compare two lists. What are their
respective trains of thought?


Lisp programmer:

Well, there is a standard function called mismatch that does it, but I
can't recommend it. First of all, you don't know where that
function's been. Anyone and their mother could have worked on it, did
they have good, sound programming practice in mind when they wrote
it? Of course not. Let's be real here, we have to implement this by
hand.

(defun lists-are-equal (a b)
(or (and (not a) (not b))
(and (= (car a) (car b)) (lists-are-equal (cdr a) (cdr b))))

There, much better than the standard function, and better yet, it's in
the *absolute minimal form possible*. There is no way to express list
comparison in a more reduced form. It's almost erotic how awesome it
is. I'm---whoa, ok, I'm getting a little excited now, settle down.
Well, come to think of it, that's really not that good. First of all
it's a function. I mean, it just sits there and does nothing till you
call it. How boring is that? It can't react to the current
situation. Plus it compares all lists the same way, and that is
really inefficient. Every list compare is a new problem. Different
lists need different comparative strategies. This function simply
won't do. I need a macro that can intelligently compile the right
list compare methodology in. For instance, if we want to compare two
lists that are known at compile time, we don't want to waste time
comparing them at runtime. No, the macro must detect constant
arguments and special case them. Good start. Now, we have to
consider the conditions this comparison is being done under. If the
user is passing the result of a sort to this macro, it's almost
certain that they are trying to see whether the lists have the same
elements. We can do that a lot more efficiently with a countset. So
let's have the macro check to see if the forms passed to it are all
sort calls. Better yet, let's check for my own powerful sort macro.
Hmm. Wait... I think my 4600-line sort macro already checks its
calling context to see if its results are being fed to a list
comparison. I'll have to refactor that together with this macro. Ok,
good, now I am sure other users will eventually want to customize list
comparison for their own use, after all every list comparison is
different and I can't possibly anticipate all of them. A user needs
to be able to adapt to the situation, so it's vitally important to
create a plug-in infrastructure to give them that flexibility. Now,
what about exceptions, there's a millions ways to deal with that...

....and so on until eyelids can no longer stay open....



Python programmer:

a == b. Next question.



Carl Banks, who might be exaggerating

....a little.

 

[Martin v. Löwis]

Ok... Then what's pythonic? Please give a pythonic implementation...

But how about extensibility?

== is extensible. To compare two things for equality, use ==.

This is what Carl Banks referred in his original posting. You just
*don't* implement a function that compares two lists, not even as
an exercise for estetic, optimal, and useful implementations -
because you'll know that it won't be useful, anyway, if you can already
use the builtin == in the first place.

I see that you allow for a different comparison function. I do wonder
what the use case for this is - in what application do you have to
compare two lists for equality, and the item's __eq__ is inappropriate?
What would break if you fix the item's __eq__, instead of writing
your own comparison algorithm?

If I would ever run into such a case, I would inline the loop, rather
than writing a function that receives a callable:

for index, v1 in enumerate(l1):
if not comp(v1, l2[index]):
res = False
break
else:
res = True

Regards,
Martin

 

[Ciprian Dorin, Craciun]

I'd claim that this is still theoretical: what are these strings, and
why do you have lists of them that you want to compare?

Why don't you try to lower-case the strings in the list as you
add them?

Also, are you sure a list is the best data structure for your problem?
Perhaps using a set would be better?

Indeed the example I've given is purely theoretical. But still, I
could find a use case for such a thing: just imagine we are building a
small shell-like application that reads one line (the commands),
splits it by spaces and always expects to have 4 elements and that
each respects a given regular expression, one specific for each index.
In this case I could syntactically check for correctness by doing
this:

compare (regular_expressions, splitted_line, re.match)

Of course I could have just created a big regular expression for
the entire line. But maybe my 4 elements come from variables obtained
from a web-server query, or the regular expressions are not static but
dynamically generated at run-time.

If you want to compare the same lists in many places, this is indeed
bad coding practice. You should try to centralize whatever reasons
you have for comparing the lists into a few methods of the objects
holding the lists.

Regards,
Martin

I like object oriented programming, but most of the times we are
just throwing together code and data even when the data has no
behavior and the code is in fact just one possible processing
algorithm. Like in the case you are mentioning, if I tie the
comparison code to the actual data structure, then I'll never be able
to reuse it... But if I leave the code as a standalone function, and
just use the data (or any data that resembles the original structure)
then maybe I'll be able to reuse it...

Ciprian.

 

[Martin v. Löwis]

Indeed the example I've given is purely theoretical. But still, I

could find a use case for such a thing: just imagine we are building a
small shell-like application that reads one line (the commands),
splits it by spaces and always expects to have 4 elements and that
each respects a given regular expression, one specific for each index.
In this case I could syntactically check for correctness by doing
this:

compare (regular_expressions, splitted_line, re.match)

Of course I could have just created a big regular expression for
the entire line. But maybe my 4 elements come from variables obtained
from a web-server query, or the regular expressions are not static but
dynamically generated at run-time.

Ok, in this case I would write a function:

def validate_commandline(rexes, line):
if len(rexes) != len(line):
raise ValueError("Incorrect number of arguments, expected %d,"
"got %d" % (len(rexes), len(line)))
for i in range(len(line)):
if not re.match(rexes, line):
raise ValueError, "Incorrect argument %d" % i

IOW, in this specific case, I would not only want a true/false result,
but also an indication of the actual error to report to the user.
Your universal compare function would be no good here.

Regards,
Martin

 

[Ciprian Dorin, Craciun]

    Indeed the example I've given is purely theoretical. But still, I
could find a use case for such a thing: just imagine we are building a
small shell-like application that reads one line (the commands),
splits it by spaces and always expects to have 4 elements and that
each respects a given regular expression, one specific for each index.
In this case I could syntactically check for correctness by doing
this:

    compare (regular_expressions, splitted_line, re.match)

    Of course I could have just created a big regular expression for
the entire line. But maybe my 4 elements come from variables obtained
from a web-server query, or the regular expressions are not static but
dynamically generated at run-time.

Ok, in this case I would write a function:

def validate_commandline(rexes, line):
   if len(rexes) != len(line):
       raise ValueError("Incorrect number of arguments, expected %d,"
                        "got %d" % (len(rexes), len(line)))
   for i in range(len(line)):
       if not re.match(rexes, line):
           raise ValueError, "Incorrect argument %d" % i

IOW, in this specific case, I would not only want a true/false result,
but also an indication of the actual error to report to the user.
Your universal compare function would be no good here.

Regards,
Martin

Well in fact I would have written it like:

def validate_commandline(rexes, line) :
if not compare (rexes, line, re.match) :
if len (rexes) != len (line) :
raise ValueError ("mismatch len")
mismatch = find_index (rexes, line, re.match, negate = True)
raise ValueError ("mismatch at %d" % (mismatch))

Assuming, that I would have the function find_index.

Ciprian.

 

[Hrvoje Niksic]

In Lisp I think you'd use (equal (mapcar upcase a) (mapcar upcase
b))

Or simply (equalp a b), since equalp comparisons happen to compare
strings case-insensitively. But that's Common Lisp... overflowing
kitchen sink.

or something like that. In Python, a.upper() == b.upper().

I guess you meant map(a, str.upper) == map(b, str.upper)? a and b are
lists of strings.

 

[Paul Rubin]

I guess you meant map(a, str.upper) == map(b, str.upper)? a and b are
lists of strings.

Oh, sorry. Yes, either

map(str.upper, a) == map(str.upper, b)

or

all(str.upper(x)==str.upper(y) for x,y in zip(a,b))

 

[Carl Banks]

I don't get that impression from Lisp programmers.  I suppose it's
only fair that I disclose (1) I admire Lisp, especially Scheme, (2) I
hardly know Lisp at all, and (3) I don't frequent any Lisp forums/
newsgroups/etc.

No it doesn't really apply to Scheme. (The Scheme programmer would
have stopped after implementing the new function.)

Some people seem to want to hypertarget everything. Common Lisp seems
to attract these people in big numbers because it has powerful
metaprogramming facilities and expressivity on steroids, making
hypertargeting easy. It's like the Ultimate Enabler language.

I do get the impression that Lispers tend to feel Lisp is superior to
all other languages, and I agree that in some ways it is.  I don't
think most Lispers' main objection to Python is about "only one
obvious way" but rather things like the limitations, compromises, and
impurities in the language.

I totally disagree. Scheme might be a pure language with no
compromises and impurities, but Common Lisp is certainly not. The
"One Obvious Way" philosophy isn't their main objection so much as the
most emblematic difference.

 Certainly compared to Scheme, Python
sacrifices a lot of purity for practicality.  (And I guess some fans
of Scheme would argue that Common Lisp does the same!)

Ultimately, Lisp is first and foremost academic (Scheme especially so)
while Python is first and foremost practical.  I think Paul Graham's
essays on Lisp exemplify the Lisp mentality.

I don't agree. I agree that Lisp programmers think that's their
mentality; I doubt many can actually take fullest advantage of Lisp
the way Graham has. I think Paul Graham is a freak of nature whose
brain is hardwired to notice patterns in places different from where
most peoeple see patterns. Graham, for his part, doesn't seem to
appreciate that what he does is beyond hope for average people, and
that sometimes reality requires average people to write programs.


Carl Banks

 

[Carl Banks]

    Ok... Then what's pythonic? Please give a pythonic implementation...
Use the builtin a==b, similar to (equal a b)
    But how about extensibility? [...]

I see that you allow for a different comparison function. I do wonder
what the use case for this is - in what application do you have to
compare two lists for equality, and the item's __eq__ is inappropriate?
What would break if you fix the item's __eq__, instead of writing
your own comparison algorithm?

[...]

    A practical example: I have lists that contain strings, but I want
to compare them in an case-insensitive way... Should I update the
__eq__ method (for str class) and break almost everything?

The practical way to deal with this issue is to write your own
function when you encounter a situation where == doesn't suffice.


Carl Banks

 

[Brett Hoerner]

    Well in fact I would have written it like:

def validate_commandline(rexes, line) :
    if not compare (rexes, line, re.match) :
        if len (rexes) != len (line) :
            raise ValueError ("mismatch len")
        mismatch = find_index (rexes, line, re.match, negate = True)
        raise ValueError ("mismatch at %d" % (mismatch))

    Assuming, that I would have the function find_index.

    Ciprian.

I think you've hit on the definition of "unpythonic". (No, I don't
have a dictionary definition for you, sorry).

Using a function called "compare" to run a list of regexes against
another list of regexes to get a boolean? And then another find_index
function doing the same where you pass in negate? What is even going
on here?

I, for one, would take Martin's any day of the week. It reads like
good pseudocode as much "proper" Python does.

Brett

 

[Ciprian Dorin, Craciun]

I think you've hit on the definition of "unpythonic".  (No, I don't
have a dictionary definition for you, sorry).

Using a function called "compare" to run a list of regexes against
another list of regexes to get a boolean?  And then another find_index
function doing the same where you pass in negate?  What is even going
on here?

I, for one, would take Martin's any day of the week.  It reads like
good pseudocode as much "proper" Python does.

Brett

From your comments I understand that the only problem with my code
proposal are the function names... Well instead of compare (which was
kept from the beginning of the post) we could just rename it to
"matches". Does the name "matches" matches what it does? (If not we
can keep discussing for a proper name...)

And about the find_index, we could rename it to
first_matching_index. About the negation optional parameter, we could
eliminate it if we allow either: to have another function
first_missmatching_index, but this leads to namespace bloat, or we
have a function named negate, that takes another function, and negates
it meaning. (Although i don't see anything wrong in the negate
argument... It's just like having order by asc | desc in SQL...)

Thus the code would have been rewritten as: (we also put the
function on the first argument as its the most important argument)

def validate_commandline(rexes, line) :
if not matches (re.match, rexes, line) :
if len (rexes) != len (line) :
raise ValueError ("mismatch len")
mismatch = first_matching_index (negate (re.match), rexes, line)
raise ValueError ("mismatch at %d" % (mismatch))

Ciprian.

 

[Mark Wooding]

Graham, for his part, doesn't seem to appreciate that what he does is
beyond hope for average people, and that sometimes reality requires
average people to write programs.

I think he understands that perfectly well. But I think he believes
that the sorts of tools which help average people write programs get in
the way of true wizards.

I think I agree.

On the other hand, I don't think Python actually does get in the way
very much.

-- [mdw], Lisp hacker.

 

[Carl Banks]

Graham, for his part, doesn't seem to appreciate that what he does is
beyond hope for average people, and that sometimes reality requires
average people to write programs.

I think he understands that perfectly well.  But I think he believes
that the sorts of tools which help average people write programs get in
the way of true wizards.

I think I agree.  

On the other hand, I don't think Python actually does get in the way
very much.

-- [mdw], Lisp hacker.

 

[John Yeung]

I think he understands that perfectly well.  But I think he
believes that the sorts of tools which help average people
write programs get in the way of true wizards.

I think I agree.  

On the other hand, I don't think Python actually does get
in the way very much.

Actually, Graham doesn't have particularly strong objection to
Python. Partly this is because he sees it as being largely as capable
and expressive as Lisp (mainly sans macros, of course); partly because
he sees that Python tends to attract good programmers (chief among
them Trevor Blackwell).

In my view, what is remarkable about Python is that it is so
accessible to average programmers (and frankly, even rather poor
programmers) while still managing to stay appealing to top-notch
programmers.

That said, my experience with Lisp programmers has mainly been with
people who like Scheme, which may explain why Carl Banks and I have
different impressions of Lisp programmers. (We also seem to differ on
how accurate it is to refer to Scheme as Lisp.) But in my experience,
Lisp in any form tends not to attract average programmers, and
certainly not poor programmers. I don't mean to say Banks is wrong; I
said up front my exposure to the Lisp community is limited. I am just
giving my own impressions.

Python is easily powerful enough and expressive enough to be an
"enabler language". I guess not Ultimate, but close enough that
Graham isn't particularly turned off by it!

John

 

[Steven D'Aprano]

Lisp programmer:

Well, there is a standard function called mismatch that does it, but I
can't recommend it. First of all, you don't know where that function's
been. Anyone and their mother could have worked on it, did they have
good, sound programming practice in mind when they wrote it? Of course
not. Let's be real here, we have to implement this by hand.

[snip]


That's great stuff! An unfair and untrue caricature, but still great

 

[Michele Simionato]

I totally disagree.  Scheme might be a pure language with no
compromises and impurities, but Common Lisp is certainly not.

I can assure you that even Scheme is a language full
of compromises and inconsistencies :-/

Michele, who is now writing a book about Scheme

http://www.phyast.pitt.edu/~micheles/scheme/TheAdventuresofaPythonistainSchemeland.pdf

 

[Steven D'Aprano]

Ciprian Dorin, Craciun:

Python way:
---------
def eq (a, b) :
    return a == b

def compare (a, b, comp = eq) :
    if len (a) != len (b) :
        return False
    for i in xrange (len (a)) :
        if not comp (a, b) :
            return False
    return True

That's not "pythonic".

Bye,
bearophile

Ok... Then what's pythonic? Please give a pythonic implementation...

Don't re-invent the wheel. Instead of creating your own functions, use
existing tools to your advantage.

import operator

def compare(a, b, comp=operator.eq):
    if len(a) != len(b):
        return False
    for a, b in zip(a, b):
        if not comp(a, b):
            return False
    return True


But we can re-write that to be even more pythonic, by using the built-in
all():

def compare(a, b, comp=operator.eq):
    if len(a) != len(b):
        return False
return all(comp(x, y) for (x, y) in zip(a, b))

or even:

def compare(a, b, comp=operator.eq):
    return (len(a) == len(b)) and all(comp(*t) for t in zip(a, b))


(All the above are untested, so please excuse any errors.)

Ciprian Craciun.

P.S.: Also, I'm tired of hearing about the pythonic way... Where
do I find a definitive description about the pythonic way?

There is no such thing as a definitive description of pythonic -- it is
like art, and pornography: you can recognise it when you see it (except
when you can't).

However, you can get close by doing:

import this

in the Python interactive interpreter. Or from a shell prompt:

python -m this

I think that
this word is used only when someone sees something that he doesn't like,
he doesn't know what he doesn't like at it, and just goes to say its
un-pythonic, without saying what would be... Wouldn't be just easier to
say "I don't know" or "I doesn't feel right to me"?

I think that there's a risk that people over-use unpythonic when they
mean "I don't like it", but that doesn't mean that pythonic isn't a
meaningful concept. However, it is a matter of degree, not kind: like
mole-hills and mountains, unpythonic and pythonic are very different
things, but there's no precise dividing line between them.

 

[Steven D'Aprano]

== is extensible. To compare two things for equality, use ==.

This is what Carl Banks referred in his original posting. You just
*don't* implement a function that compares two lists, not even as an
exercise for estetic, optimal, and useful implementations - because
you'll know that it won't be useful, anyway, if you can already use the
builtin == in the first place.

I see that you allow for a different comparison function. I do wonder
what the use case for this is - in what application do you have to
compare two lists for equality, and the item's __eq__ is inappropriate?

The above doesn't really compare for equality, it's a generic element-by-
element comparison function, and so it is inappropriate to contrast it to
__eq__ alone. Defaulting to equality testing is misleading, and if I were
writing such a function I'd remove the default.

compare(a, b, operator.eq) gives the same result as the simpler a == b,
but compare(a, b, operator.lt) does something very different to a < b. I
can't think of an application for element-by-element comparisons off the
top of my head, but if the numpy people use it, there must be a need

 

[anonymous.c.lisper]

In answering the recent question by Mark Tarver, I think I finally hit
on why Lisp programmers are the way they are (in particular, why they
are often so hostile to the "There should only be one obvious way to
do it" Zen).

Say you put this task to a Lisp and a Python programmer: Come up with
a good, generic, reusable way to compare two lists.  What are their
respective trains of thought?

Lisp programmer:

Well, there is a standard function called mismatch that does it, but I
can't recommend it.  First of all, you don't know where that
function's been.  Anyone and their mother could have worked on it, did
they have good, sound programming practice in mind when they wrote
it?  Of course not.  Let's be real here, we have to implement this by
hand.

(defun lists-are-equal (a b)
   (or (and (not a) (not b))
       (and (= (car a) (car b)) (lists-are-equal (cdr a) (cdr b))))

There, much better than the standard function, and better yet, it's in
the *absolute minimal form possible*.  There is no way to express list
comparison in a more reduced form.  It's almost erotic how awesome it
is.  I'm---whoa, ok, I'm getting a little excited now, settle down.
Well, come to think of it, that's really not that good.  First of all
it's a function.  I mean, it just sits there and does nothing till you
call it.  How boring is that?  It can't react to the current
situation.  Plus it compares all lists the same way, and that is
really inefficient.  Every list compare is a new problem.  Different
lists need different comparative strategies.  This function simply
won't do.  I need a macro that can intelligently compile the right
list compare methodology in.  For instance, if we want to compare two
lists that are known at compile time, we don't want to waste time
comparing them at runtime.  No, the macro must detect constant
arguments and special case them.  Good start.  Now, we have to
consider the conditions this comparison is being done under.  If the
user is passing the result of a sort to this macro, it's almost
certain that they are trying to see whether the lists have the same
elements.  We can do that a lot more efficiently with a countset.  So
let's have the macro check to see if the forms passed to it are all
sort calls.  Better yet, let's check for my own powerful sort macro.
Hmm.  Wait... I think my 4600-line sort macro already checks its
calling context to see if its results are being fed to a list
comparison.  I'll have to refactor that together with this macro.  Ok,
good, now I am sure other users will eventually want to customize list
comparison for their own use, after all every list comparison is
different and I can't possibly anticipate all of them.  A user needs
to be able to adapt to the situation, so it's vitally important to
create a plug-in infrastructure to give them that flexibility.  Now,
what about exceptions, there's a millions ways to deal with that...

...and so on until eyelids can no longer stay open....

Python programmer:

a == b.  Next question.

Carl Banks, who might be exaggerating

...a little.

(equal a b) or (equalp a b)

Next question??

I understand where you are going with the analogy, but I think a lot
of what you describe as 'over-thinking' of the problem in lisp comes
from people having an honest desire to answer the /right/ question.

Lisp gives you a bit of granularity with regard to certain things
(like comparison), that in many languages amount to a single '=='
operator. I /believe/ that this is because of its origins in symbolic
programming.

You don't compare numeric functions in Perl and C, and then say 'Oh
those silly C programmers over-thinking things, they must have 10
different types of numbers!'

 

[Ciprian Dorin, Craciun]

Ciprian Dorin, Craciun:
Python way:
---------
def eq (a, b) :
    return a == b

def compare (a, b, comp = eq) :
    if len (a) != len (b) :
        return False
    for i in xrange (len (a)) :
        if not comp (a, b) :
            return False
    return True

That's not "pythonic".

Bye,
bearophile

    Ok... Then what's pythonic? Please give a pythonic implementation...

Don't re-invent the wheel. Instead of creating your own functions, use
existing tools to your advantage.

import operator

def compare(a, b, comp=operator.eq):
    if len(a) != len(b):
        return False
    for a, b in zip(a, b):
        if not comp(a, b):
            return False
    return True


But we can re-write that to be even more pythonic, by using the built-in
all():

def compare(a, b, comp=operator.eq):
    if len(a) != len(b):
        return False
   return all(comp(x, y) for (x, y) in zip(a, b))

or even:

def compare(a, b, comp=operator.eq):
    return (len(a) == len(b)) and all(comp(*t) for t in zip(a, b))


(All the above are untested, so please excuse any errors.)

    Ciprian Craciun.

    P.S.: Also, I'm tired of hearing about the pythonic way... Where
do I find a definitive description about the pythonic way?

There is no such thing as a definitive description of pythonic -- it is
like art, and pornography: you can recognise it when you see it (except
when you can't).

However, you can get close by doing:

import this

in the Python interactive interpreter. Or from a shell prompt:

python -m this

I think that
this word is used only when someone sees something that he doesn't like,
he doesn't know what he doesn't like at it, and just goes to say its
un-pythonic, without saying what would be... Wouldn't be just easier to
say "I don't know" or "I doesn't feel right to me"?

I think that there's a risk that people over-use unpythonic when they
mean "I don't like it", but that doesn't mean that pythonic isn't a
meaningful concept. However, it is a matter of degree, not kind: like
mole-hills and mountains, unpythonic and pythonic are very different
things, but there's no precise dividing line between them.

I liked very much your implementation for the compare function, it
is very short and at the same time readable:

def compare(a, b, comp=operator.eq):
return (len(a) == len(b)) and all(comp(*t) for t in zip(a, b))

But I have only one problem, it is suboptimal, in the sense that:
* it constructs two intermediary lists (the list comprehension and
the zip call);
* it evaluates all the elements, even if one is false at the beginning;

So, could you overcome these problems?

About the pythonic vs unpythonic words, I agree with you, there
are ofter overused and misused...

Ciprian.