Python syntax in Lisp and Scheme

[Joe Marshall]

Actually I've noticed that I usually cut myself when I *switch* from
a dull knife to a sharp one.

Grasp the sharp knife by the *handle* when switching.

 

[Raffael Cavallaro]

:> I can't see why a LISP programmer would even want to write a macro.
: That's because you are approaching this with a fundamentally flawed
: assumption. Macros are mainly not used to make the syntax prettier
: (though they can be used for that). They are mainly used to add features
: to the language that cannot be added as functions.

Really? Turing-completeness and all that... I presume you mean "cannot
so easily be added as functions", but even that would surprise me.
(Unless you mean cannot be added _to_Lisp_ as functions, because I don't
know as much as I'd like to about Lisp's capabilities and limitations.)

Two words: code duplication.

Yes, anything that can be done with macros can also be done with
functions, but if you do it with functions, you will end up with more
code, and that code will be duplicated in every single source location
in which that abstraction it utilized.

With a macro, the abstraction is defined once, and the source code
reflects that abstraction everywhere that abstraction is used
throughout your program. For large projects this could be hundreds of
source locations.

Without a macro, you have multiple points of maintenance. If your
abstraction changes, you have to edit scores or hundreds of source
locations. With a macro, you redefine a single form, in one source
location, and recompile the dependent code.

All turing complete languages are computationally equivalent. That
doesn't mean you'll see me programming by punching holes in a paper
tape any time soon though.

Finally, there is one thing that macros can do that ordinary functions
cannot do easily - change the language's rules for functional
evaluation. This can only be accomplished with functions if you're
willing to write a set of functions that defer evaluation, by, say
parsing input, massaging it appropriately, and then passing it to the
compiler. At that point, however, you've just written your own macro
system, and invoked Greenspun's 10th Law.

 

[Thomas F. Burdick]

Actually I've noticed that I usually cut myself when I *switch* from
a dull knife to a sharp one.

I don't think the truism about cutting yourself with dull vs sharp
knives means to say anything about superficial cuts. You don't lop
your finger off with a sharp knife, because you're handling it
carefully. With a dull knife, your best bet is to put your weight
behind it; that's also a good way to lose a finger / dump core.

--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'

 

[Paul Rubin]

I would say that one of the reasons why I program in Scheme is
*because* I value simplicity and uniformity. The way that Python has
been described in this discussion make me think that I would really
*hate* Python for it's unecessary complications if I went back to it.

I programmed in Lisp for a long time before starting to use Python. I
can't see even a die-hard Lisper really hating Python. Python has a
bunch of unnecessary little quirks that a Lisper would find annoying,
but also has a lot going for it, and the shortcomings are mostly
pretty minor.

 

[Jock Cooper]

You left out the with-collector part.

But it's true that my examples are less convincing given the existence of
yield (which I had forgotten about). But the point is that in pre-yield
Python you were stuck until the langauge designers got around to adding
it.

I'll try to come up with a more convincing short example if I find some
free time today.

I'm afraid it's very hard to give any convincing examples of the
utility of macros -- as long as you are sticking to trivial examples.
On the other hand, you can't exactly post real world complex examples
of how macros saved you time and LOC (we all have em) because reader's
eyes would just glaze over. I think macros are just another one of
CL's features that some most people just don't get until they actually
use them. But here's a small one:

I wrote about 60 lines worth of macro based code (including a few reader
macros) that allows me to write things like:

(with-dbconnection
(sql-loop-in-rows
"select col1, col2 from somewhere where something"
:my-package row-var "pfx"
...
...some code...
...))

In the "some code" section, the result columns' values are accessed by
#!pfx-colname (eg #!pfx-col1), or directly from row-var using
#?pfx-colname (which returns the position). Also, error handling code
can be automatically included by the macro code. How much time and
effort (and possible bugs) has this saved me? Well at least 60+ lines
or more of boilerplate every time I use this pattern.. Plus the expansions
for #!colname include error checks/warnings etc. -- all hidden from view.

Jock

 

[Hans Nowak]

[...] Fortunately, in either case, the rich
and stong suite of unit tests that surely accompanies such a
"mission-critical application" will easily detect the nefarious
deed and thus easily defang the nanoviruses' (nanovirii's? nanovirorum...?)
menace, [...]

Hmm, if I recall correctly, in Latin the plural of 'virus' is 'virus'. (I like
to write 'virii' myself, knowing that it's wrong, but it looks cute.

 

[Daniel P. M. Silva]

[...] But the point is that in pre-yield
Python you were stuck until the langauge designers got around to adding
it.

I'll try to come up with a more convincing short example if I find some
free time today.

Haven't some people implemented an entire class system as one huge macro?

- Daniel

 

[Bengt Richter]

IMHO, these discussions are less usefull when not accompanied by
specific examples. What are these macros good for? Some examples
where you might have difficulties with using ordinary functions:

1.) Inventing new control structures (implement lazy data structures,

You mean like an object olazy with olazy.x and olazy.y, where x might
be an integer 123 and y might be the text of the latest bug python report
on sourceforge? Not so hard with properties (you'd proabably want to cache
the latest for y and not re-check for some reasonable interval, but that
doesn't change client code (unless you later decide you want the value to
be a tuple of (timestamp, text), etc.)

implement declarative control structures, etc.)

You mean like case or such? It can be done, but Python could use a way
to define a nested functions that just use the local namespace of their lexically
enclosing function. Then case would be easy to build with a dictionary-based dispatch
of such functions.

Wild idea: It might be interesting to be able to do a kind of (quote thing) or 'thing
in Python, but to defer "trailer" evaluation (where trailer is what comes after an
object-specifying expression. E.g., in x.y[z+2](a=3) x has the trailer ".y[z+2](a=3)"
and x.y has the trailer [z+2](a=3), etc. Ok, you could currently write a class to
do deferred trailer evaluation in the form qt=QT(x.y, '[z+2](a=3)') and then
evaluate/unquote it with qt.value ot qt.evaluate(), etc., but you couldn't just
bind and pass a QT instance like an ordinary object and expect an automatic evaluation
on access, as if the quoted expression appeared in that context.

BTW, with trailer, I am just generalizing the first impulse, which was just to do it
for an attribute name, so as to be able to wrap property as well as ordinary attribute
accesses for unqualified (i.e., deferred-qualification -- we're not talking about
rebinding a name (e.g., x above) that is unknown at access time, since it was lost
when the leader object was pased to QT).

There is an ambiguity when assigning to a name that's bound to a QT object though,
i.e., whether to rebind the name as usual without considering its current binding,
or whether to notice the binding to a QT object, and do as-if-the-entire-trailer-
expression-were-there target processing for the assignment. The latter would make
for "interesting" programming ;-) To rebind such a name, you'd have to get at it
via an indirect route, much as you have to do with an object's attribute name if
it is bound to a property.

=> This one is rarely needed in everyday application programming and
can easily be misused.

2.) Serve as abbreviation of repeating code. Ever used a code
generator? Discovered there was a bug in the generated code? Had
to fix it at a zillion places?
=> Macros serve as extremely flexible code generators, and there
is only one place to fix a bug.
=> Many Design Patterns can be implemented as macros, allowing you
to have them explicitly in your code. This makes for better
documentation and maintainability.

You can generate code many ways in Python. What use case are you thinking of?

3.) Invent pleasant syntax in limited domains.
=> Some people don't like Lips' prefix syntax. It's changeable if you
have macros.
=> This feature can also be misused.

You can do this also.

4.) Do computations at compile time instead of at runtime.
=> Have heard about template metaprogramming in the C++ world?
People do a lot to get fast performance by shifting computation
to compile time. Macros do this effortlessly.

This also, but Python has so many possible compile times ;-)

These are four specific examples which are not easy to do without
macros. In all cases, implementing them classically will lead to code
duplication with all the known maintainability issues. In some cases
misuse will lead to unreadable or buggy code. Thus, macros are
powerful tools for the hand of professionals. You have to know if you
want a sharp knife (which may hurt you when misused) or a less sharper
one (where it takes more effort to cut with).

Python is pretty sharp ;-)
I think we need some realistic use cases for your "specific" [categories of]
examples in order to compare how problems would be approached.

Regards,
Bengt Richter

 

[Andrew Dalke]

Gerrit Holl:

a = 2
a = 1 + 1
a = math.sqrt(4)
a = int((sys.maxint+1) ** (1/31))

...all mean the same thing.

Be careful with that. The first two return the integer 4, the third returns
the floating point number 4.0 and the last returns 1 because 1/31 is
0 (unless you are using true division). Even if you use 1/31. you'll
get a different value on a 32 bit machine vs. a 64 bit machine.

pedantically yours,

Andrew
(e-mail address removed)

 

[Michael Geary]

[...] Fortunately, in either case, the rich
and stong suite of unit tests that surely accompanies such a
"mission-critical application" will easily detect the nefarious
deed and thus easily defang the nanoviruses' (nanovirii's?
nanovirorum...?) menace, [...]

Hmm, if I recall correctly, in Latin the plural of 'virus' is 'virus'.
(I like to write 'virii' myself, knowing that it's wrong, but it
looks cute.

Ah, but we're not speaking Latin here.

In English, the plural of "virus" is "viruses":

http://dictionary.reference.com/search?q=virus

-Mike

 

[Bengt Richter]

<my-first-name.my-last-name-0610030955090001@k-137-79-50-101.jpl.nasa.go
v>,

The net effect is a filter, but again, you need to stop thinking about the
"what" and start thinking about the "how", otherwise, as I said, there's
no reason to use anything other than machine language.

Answer 1: literal translation into Python. The closest analogue of
with-collector etc would be Python's simple generators (yield keyword)
and do-with-file-lines is expressed in python with a for loop. So:

def lines_with_some_property(some_file_name):
for l in some_file_name:
if some_property(l):
yield l

Your only use of macros in this example is to handle the with-collector
syntax, which is handled in a clean macro-free way by Python's "yield".
So this is unconvincing as a demonstration of why macros are a necessary
part of a good programming language.

Of course, with-collector could be embedded in a larger piece of code,
while using yield forces lines_with_some_property to be a separate
function, but modularity is good...

Answer 2: poetic translation into Python. If I think about "how" I want
to express this sort of filtering, I end up with something much less
like the imperative-style code above and much more like:

[l for l in some_file_name if some_property(l)]

Just thought of this: a generator list comprehension, so your could write

[yield l for l in some_file_name if some_property(l)]

and get a generator that is the equivalent of your lines_with_some_property above.

I.e., in general you should be able to convert any list comprehension into a generator by
putting 'yield ' at the front. Has someone proposed this already? I seems a natural, unless
I am blindly optimistic, which is quite possible ;-)

I have no problem with the assertion that macros are an important part
of Lisp, but you seem to be arguing more generally that the lack of
macros makes other languages like Python inferior because the literal
translations of certain macro-based code are impossible or more
cumbersome. For the present example, even that argument fails, but more
generally you'll have to also convince me that even a freer poetic
translation doesn't work.

Regards,
Bengt Richter

 

[Mark Carter]

pedantically yours,

Andrew
(e-mail address removed)

Shouldn't that be "yours pedantically"?

 

[Pascal Costanza]

I don't know a lot about Python, so here is a question. Is something
along the following lines possible in Python?

(with-collectors (collect-pos collect-neg)
(do-file-lines (l some-file-name)
(if (some-property l)
(collect-pos l)
(collect-neg l))))


I actually needed something like this in some of my code...

Not using simple generators afaik. The easiest way would probably be to
append into two lists:

collect_pos = []
collect_neg = []
for l in some_file_name:
if some_property(l):
collect_pos.append(l)
else:
collect_neg.append(l)

....but this means that

collect = []
for l in some_file_name
if some_property:
collect.append(l)

....is another solution for the single collector case. Now we have two
ways to do it. Isn't this supposed to be a bad sign in the context of
Python? I am confused...

If you needed to do this a lot of times you could encapsulate it into a
function of some sort:

def posneg(filter,iter):
results = ([],[])
for x in iter:
results[not filter(x)].append(x)
return results

collect_pos,collect_neg = posneg(some_property, some_file_name)

What about dealing with an arbitrary number of filters?

(defmacro predicate-collect (list &body predicates)
(let ((collectors (mapcar (lambda (predicate)
(declare (ignore predicate))
(gensym "COLLECT"))
predicates))
(collect-t (gensym "COLLECT")))
`(with-collectors (,@collectors ,collect-t)
(dolist (l ,list)
(cond ,@(mapcar (lambda (predicate collector)
`((funcall ,predicate l) (,collector l)))
predicates collectors)
(t (,collect-t l)))))))

An example:

> (predicate-collect '(-5 -4 -3 -2 -1 0 1 2 3 4 5)

(function evenp)
(lambda (n) (< n 0))
(lambda (n) (> n 3)))
(-4 -2 0 2 4)
(-5 -3 -1)
(5)
(1 3)


I use the list collector macros by Tim Bradshaw here - see
http://www.tfeb.org/lisp/hax.html#COLLECTING


Pascal

 

[Matthias]

You mean like an object olazy with olazy.x and olazy.y, where x might
be an integer 123 and y might be the text of the latest bug python report
on sourceforge? Not so hard with properties (you'd proabably want to cache
the latest for y and not re-check for some reasonable interval, but that
doesn't change client code (unless you later decide you want the value to
be a tuple of (timestamp, text), etc.)

Actually, I meant more lazy-like-lazy-in-Haskell. Infinite data
structures and such. "primes" being a list representing _all_ prime
numbers for instance. You can build this as soon as you have closures
but making the construction easy to use for the application programmer
might be a challenge without macros. But I don't know what
"properties" are in Python, possibly they are built for exactly that.

You mean like case or such?

No. I was thinking about Prolog and such. Or nondeterministic
programming. Or multimethod dispatch.

You can generate code many ways in Python. What use case are you thinking of?

I was not talking about generating code /in/ Python but generating
code /for/ Python /within/ it. For the Design Pattern use case take a
look at http://norvig.com/design-patterns/

You can do this also.

You can change Python's syntax? Easily?

This also, but Python has so many possible compile times ;-)

I'm not sure I understand.

Python is pretty sharp ;-)
I think we need some realistic use cases for your "specific" [categories of]
examples in order to compare how problems would be approached.

Well, if you don't want to learn about syntactic abstraction you'll
probably never miss it during your programming. Just keep in mind
that before oo-abstraction became fashionable people didn't miss OOP
either.

 

[Marco Antoniotti]

<my-first-name.my-last-name-0610030955090001@k-137-79-50-101.jpl.nasa.go
v>,



Answer 1: literal translation into Python. The closest analogue of
with-collector etc would be Python's simple generators (yield keyword)
and do-with-file-lines is expressed in python with a for loop. So:

def lines_with_some_property(some_file_name):
for l in some_file_name:
if some_property(l):
yield l

Your only use of macros in this example is to handle the with-collector
syntax, which is handled in a clean macro-free way by Python's "yield".
So this is unconvincing as a demonstration of why macros are a necessary
part of a good programming language.

Of course, with-collector could be embedded in a larger piece of code,
while using yield forces lines_with_some_property to be a separate
function, but modularity is good...

BZZZZZT! In CL the WITH-COLLECTOR macro is best defined as

(defmacro with-collector (collector-name &body forms) ...)

That is to say that you can write things as

(with-collector collect
(let ...
(block ...
(loop ...
(dotimes ... (collect whatever))))))

This means that your Python function is *NOT* equivalent to the CL
macro. Not even with your 'yield' (easily implementable in CL: meaning,
in CL).

Answer 2: poetic translation into Python. If I think about "how" I want
to express this sort of filtering, I end up with something much less
like the imperative-style code above and much more like:

[l for l in some_file_name if some_property(l)]

Very interesting. The CL equivalent is

[l in (lines-of some-file-name) if (some-property l)]

Now: I do have an implementation of this in CL. (I assume that your
'some_file_name' is not just a string).

I have no problem with the assertion that macros are an important part
of Lisp, but you seem to be arguing more generally that the lack of
macros makes other languages like Python inferior because the literal
translations of certain macro-based code are impossible or more
cumbersome. For the present example, even that argument fails, but more
generally you'll have to also convince me that even a freer poetic
translation doesn't work.

Well, the argument does not fail even in these cases as closer
inspection reveals. There are things you cannot do in Python that you
can easily do in CL. It is the converse that is only asymptotically
provable as a rhethoric exercise

Greenspun's Tenth Rules!

Cheers

 

[Sean Ross]

Just thought of this: a generator list comprehension, so your could write [snip]
Has someone proposed this already? I seems a natural, unless
I am blindly optimistic, which is quite possible ;-)

[snip]

Hi.
Yes, they've been proposed (by Raymond Hettinger, PEP289) and rejected
(http://www.python.org/peps/pep-0289.html).

Sean

 

[Marco Antoniotti]

I don't know a lot about Python, so here is a question. Is something
along the following lines possible in Python?

(with-collectors (collect-pos collect-neg)
(do-file-lines (l some-file-name)
(if (some-property l)
(collect-pos l)
(collect-neg l))))


I actually needed something like this in some of my code...

Not using simple generators afaik. The easiest way would probably be to
append into two lists:

collect_pos = []
collect_neg = []
for l in some_file_name:

^^^^^^^^^^^^^^

Please be precise. You are missing the definition of 'some_file_name'.
The CL macro can hide the fact that you are passing the iterator
either a string or a structured pathname or an iterator instance. In
Python you have to include the definition of 'some_file_name'.

In CL the following will work (assuming that the DO-FILE-LINES CL macro
is written correctly)

(let ((some-file-name "/tmp/foo.txt"))
(with-collectors (collect-pos collect-neg)
(do-file-lines (l some-file-name)
(if (some-property l)
(collect-pos l)
(collect-neg l)))))

In Pyhton

some_file_name = '/tmp/foo.txt'
collect_pos = []
collect_pos = []
for l in some_file_name:

will happily bind 'l' to the characters in '/tmp/foo.txt'.


Cheers

 

[Hannu Kankaanp??]

...but this means that

collect = []
for l in some_file_name
if some_property:
collect.append(l)

...is another solution for the single collector case. Now we have two
ways to do it. Isn't this supposed to be a bad sign in the context of
Python? I am confused...

It's all about what's the "correct" way to do it. In this case, it
would be

collect = [l for l in some_file_name if some_property]

And this would be expanded to the for-loop form *only* if it is needed.
Of course I can do

x = 5
x += y

Or I can do

x = 5 + y

That's not breaking against Python's principles. The latter
way is the right way.

(surely there are many debatable cases of which is the right
way, but the principle is adhered to as often as possible.
That's the best one can ever have)

 

[Bengt Richter]

I don't know a lot about Python, so here is a question. Is something
along the following lines possible in Python?

(with-collectors (collect-pos collect-neg)
(do-file-lines (l some-file-name)
(if (some-property l)
(collect-pos l)
(collect-neg l))))


I actually needed something like this in some of my code...

Not using simple generators afaik. The easiest way would probably be to
append into two lists:

collect_pos = []
collect_neg = []
for l in some_file_name:
if some_property(l):
collect_pos.append(l)
else:
collect_neg.append(l)

If you needed to do this a lot of times you could encapsulate it into a
function of some sort:

def posneg(filter,iter):
results = ([],[])
for x in iter:
results[not filter(x)].append(x)
return results
Um, about choosing names ...

collect_pos,collect_neg = posneg(some_property, some_file_name)

Nah, just abuse a list comprehension:

>>> seq = range(10)
>>> test = lambda x: x%3==0
>>> pos,neg = [(p,n) for p,n in[map(list,['']*2)]

... for q in [(x,test(x),i) for i,x in enumerate(seq)]
... if q[1] and p.append(q[0]) or not q[1] and n.append(q[0]) or not q[2]][0]

>>> pos [0, 3, 6, 9]
>>> neg

[1, 2, 4, 5, 7, 8]

;-)

Regards,
Bengt Richter

 

[Terry Reedy]

What about dealing with an arbitrary number of filters?

[macro snipped]

What about it? Using macros for somewhat simple functions stikes me
as overkill.

An example:

(function evenp)
(lambda (n) (< n 0))
(lambda (n) (> n 3)))
(-4 -2 0 2 4)
(-5 -3 -1)
(5)
(1 3)

In Python:

def multisplit(seq, *preds):
predn = len(preds)
bins = [[] for i in range(predn+1)]
predpends = [(p,b.append) for (p,b) in zip(preds,bins)]
rpend = bins[predn].append
for item in seq:
for pred,pend in predpends:
if pred(item):
pend(item)
break
else: rpend(item)
return bins

multisplit(range(-5,6), lambda i: not i%2, lambda i: i<0, lambda i:
i>3)

[[-4, -2, 0, 2, 4], [-5, -3, -1], [5], [1, 3]]

Terry J. Reedy