Explanation of macros; Haskell macros

[Joachim Durchholz]

Hmmm. If it will make you feel any better, macros are just fuctions
whose domain and range happens to be Lisp expressions. That happen to
be run by the compiler. So eventually the compiler is evaluating
constant expressions, just some of them were automatically derived
from the written source.

Hmm... you're right here.
The HOF approach has one advantage over the DEFMACRO approach: code
written using HOFs will automagically adapt if some of the constant
inputs become variable, or vice versa. For DEFMACRO, if a constant
becomes input, the macro will become inapplicable and the source code
will have to change; for HOFs, the compiler will be able to
automatically adapt.

Well it depends whether you consider syntax to be "anything". I think
it was you who objected to one of my examples by saying, "that's
just syntactic sugar". Macros can (and many do) do large amount of
under-the-covers bookkeeping. For instance here are a few rules from
a grammar for a lexer for Java source code:

(defprod line-terminator () (/ #\newline (#\return (? #\newline))))

(defprod white-space () (/ #\space #\tab #\page line-terminator))

(defprod input () ((* input-element) (? #\Sub)))

(defprod input-element () (/ white-space comment token))

(defprod token () (/ identifier java-keyword literal separator
operator))

DEFPROD is a macro that squirrels away the stuff on the right which
is an s-expy form of BNF. The rest of the grammar is more of the
same. At the bottom of the grammar file where the productions are
diffined I have this form:

(deflexer java-lexer (input) (tokens identifier java-keyword
literal separator operator)))

That DEFLEXER call (another macro) expands into a single parsing
function built out of all the productions created by DEFPROD calls,
appropriately wired together and embedded into code that takes care
of the stepping through the input and gather up values, etc. And that
function is compiled into extremely efficient code because all the
intercommunication between productions goes through lexical
variables. And the file containing these calls to DEFPROD and
DEFLEXER is legal Lisp source which I can feed to the compiler and
get native machine code back.

So I don't know if that is "anything" or not.

It most definitely is "something"

I don't know how I would write such a thing in Haskell, et al. but I
know this is a *lot* cleaner than what *I'd* be able to do in Java,
Perl, Python, or C.

I'm looking at things from a Haskell perspective.

Actually, functional languages do similar things; it's called
"combinator parsing".
The basic approach is this: you have parsing functions that each
recognize a particular language, trivial parsers that each recognize
just one element of the alphabet, and parser combinators that take one,
two, or more subparsers and combine them into a bigger one (for
constructing alternatives, options, repetitions and whatever your
personal flavor of BNF can do).

I don't know enough about any approach to do an in-detail comparison,
but the rough picture seems to be pretty similar.

Downside of combinator parsing: it's difficult to get bottom-up parsers
done that way. Also, simple-minded combinator parsers tend to do
backtracking, though it's not very difficult to make the combinators
diagnose and report violations of LL(whatever) properties.

Ah - I see one other thing that HOFs cannot do: issue compile-time error
messages.
Unless, of course, there is a data type that, when evaluated at compile
time, causes the compiler to emit an error message... not /that/
difficult to do, but might require some careful thought to make the
mechanism interact well with other properties of the language.

Actually, changing the syntax is--if one thinks one must--is really
done by read-macros which are quite different. But most Lispers agree
with you--there's just not enough benefit to changing the syntax to
be worth it.
OK.

Except for occasionally making a new syntax for expressing certain
frequently created literal objects that otherwise would require a
much more verbose creation form. (Someone gave a great example the
other day in another thread of an airline reservation system (Orbitz
I think) that has a special kind of object used to represent the
three-letter airport codes. Since they wanted to always have the same
object representing a given airport they needed to intern the objets
with the TLA as the key. But rather than writing (intern-airport-code
"BOS") everywhere, they wrote a reader macro that let them write:
#!BOS. Since this was an incredibly common operation in their system,
it was worth a tiny bit of new syntax. But note, again, that's not
*changing* the syntax so much as extending it.)

In my book, "extending" isn't so much different than "changing".
I agree it's the kind of worthwhile change that makes sense.
In Haskell, one would probably have an "Airport" module that defined
these codes, and write something like
TLA "BOS"
which is more syntax than #!BOS but seems good enough for me. (YMMV.)

Fair enough. But do you object to the ability to write new functions
on the grounds that that just means you have a lot of new functions
to learn and that complicates things needlessly? That's obviously a
rhetorical question but I am actually curious why you find them
different, if you do.

It's just the KISS principle: why two abstraction facilities (macros and
functions) if one suffices?
Provided that functions suffice, actually

The funny thing is to me, when you say "two-tier thinking" that
perfectly describes how I think about the process of making
abstractions. Regardless of the *kind* of abstraction one is
creating, one has to be facile at switching mental gears between
*building* the abstraction and *using* it. You are probably so used
to doing this when writing functions that you don't even notice the
switch.

Hmm... not consciously, but there is certainly a difference.
It seems to be smaller with functional languages, particularly if you're
working at higher levels.
In an FPL with proper syntactical minimalism, programming enters a
"we're sticking functions together" style, which partially abstracts
away the parameters (at least as entities you're conscious about). I.e.
the "using abstractions" thinking mode diminishes. (I'm not far enough
into that style to say how this works after that style was fully adopted.)

But because macros are a bit strange you *notice* the switching and
it annoys you. I suspect that anyone who's capable of building
functional abstractions would--if they actually used macros--quickly
learn to switch gears equally smoothly when writing and using macros.

Possibly... can't tell.

Regards,
Jo

 

[Rene de Visser]

Does anybody have a keyword-style list of useful applications of the
macro facilities?

I don't write macros that often, I mainly use HOF's.

I always think of a macro as a function that takes some code and returns
some code, and takes place at compile time.

I think that macro's can be categorized as follows:

1) Create new bindings (variable for function bindings).

One of the simplest macros that I use is mvlet* (for multiple value
bindings). This lets you do things like

(mvlet ((
(c (func1 d)
values a b) (func2 e))
...
which binds a b to the values returned by func2. Due to an accident of fate
multiple values aren't so well integrated into common lisp,
so macros can be used to remove deficiencies from langauge.

2) Generating/changing code from meta data.

i)

(with-extracting-expressions (a b c)
.... )

where a b c are functions that can be assumed to be side-effect free. Based
on this expressions should be bubbled up to the outside
to improve performance.

ii) Removing almost repeated code.

Often I notice that in a program there is code that is similar but not
identical. Then a macro can be written that based on meta data generates the
various different versions of this code.

3) New languages or semantics which expand to lisp code.

e.g. relation lisp, screamer (non-deterministic lisp), parsers, series,
CLOS.

As a side note, I note that in the GHC source code that the parser is
generated Haskell code. Why is not the parser directly specified in Haskell?

In common lisp you would specify the parser directly in LISP and macros
would expand/transform it to its final runable form.

Rene.

 

[prunesquallor]

Including macros invocations?

I don't think MIT Scheme tracks macro expansions.

However, PLT Scheme does. The compiler and macroexpander are
orthogonal, so in principle one could simply merge the two methods.

When you say "source code", do you really mean source code, or do
you mean the results of macro-expansion?

In PLT Scheme I really mean the source code.

If I have a function foo whose body invokes a macro bar which calls a
function baz, can I get a "stack trace" or equivalent which shows me the
line in the definition of bar which invokes baz, and the line in the
definition of foo which invokes bar? Can I see the bindings of the
parameters of bar?

(define (baz x)
(+ x 3))

(define-syntax bar
(syntax-rules ()
((bar x) (begin0 (baz x) #f))))

(define (foo y)
(begin (bar y) #f))

(foo 'a)

+: expects type <number> as 1st argument, given: a; other arguments were: 3
c:\home\jrm\test.ss:3:2: (+ x 3)
c:\home\jrm\test.ss:10:9: (begin0 (baz y) #f)
STDIN::105: (foo (quote a))

 

[Peter Seibel]

Pascal Costanza wrote:

Inserting "at arbitrary places" seems rather unmodular to me.
Either the macro would have to know about the code that it's being
used in, so that it can identify the places where to insert the code.
Or the code using the macro will have to pass information to the macro
-
which, in a HOF context, means that the code using the HOF approach
will have to pass the code for "before the insertion point" and "after
the insertion point" as separate parameters, and the HOF would do the
necessary linkage between the two codes.
I'm not sure that macros offer a significant gain here.

So here's one sort of trivial example of something that demonstrates
one aspect of what Pascall was talking about. I'm not sure how you do
this with HOFs since they don't give you any ability to get at the
code at compile time.

(defmacro show (form)
"Trace the evaluation of a form and return its value.
(Could be made smarter about multiple values.)"
(let ((form-value (gensym)))
`(let ((,form-value ,form))
(format *trace-output* "TRACE>> Evaluating ~s; got ~a~%" ',form ,form-value)
,form-value)))

CL-USER: (show (+ 1 2))
TRACE>> Evaluating (+ 1 2); got 3
3
CL-USER: (defun foo (x) (+ (show (+ 1 x)) (+ 3 4) (show (* 5 (* x 4)))))
FOO
CL-USER: (foo 10)
TRACE>> Evaluating (+ 1 X); got 11
TRACE>> Evaluating (* 5 (* X 4)); got 200
218

Now that's pretty trivial. But now we can use SHOW to build something
a bit more complex:

(defmacro show-calls-to ((&rest names) &body body)
`(progn
,@(loop for form in body
when (member (first form) names) collect `(show ,form)
else collect form)))

This macro takes a lis of function names to SHOW and a body of forms.
All the top level forms in the body that contain calls to the named
functions are wrapped in a call to SHOW. A more sophisticated, and
useful, version of this macro would use code walker to find
interesting calls other than at the top level.

CL-USER: (show-calls-to () (- 3 4) (+ 4 5) (* 6 7))
42
CL-USER: (show-calls-to (+) (- 3 4) (+ 4 5) (* 6 7))
TRACE>> Evaluating (+ 4 5); got 9
42
CL-USER: (show-calls-to (+ *) (- 3 4) (+ 4 5) (* 6 7))
TRACE>> Evaluating (+ 4 5); got 9
TRACE>> Evaluating (* 6 7); got 42
42

This is not necessarily a typical use of macros, but it does perhaps
demonstrate how a macro can manipulate the code in its arguments
without any particular "cooperation" from the code being manipulated.

-Peter

 

[Pascal Costanza]

Inserting "at arbitrary places" seems rather unmodular to me.
Either the macro would have to know about the code that it's being used
in, so that it can identify the places where to insert the code.
Or the code using the macro will have to pass information to the macro -
which, in a HOF context, means that the code using the HOF approach will
have to pass the code for "before the insertion point" and "after the
insertion point" as separate parameters, and the HOF would do the
necessary linkage between the two codes.
I'm not sure that macros offer a significant gain here.

a) "insert instructions at arbitrary places in the code being passed to
a macro" means that the macro modifies the code that is being passed to
it - not the code that surrounds the use site of a macro.

b) For macros, you don't have to partition the information the macro
needs to do its job into several parameters. A macro just analyzes the
code that is being passed to it, and automagically does the right thing.
(This is part of the reason why macros allow you to build
abstractions that don't leak: you don't have to arrange the parameters
for the macro according to the implementation details of the macro, but
in more or less completely arbitrary ways.)

c) The significant gain of macros is this: Macros rewrite syntax trees,
and you can implement any language feature, domain-specific or not, via
syntax tree rewriting. This means that, no matter what happens during
software development, you can always find a way to express things in a
convenient way.

This is mainly a psychological effect: macros greatly reduce the need to
plan ahead and allow for experimenting with program designs: Just write
down the code as you imagine to be suitable for the concrete problem at
hand, and then start to implement it from there. It's very likely that
you will find out how to implement the machinery beneath in that process
and then write a bunch of macros that translate your original ideas on
top of that machinery. You don't have to plan ahead what the "right"
design is for you problem.

Switch from FP to OOP later on? No problem: rewrite the machinery,
reimplement your macros, don't change the high-level code. Oops, your
field should have been methods, or externally stored in a database? No
problem: rewrite the machinery, reimplement your macros, don't change
the high-level code.

Without macros, it's very likely that in these cases, when you have to
change the design of your libraries, you also have to change client
code. With macros, you don't have to. At least not in 99% of the cases. ;-P

OK, but the same can be said for HOFs.
The question is: in what ways are macros superior to HOFs to accomplish
this?

Most importantly: HOFs as such are rarely domain-specific. So whenever
your library requires you to express a piece of code with an anonymous
function, you give away details about the implementation of a specific
feature.

See it this way: Macros are a way to create blocks of code. HOFs that
return functions do the same: they take a few functions (and possibly
some value arguments), stick them together, and return the result. If
the HOF is evaluated at compile time (something that I'd expect if both
functions and values submitted to the HOF are constant), then you too
have a mechanism that creates a block of code.

I don't care about blocks of code or compile-time evaluation, at least
not most of the time. When I see this...


(with-open-file (stream filename
:direction utput
:if-does-not-exist :create)
(format stream "some output"))


....I see a piece of code that tells the machine to open a file in a
specified way, write some output to it and then closes it gracefully. I
don't care whether this is achieved by means of OOP abstraction, HOFs,
or whatever. Such code "talks" only about the necessary details and
nothing else.

I think you forgot a smiley - at least, that question is entirely silly.
If you have a real point to make, please make it explicit; I have too
little time to devote to decoding your intents. (And if you don't want
to spend time on trivialities, then please understand that I don't, too.)

Your question is as silly as mine.

If you want to get an idea what you can do with macros, Paul Graham
provides an excellent overview in his book "On Lisp" that you can
download for free at http://www.paulgraham.com/onlisp.html


Pascal

 

[Fergus Henderson]

(define (baz x)
(+ x 3))

(define-syntax bar
(syntax-rules ()
((bar x) (begin0 (baz x) #f))))

(define (foo y)
(begin (bar y) #f))


+: expects type <number> as 1st argument, given: a; other arguments were: 3
c:\home\jrm\test.ss:3:2: (+ x 3)
c:\home\jrm\test.ss:10:9: (begin0 (baz y) #f)
STDIN::105: (foo (quote a))

That's still missing an entry from the stack trace.
There's no entry for the call to bar.

 

[Paul Dietz]

Does anybody have a keyword-style list of useful applications of the
macro facilities?

Here's an example of a utility that is easy to implement with lisp
macros, but rather more difficult to do without them:

http://www-2.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/lisp/code/testing/cover/0.html
(described in http://www.merl.com/papers/TR91-04/ )

Paul

 

[Stephen J. Bevan]

Now that's pretty trivial. But now we can use SHOW to build something
a bit more complex:

(defmacro show-calls-to ((&rest names) &body body)
`(progn
,@(loop for form in body
when (member (first form) names) collect `(show ,form)
else collect form)))

This macro takes a lis of function names to SHOW and a body of forms.
All the top level forms in the body that contain calls to the named
functions are wrapped in a call to SHOW. A more sophisticated, and
useful, version of this macro would use code walker to find
interesting calls other than at the top level.

CL-USER: (show-calls-to () (- 3 4) (+ 4 5) (* 6 7))
42
CL-USER: (show-calls-to (+) (- 3 4) (+ 4 5) (* 6 7))
TRACE>> Evaluating (+ 4 5); got 9
42
CL-USER: (show-calls-to (+ *) (- 3 4) (+ 4 5) (* 6 7))
TRACE>> Evaluating (+ 4 5); got 9
TRACE>> Evaluating (* 6 7); got 42
42

Is the above something you find you use with any regularity or would
expect others to use? I ask because unless other people look at it
and go "Wow, that's something I really want and I can now see how to
get it via macros", then macro examples like the above aren't going to
win anyone over. For my part, show-calls-to is not something I would
ever envisage using. I can envisage, and occasionally do use, the
trace functionality found in some implementations that allows the
inputs and output(s) of a specified function to be displayed but that
is rather a different beast (though in some Lisp implementations it
might well be partially implemented using a macro).

 

[Peter Seibel]

Is the above something you find you use with any regularity or would
expect others to use? I ask because unless other people look at it
and go "Wow, that's something I really want and I can now see how to
get it via macros", then macro examples like the above aren't going
to win anyone over.

No, as it stands I don't think show-calls-to all that useful. It was
just a quick hack to show how a macro can analyze the its arguments
and generate new code with some new bit of functionality interleaved.
into the code it was passed without that code having to necessarily be
aware of the macro. I don't even think that's a very typical use of
macros but someone was questioning how such a thing could even be
done.

I'm about ready to give up on trying to evangelize macros on
usenet--if you show a simple macro to demonstrate a point people say:
"That's silly, who'd want to do that?" If you show a macro that allows
some simple code to expand into some complex code people say, "I don't
understand that code it expands into, show me a simpler example."

Hopefully I'll finish my chapters about macros soon and I can just
point people to them. (Which will, of course, happen sooner if I stop
spending so much time on Usenet.)

For my part, show-calls-to is not something I would ever envisage
using. I can envisage, and occasionally do use, the trace
functionality found in some implementations that allows the inputs
and output(s) of a specified function to be displayed but that is
rather a different beast (though in some Lisp implementations it
might well be partially implemented using a macro).

All of them, probably, since the standard (if you're talking about
Common Lisp) specifies that it's a macro. ;-)

-Peter

 

[Coby Beck]

(e-mail address removed) (Stephen J. Bevan) writes:
I'm about ready to give up on trying to evangelize macros on
usenet--if you show a simple macro to demonstrate a point people say:
"That's silly, who'd want to do that?" If you show a macro that allows
some simple code to expand into some complex code people say, "I don't
understand that code it expands into, show me a simpler example."

Yes, it is always a moving target...

I posted this way way upthread but it only made it to one of the
cross-posted groups so I'll try again:

Here's a nice example from some production code I wrote that is easy to
grok.

The purpose: a socket server excepts a set of specific commands from
clients. Client commands must be verified as allowed and the arguments
marshalled before applying the appropriate function to the arguments. I
wanted a clean way to express this and automate the writing of error
catching code.

Usage: define-server-cmd name (method-parameters) constraints code

Sample usage:
(define-server-cmd set-field-sequence ((session morph-configuration-session)
field-list)
((listp field-list)
(remove-if-not #'(lambda (key) (member key *logical-types*))
field-list))
(with-slots (client source-blueprint state) session
(setf (field-sequence (source-blueprint session)) field-list)
(setf state :blueprint-set)
(send session (write-to-string state))))

The resulting expansion:
(PROGN
(DEFMETHOD SET-FIELD-SEQUENCE
((SESSION MORPH-CONFIGURATION-SESSION) FIELD-LIST)
(WITH-SLOTS (CLIENT SOURCE-BLUEPRINT STATE)
SESSION
(SETF (FIELD-SEQUENCE (SOURCE-BLUEPRINT SESSION))
FIELD-LIST)
(SETF STATE :BLUEPRINT-SET)
(SEND SESSION (WRITE-TO-STRING STATE))))
(DEFMETHOD MARSHAL-ARGS-FOR-CMD
((CMD (EQL 'SET-FIELD-SEQUENCE))
(SESSION MORPH-CONFIGURATION-SESSION))
(LET (FIELD-LIST)
(PROGN
(SETF FIELD-LIST
(RECEIVE SESSION :TIMEOUT *COMMAND-PARAMETER-TIMEOUT*))
(UNLESS FIELD-LIST
(ERROR 'TIMEOUT-ERROR
:EXPECTATION
(FORMAT NIL "~A parameter to ~A command" 'FIELD-LIST CMD)
:TIMEOUT
*COMMAND-PARAMETER-TIMEOUT*)))
(UNLESS (LISTP FIELD-LIST)
(ERROR 'COMMAND-CONSTRAINT-VIOLATION
:CONSTRAINT
'(LISTP FIELD-LIST)
:COMMAND
CMD))
(UNLESS (REMOVE-IF-NOT #'(LAMBDA (KEY)
(MEMBER KEY *LOGICAL-TYPES*))
FIELD-LIST)
(ERROR 'COMMAND-CONSTRAINT-VIOLATION
:CONSTRAINT
'(REMOVE-IF-NOT #'(LAMBDA (KEY)
(MEMBER KEY *LOGICAL-TYPES*))
FIELD-LIST)
:COMMAND
CMD))
(LIST FIELD-LIST)))
(PUSHNEW 'SET-FIELD-SEQUENCE *CONFIG-SERVER-COMMANDS*))

Usage of what the macro gave me in context (some error handling noise
removed):

(defmethod run-config-command-loop ((session morph-configuration-session))
(let ((*package* (find-package :udt)))
(unwind-protect
(with-slots (client) session
(loop
(let (cmd)
(setf cmd (receive session :timeout *command-timeout*
:eof-value :eof))
(cond
((or (eq cmd :eof) (eq cmd :stop)) (return))
((member cmd *config-server-commands*)
(let ((cmd-args (marshal-args-for-cmd cmd session)))
(apply cmd session cmd-args)))
(t (execute-generic-command cmd client)))))

(send session "session loop terminated"))
(when (eq (state session) :finalized)
(setf *active-sessions* (delete session *active-sessions*))))))

The macro definition:

(defmacro define-server-cmd (name (session-specializer &rest args)
constraints &body body)
(let ((session-var (car session-specializer))
(session-class (cadr session-specializer)))
`(progn
(defmethod ,name ((,session-var ,session-class)
,@(mapcar #'(lambda (arg)
(if (symbolp arg) arg (car arg)))
args))
,@body)
(defmethod marshal-args-for-cmd
((cmd (eql ',name)) ,session-specializer)
(let (,@args)
,@(loop for var in args
collect
`(progn
(setf ,var (receive ,session-var
:timeout *command-parameter-timeout*))
(unless ,var
(error 'timeout-error
:expectation (format nil "~A parameter to ~A command"
',var cmd)
:timeout *command-parameter-timeout*))))
,@(loop for con in constraints
collect
`(unless ,con
(error 'command-constraint-violation
:constraint ',con
:command cmd)))
(list ,@args)))
(pushnew ',name *config-server-commands*))))

I think the advantages are tremendously obvious, and very satisfying to make
use of!

 

[Espen Vestre]

The purpose: a socket server excepts a set of specific commands from
clients. Client commands must be verified as allowed and the arguments
marshalled before applying the appropriate function to the arguments. I
wanted a clean way to express this and automate the writing of error
catching code.

Usage: define-server-cmd name (method-parameters) constraints code

AOL! I have almost the same thing myself, except that my macro does
less, it just registers the code in a hash table and lets other code
do the rest of the work.

 

[Stephen J. Bevan]

Here's a nice example from some production code I wrote that is easy to
grok.

I follow it to a certain extent but the problem now is that we are
given a solution with together with an explanation of what it does,
but we don't know all the details of the problem that shaped the
solution. Therefore while it is possible to agree that the macro
version obviously hides a lot of detail, it isn't entirely clear if
that is the best way to hide it or even if that detail is required.

I'm not asking you to give the full requirements for your program,
only explaining why if it isn't clear what the problem is then nobody
can easily tell if the macro-based solution is an improvement over
anything else. For example, if the commands have any kind of grammar
then solution is to define the grammar and let a parser generator take
care of parsing and building an AST. Whether the parser generator is
an extension of the language (as is possible in Lisp and Forth) or a
separate language (most other languages) is to a certain extent an
implementation detail. At the other extreme is a simple flat file
that is processed by an AWK program to generate source in the target
language. To someone used to using macros, the AWK approach looks
like a really poor substitute for a subset of what macros can do and
it is from that perspective. However, the AWK route is also an
approach that can solve a lot of simple problems, probably even the
one you solved using macros. Note I'm not going to argue that it any
solution using AWK is *better* than using a macro based solution, only
that to sell macros to someone using the AWK approach the pitch has to
convince them that the macro approach is significantly better in
order to be worth investing the time and effort to learn. What
constitues "significant" obviously varies.

 

[Kenny Tilton]

....

anything else. For example, if the commands have any kind of grammar
then solution is to define the grammar and let a parser generator take
care of parsing and building an AST. Whether the parser generator is
an extension of the language (as is possible in Lisp and Forth) or a
separate language (most other languages) is to a certain extent an
implementation detail.

No, this misses the whole point. The idea precisely is not to have to
jump out of the IDE, move to external tool X that has none of the
knowledge available to code within the application, define a grammar
which must now be kept in synch with new decisions made back in the HLL
IDE (that is always /so/ much fun) to generate transformed code which
must be regenerated in its entirety anytime the grammar changes.

That is like saying I do not need OO because I can have a type slot and
all my code can dispatch off that, so there is my OO.

The world gets better when something you can do the hard way (and using
AWK instead of macros really takes the cake on that score) gets
supported by a polished language mechanism.

To a large degree, all these Turing-equivalent non-repsonses are doing a
good job of explaining why macros are useful: who needs a jet to get
from NYC to Beijing? C'mon, Amtrak, kayak, Great Wall...good morning,
Beijing!

kenny

--
http://tilton-technology.com

Why Lisp? http://alu.cliki.net/RtL Highlight Film

Your Project Here! http://alu.cliki.net/Industry Application

 

[Stephen J. Bevan]

No, this misses the whole point. The idea precisely is not to have to
jump out of the IDE, move to external tool X that has none of the
knowledge available to code within the application, define a grammar
which must now be kept in synch with new decisions made back in the
HLL IDE (that is always /so/ much fun) to generate transformed code
which must be regenerated in its entirety anytime the grammar changes.

As I noted in the section you snipped, an AWK-like solution looks
pretty poor to someone used to using Lisp style macros. However, lots
of people take the AWK-like approach because it works for them with
whatever language they are currently using. Simply telling them it is
a poor way to work is not going to win them over to macros. Showing
them examples of how macros simplfy a given problem in Common Lisp are
useful but can still fall flat if they can't identify with that
problem. What they want to see is either a Common Lisp (with macros)
solution to their problem or an example of a known problem with a
solution recast in Common Lisp utilising macros. That obviously
pushes a lot of effort on those who want to make the case for macros.

That is like saying I do not need OO because I can have a type slot
and all my code can dispatch off that, so there is my OO.

I don't think it is like saying that at all, but rather than digress
let's just try and avoid similies or analogies altogether.

The world gets better when something you can do the hard way (and
using AWK instead of macros really takes the cake on that score) gets
supported by a polished language mechanism.

To a large degree, all these Turing-equivalent non-repsonses are doing
a good job of explaining why macros are useful: who needs a jet to get
from NYC to Beijing? C'mon, Amtrak, kayak, Great Wall...good morning,
Beijing!

Telling people that they are producing "Turing-equivalent
non-responses" isn't going to win them over either.

 

[Kenny Tilton]

> However, lots
> of people take the AWK-like approach because it works for them with
> whatever language they are currently using.

But the question is "why macros?" Answer, so you do not have to use Awk.

Now you are in some fantasy-land where someone addresses a domain by
creating a domain-specific langugae under Awk, then generates all their
Java, Perl, Python, Ruby, PHP, and STP code from that. Cool! Is that in
Matrix 4?

That was not an analogy, btw. I was already using the C switch statement
to dispatch off an explicit "type" slot in C structs to achieve (what?)
polymorphism when I heard about Smalltalk. I did not think "I already
have that, however crappy and tedious to deal with when I come up with a
new type". I thought, "Brilliant. Somebody thought to build that into a
language."

But then I am not afraid to learn new languages.

kenny

--
http://tilton-technology.com

Why Lisp? http://alu.cliki.net/RtL Highlight Film

Your Project Here! http://alu.cliki.net/Industry Application

 

[Bruce Lewis]

I'm not asking you to give the full requirements for your program,
only explaining why if it isn't clear what the problem is then nobody
can easily tell if the macro-based solution is an improvement over
anything else.

I have to get back to class soon, but here's a quick rundown of two
macros in BRL (http://brl.sourceforge.net/) and why I think they're
nicer than HOFs for the same purpose:

In ASP/VBScript you might do something like

a=request("a")
b=request("b")
c=request("c")

A BRL macro makes it simply:

(define-input a b c)

Half as many opportunities for typos, as you can both use the name of
the variable and bind the variable.

BRL also has a sql-repeat macro. Within it is some syntax, e.g.
(group-ending? expr)

Here, expr is any expression where the free variables are bound to data
columns returned from an SQL query. It returns true if you're at the
end of the results, or if expr will be different for the next row. This
kind of predicate turns eager evaluation on its ear, but lazy evaluation
won't give it to you either. You can do it with HOFs, but not so
concisely. As this is a simple, intuitive mental concept, the
conciseness is nice.

That's all for now. Class is starting.

 

[Coby Beck]

I follow it to a certain extent but the problem now is that we are
given a solution with together with an explanation of what it does,
but we don't know all the details of the problem that shaped the
solution. Therefore while it is possible to agree that the macro
version obviously hides a lot of detail, it isn't entirely clear if
that is the best way to hide it or even if that detail is required.

The problem was a fairly ordinary one and is easy to describe at a high
level. The server existed to control a complicated configuration process
and later process data according to the result. It was a simple socket
server and I got to dictate the form of the commands and arguments.
Ultimately, it boiled down to (funcall command client-session args).

So I needed (wanted) a way to, in one stroke,
- define the method, properly specialized on the session object
- ensure it became an allowed function to call
- define a way to gather the arguments needed from the client
according to number of args and the type of each.
- provide a facility for arbitrarily complex validation of any
of the arguments or combinations thereof.
- ensure that any changes, enhancements or additions to
argument passing would require a single point of change in my code.

Additional benefits:
- automatically provided information to a kind of "help" facility.
- allowed for very informative error messages, both in the debug
environment and as returned data for clients.
- simplified the main server loop while remaining completely flexible

I'm not asking you to give the full requirements for your program,
only explaining why if it isn't clear what the problem is then nobody
can easily tell if the macro-based solution is an improvement over
anything else.

Hopefully the above is enough of a spec. I would be very happy to learn
other approaches, there are always many ways to skin a cat. (how un-PC is
that saying these days

For example, if the commands have any kind of grammar
then solution is to define the grammar and let a parser generator take
care of parsing and building an AST.

I was fortunate to be the one defining the API so I came up with a very
clever way of having named and optional arguments that could be atoms or
lists defined by ( and ) characters. (I should patent that

I did have to define a set of safe-read functions for lists and integers and
strings etc for a combination of security and timing-out reasons. But the
grammar was just s-expressions. (It was culture shock for the Java client
coders, and I think they still would say SOAP was better though it is beyond
me why).

Whether the parser generator is
an extension of the language (as is possible in Lisp and Forth) or a
separate language (most other languages) is to a certain extent an
implementation detail. At the other extreme is a simple flat file
that is processed by an AWK program to generate source in the target
language. To someone used to using macros, the AWK approach looks
like a really poor substitute for a subset of what macros can do and
it is from that perspective.

If the target language is Lisp, then not "looks like", but "is". I think
you know that, though. I actually have macros I use now that generate
strings of SQL code, some of it as format strings that will be used at a
later stage of processing (a "format string" would be something like "SELECT
NEW.~A FROM ~A WHERE ~A > 3" where the ~A's get filled with variables later)

However, the AWK route is also an
approach that can solve a lot of simple problems, probably even the
one you solved using macros. Note I'm not going to argue that it any
solution using AWK is *better* than using a macro based solution, only
that to sell macros to someone using the AWK approach the pitch has to
convince them that the macro approach is significantly better in
order to be worth investing the time and effort to learn. What
constitues "significant" obviously varies.

This trade-off is not unique to macros, it is what we all face all the time
with new and maybe better tools or approaches. The start-up cost makes the
first uses too expensive. I don't know AWK at all, I'm sure if I did I
would not just open that file in emacs and use a few keyboard macros as
often...

"By the time I walk to the shed and get the socket wrench, I'll have already
got this nut off with the vice-grips..." And it's true! The problem being
that then at nut number five I'm thinking "if I had just gone to get that
socket wrench in the beginning, this next nut would already be off" I have
to confess I am guilty of that all the time....But I try not to kid myself
that vice-grips are generally the best tool for removing nuts and bolts.

But back to macros, and really any language feature, they are all just tools
and it is always a judgment call as to what the best tool for a job is and
judgements are always subjective. It is very hard to convince anyone that a
tool they are completely unfamiliar with is the best one for some problem
they never thought they had.

 

[Stephen J. Bevan]

But the question is "why macros?" Answer, so you do not have to use Awk.

That seems to be the question you want to answer, but I don't think it
is the question others are really asking based on the fact that the CL
macro examples haven't obviously won many over. I explained in the
message to which you responded one approach that can be taken in order
to sell macros more effectively IMHO. I have no desire to debate the
merits of that approach, people can use it or ignore it as they see fit.

 

[Kenny Tilton]

That seems to be the question you want to answer, but I don't think it
is the question others are really asking based on the fact that the CL
macro examples haven't obviously won many over.

<heh-heh> The stuff I write in response to macro-resistant argumentation
is not for the arguers, it is for lurkers who might actually be thinking
while they are reading. jeez, Paul Graham wrote a whole book on macros,
lispniks swear by macros. That's all it takes to get "I am Curious
(language)" types to get interested.

NG arguers are just NG arguers.

Here's a real-live working macro I used last week:

(defmodel ring-net (family)
(
(ring-ids :cell nil :initform nil
:accessor ring-ids :initarg :ring-ids)
(sys-id :cell nil :initform nil :accessor sys-id :initarg :sys-id)
(reachable-nodes :initarg :reachable-nodes :accessor reachable-nodes
:initform (c? (let (reachables)
(map-routers-up
(lambda (node visited)
(declare (ignore visited))
(push node reachables))
(find (sys-id self) (^kids)
:key 'md-name))
reachables)))
(clock :initform (cv 0) :accessor clock :initarg clock)
))

DEFMODEL wraps DEFCLASS, itself a macro. Let's expand the above:

<<<BEGIN>>>>>
(progn (eval-when compile-toplevel :execute :load-toplevel)
(setf (get 'ring-net :cell-defs) nil))
nil nil
(eval-when compile-toplevel :execute :load-toplevel)
(setf (md-slot-cell-type 'ring-net 'reachable-nodes) t)
(unless (macro-function '^reachable-nodes)
(defmacro ^reachable-nodes (&optional (model 'self) synfactory)
(excl::bq-list `let (excl::bq-list (excl::bq-list
`*synapse-factory* synfactory))
(excl::bq-list 'reachable-nodes model)))))
(eval-when compile-toplevel :execute :load-toplevel)
(setf (md-slot-cell-type 'ring-net 'clock) t)
(unless (macro-function '^clock)
(defmacro ^clock (&optional (model 'self) synfactory)
(excl::bq-list `let (excl::bq-list (excl::bq-list
`*synapse-factory* synfactory))
(excl::bq-list 'clock model)))))
(progn (defclass ring-net (family)
((ring-ids :initform nil :accessor ring-ids
:initarg :ring-ids)
(sys-id :initform nil :accessor sys-id
:initarg :sys-id)
(reachable-nodes :initarg :reachable-nodes
:accessor reachable-nodes
:initform
(c? (let (reachables)
(map-routers-up (lambda
(node
visited)

(declare (ignore visited))
(push
node reachables))
(find


(sys-id self)
(^kids)

:key
'md-name))
reachables)))
(clock :initform (cv 0) :accessor clock
:initarg clock))
documentation "chya") default-initargs)
metaclass standard-class))
(defmethod shared-initialize :after ((self ring-net)
slot-names &rest iargs)
(declare (ignore slot-names iargs))
(unless (typep self 'model-object)
(error "if no superclass of ~a inherits directly
or indirectly from model-object, model-object must be included as a
direct super-class in
the defmodel form for ~a" 'ring-net 'ring-net)))
nil nil
(progn (defmethod reachable-nodes ((self ring-net))
(md-slot-value self 'reachable-nodes))
(defmethod (setf reachable-nodes) (new-value (self
ring-net))
(setf (md-slot-value self 'reachable-nodes)
new-value))
nil)
(progn (defmethod clock ((self ring-net)) (md-slot-value
self 'clock))
(defmethod (setf clock) (new-value (self ring-net))
(setf (md-slot-value self 'clock) new-value))
nil)
(find-class 'ring-net)))

<<<END>>>>

Oh, I'm sorry, did I fill up your hard drive? Were you looking forward
to typing all that by hand? And revisiting a hundred or so like that
when you decided to change what DEFMODEL does?

That is point #1.

Point #2: some but not all of the above went away when I did a version
using a metaclass. But then I wanted to open source the hack, and not
all Lisps do the MOP. The point being that in general one can get very
cool semantics absent cool built-in language stuff precisely because
sufficient plumbing can achieve most cool effects, and macros let one
hide all that plumbing. This point will be too subtle for many, whow
might yell out "But my language has a MOP and there is only one imp". I
am making a more general point: macros mean one is not limited by ones
chosen language.

Point #3: DEFMODEL is part of a VLH (Very Large Hack). Now you might
say, "I don't do Very Large Hacks!". Well, maybe you do, but maybe you
just ahve a lot of function calls all over with a lot of redundant,
boilerplate code arranged Just So to conform to the requirements of the
hack. With macros the boilerplate disappears (and again) you thank your
lucky stars when you make a neat improvement to the VLH and you do not
need to revisit all the boilerplate.

Now let's look in one level, to:

(c? (let (reachables)
(map-routers-up (lambda (node visited)
(declare (ignore visited))
(push node reachables))
(find (sys-id self) (^kids)
:key 'md-name))
reachables))

and expand that:

(MAKE-C-DEPENDENT
:CODE '((LET (REACHABLES)
(MAP-ROUTERS-UP (LAMBDA (NODE VISITED)
(DECLARE (IGNORE VISITED))
(PUSH NODE REACHABLES))
(FIND (SYS-ID SELF) (^KIDS) :KEY 'MD-NAME))
REACHABLES))
:RULE (C-LAMBDA (LET (REACHABLES)
(MAP-ROUTERS-UP (LAMBDA (NODE VISITED)
(DECLARE (IGNORE VISITED))
(PUSH NODE REACHABLES))
(FIND (SYS-ID SELF) (^KIDS)
:KEY 'MD-NAME))
REACHABLES)))

That exaggerates things a little, because that code keyword is receiving
at runtime the /source/ of the form expanded into the code to be
compiled as an anonymous function, so I can figure out what code crashed
when I end up in a backtrace. Sorry if I scared the children with that one.

Now don't feel bad if you don't recognize C-LAMBDA, that's mine:

(LAMBDA (#:G1000 &AUX (SELF (C-MODEL #:G1000))
(.CACHE (C-VALUE #:G1000)))
(DECLARE (IGNORABLE .CACHE SELF))
(ASSERT (NOT (CMDEAD #:G1000)) NIL
"cell dead entering rule ~a" #:G1000)
(LET (REACHABLES)
(MAP-ROUTERS-UP (LAMBDA (NODE VISITED)
(DECLARE (IGNORE VISITED))
(PUSH NODE REACHABLES))
(FIND (SYS-ID SELF) (^KIDS)
:KEY 'MD-NAME))
REACHABLES))

Still there? OK, this VLH has many different kinds of macros such as C?,
but they all get called in exactly one place. They need the same lambda
signature. Before I had C-LAMBDA, a rare change to that signature was a
real pain. No mas.

OK, by now I am sure you are fascinated by: (^kids)

Expanded, we get: (KIDS SELF)

Beezlebub! Don't worry, it compiles. The ^thingy macros are meant to be
invoked only in lexical contexts supplying the Smalltalk-like ubiquitous
anaphor SELF. look back at the expansion of c-lambda to see SELF being
pulled out of the model slot of the cell. ie, big bad variable capture
has been harnessed for a good cause, emulating the anaphoric variables
of SELF and THIS of other languages. I scared a grown-up Lispnik with
this, once.

^these used to do even more work, but an improvement made them (almost)
unnecessary, but I kept them because I need them for the following and
because I kinda dig the way they shout out "I am using one of my own
slots!".

POINT #4: Sometimes macros just brighten up the code a little.

Where they are still needed is:

code: (^clock system (fSensitivity 30))
expansion: (LET ((*SYNAPSE-FACTORY* (FSENSITIVITY 30)))
(CLOCK SYSTEM))

whoa! where did all that come from? the macro decided to write a little
more code when it saw me using the synapse option. Now the important
thing here is that *SYNAPSE_FACTORY* gets picked up in the access done
by (clock system), so ya can't do a high-level function:

(func-to-apply-synapse (FSensitivity 30) (clock system))

and you can't pass it to the clock function because that is an accessor
and good CLOSians don't **** with accessor signatures.

in fact whatever trick you come up with would be uglier than:

(LET ((*SYNAPSE-FACTORY* (FSENSITIVITY 30)))
(CLOCK SYSTEM))

So yer stuck with it. Not me, tho.

POINT #5: Sometimes macros brighten up the code /and/ hide necessary
plumbing.

Now getting back to the VLH issue, all the above is about taking
something developers do in any language from 6502 to Lisp, viz, create
little code worlds in which functions and variables and stuff are
regularly used repeatedly many times, and allowing one's implementation
to seemingly expand to include the new custom code world, hiding a lot
of boilerplate in the process.

Now I know what you are going to say. No one will buy any of that
because they have no fucking idea what I am talking about. Ah, but they
can tell I am having a /lot/ of fun, and <cue Burdick> if you check the
highlight film below, you'll find a whole section on Hedonists.

kenny



--
http://tilton-technology.com

Why Lisp? http://alu.cliki.net/RtL Highlight Film

Your Project Here! http://alu.cliki.net/Industry Application

 

[Dirk Thierbach]

The problem was a fairly ordinary one and is easy to describe at a high
level. The server existed to control a complicated configuration process
and later process data according to the result. It was a simple socket
server and I got to dictate the form of the commands and arguments.
Ultimately, it boiled down to (funcall command client-session args).

I would be very happy to learn other approaches, there are always
many ways to skin a cat. (how un-PC is that saying these days

Here's a very simplified example how to do something similar in
Haskell. I probably missed lots of details (I don't know how the
session object works, and have no idea what the session-blueprints
are, I have no details about the send and receive commands, and so on),
but maybe it should give you an idea.

The main task is to define a function that implements a server
command, given some constraints and the code for the command.
The command should work on the arguments received over the socket.

So we need some code that does the marshalling und un-marshalling of
the data. Here, I will cheat and just use the "Dynamic" library
to simulate that. In reality, one could use a similar approach
to write marshalling routines, or use the standard read and show
routines, or whatever. I'll use the functions

toDyn :: Typeable a => a -> Dynamic

that converts any "typeable" value into a Dynamic value, and

fromDynamic :: Typeable a => Dynamic -> Maybe a

which converts it back, or fails with a value of "Nothing" if the type
doesn't match. The class Typeable will need instances for all the
types we want to convert.

Let's assume there is a type "Session" for sessions. Then the type
of define_server_cmd will be

define_server_cmd :: Typeable a => (a -> Bool) -> (Session -> a -> IO ())
-> (Session -> Dynamic -> IO ())

which pretty much expresses out expectations about this function given
above. The implementation is straightforward:

define_server_cmd constraints code session dyn_args = do
maybe
(error "Wrong type of arguments")
(\args -> if constraints args
then code session args
else error "Constraints violated"
)
(fromDynamic dyn_args)

Your set-field-sequence example would probably look something like
(I am guessing a lot here)

set_field_sequence = define_server_cmd constraints code where
constraints field_list = all (`elem` logical_types) field_list
code session field_list = do
let field_sequence = source_blueprint session
state = get_state session
writeIORef field_sequence field_list
send session (show state)

You don't need the first constraint, because the unmarshalling will
check if the arguments are indeed a list. I am not sure if the
second constraint is also a type check or if it checks the contents
or names of the fields.

As another example, let's define the code for a print command
seperately (so it is testable), and then make a server command out
of it with the constraint that the first argument is less then 100:

print_cmd_code :: Session -> (Integer, String) -> IO ()
print_cmd_code session (x, y) =
print ("First=" ++ show x ++ ", Second=" ++ show y)

print_cmd = define_server_cmd (\(x, _) -> x < 100) print_cmd_code


As I said, this is a very simple example, so let's see what is missing.

So I needed (wanted) a way to, in one stroke,
- define the method, properly specialized on the session object

Should work.

- ensure it became an allowed function to call

Should work.

- define a way to gather the arguments needed from the client
according to number of args and the type of each.

Should work. Note that the compiler automatically figures out at
compile time which code to combinbe to marshall or unmarshall the
"dynamic" value, and verifies the type as well. No need to do this
explicitely.

- provide a facility for arbitrarily complex validation of any
of the arguments or combinations thereof.

Should work.

- ensure that any changes, enhancements or additions to
argument passing would require a single point of change in my code.

Should work.

One potential drawback of the above approach is that you have to write
down the argument list twice, once for the contraint check, and once
for the actual code. But since one might want to do pattern matching
on the arguments, this might be exactly the right thing and not a
drawback. The type checker will verify that the arguments are of the
same for both parts.

Additional benefits:
- automatically provided information to a kind of "help" facility.

I didn't see that in your code, but one could probably handle this
in the same ways as errors below.

- allowed for very informative error messages, both in the debug
environment and as returned data for clients.

I kept the error handling *very* simple by just throwing exceptions.
In reality, this would be handled by an error function that sends back
the error message. Type errors could be automatically handled. It is
of course not possible to send back sourc code as a string without
using a macro; and while error messages of this kind are arguably
informative to someone who knows lisp, they might be very confusing to
someone who doesn't So I'd add explicit error messages, and maybe
a few infix functions as syntactic sugar to write them down nicely.

- simplified the main server loop while remaining completely flexible

The main loop would need to lookup the server functions (which all
have the same type), and then execute the functions without
unmarshalling the arguments, since that is done inside the
function. Also simple and flexible.

I did have to define a set of safe-read functions for lists and
integers and strings etc for a combination of security and
timing-out reasons.

Yes. This code would go in the typeclasses mentioned at the beginning,
and as you say, you'd need that in Lisp, too.

But the grammar was just s-expressions.

The grammar is completely up to the coder, be it s-expressions, XML, or
whatever. However, it should include the type of the arguments passed
over the socket.

I actually have macros I use now that generate strings of SQL code,
some of it as format strings that will be used at a later stage of
processing (a "format string" would be something like "SELECT NEW.~A
FROM ~A WHERE ~A > 3" where the ~A's get filled with variables
later)

It might be interesting for you to have a look at HaXML or one of the
other XML libraries that do a similar thing with XML, using only
HOFs. I don't see any reason why SQL couldn't be handled in a similar way.

But back to macros, and really any language feature, they are all
just tools and it is always a judgment call as to what the best tool
for a job is and judgements are always subjective. It is very hard
to convince anyone that a tool they are completely unfamiliar with
is the best one for some problem they never thought they had.

Amen to that.

- Dirk