Macros in Ruby

[Mikael Brockman]

Hi --



My ignorance of Lisp macros is almost complete. Mainly I was going by
the (possibly completely wrong) impression that what people usually
mean by "having macros in Ruby" is allow on-the-fly redefinition of
syntax, including punctuation.

Lisp does allow you to completely redefine the syntax, but through
features that aren't usually included in the typical definition of
``Lisp macros'' -- namely, the ability to define reader macros, which
basically lets you hook into the lexer and do whatever you want.

Lisp syntax essentially consists of a bunch of literals (integer
literals, string literals, symbols, etc) and ``forms.'' A form is
a list, and looks like (foo bar baz). In typical circumstances, the
evaluation of a form is initially dispatched on the head of the list.

If the head is a function name -- i.e., either a symbol whose value is a
function, or a LAMBDA form -- then the corresponding function is called
with the results of evaluation of every other item in the list.

If the head names a macro, the tail of the list is applied to the macro
without evaluating the arguments. The macro is like a regular Lisp
function, but what it's returned isn't the value of the form -- first,
the result is evaluated; the macro returns an expression, not a value.

Lisp doesn't make that annoying, since Lisp syntax consists solely of
Lisp values. (+ 1 2) is just a list of the symbol +, the number 1, and
the number 2.

If Lisp macros were implemented in Ruby, they wouldn't necessarily let
you change the syntax to your whims. Your macros would need to conform
to the standard method call syntax, for example.

It's true that many Lisp macros are merely kludges to avoid typing
`lambda', and Ruby's blocks address that nicely. You can even specify
some data declaratively with blocks. But Lisp macros are definitely
more powerful -- they can do arbitrary introspection and manipulation of
their arguments.

Also, I'm not very coherent at this time of night.

mikael

 

[Yukihiro Matsumoto]

Hi,

In message "Re: Macros in Ruby"

|Would true macros in Ruby be more prone to abuse than they are in Lisp?

* Lisp does not have syntax. They only have meta syntax
(i.e. S-expression), Lisp macro do not change its (meta) syntax to
rely on (I'm ignoring reader macro here). In comparison, Ruby
does have syntax but no meta syntax, so that you can easily loose
syntax to rely on by macros in Ruby. You will feel like a stranger
when you see Ruby programs with a lot of macros. I don't think
you feel same way when you see Lisp programs with macros.

* macro system more than simplest one (e.g. C preprocessor macro) is
VERY difficult to design and implement for languages with usual
syntax. If you are curious, see Dylan's macro.

* about 50% of macro usage in Lisp (extending syntax a bit) can
accomplish using blocks in Ruby. we don't need inlining functions
when we have other rooms for optimization, for example, C extensions.

* I admit disclosing abstract syntax tree to Ruby programs can open
new possibilities. it's good for users at the moment, I guess.
but it is very bad in a long run unless I design it perfectly.
I'm sure I will change my mind in AST design and alas, here comes
another big field of incompatibility.

matz.

 

[James Britt]

Agreed.

While Matz may have 'banned' macros from the core language, that doesn't
mean that if someone comes up with a way to do LISP-like macros
implemented in an external module, (that could be downloaded
from the RAA for example) that Matz would ban the module. He probably
would never consider bundling it with the Ruby distribution, of course,
but that doesn't mean it would be 'banned'.

If such a module did ever become available those who would be interested
in such things would use it and those who were not interested would
ignore it.

It's kind of like how the static typing advocates occasionally come up
with some code for emulating static typing in Ruby (and you can probably
find some on the RAA) while most of us think that such things are
ill-advised, no one is stopping the advocates of such things from using
it.

Oh, nice move. Associating Lisp-style macros with static typing.

I curse you, Phil Tomson!

James

 

[Phil Tomson]

Oh, nice move. Associating Lisp-style macros with static typing.

I curse you, Phil Tomson!

Ohhh no!! Another 7 years! ;-)

Actually, I find the idea of Lisp-style macros interesting, while I find
the idea of static typing in Ruby just plain bad... but YMMV.

Phil

 

[Robert Klemme]

You are of course right. But you can do still usefull things. And I
think it can be improved. Any ideas are welcome. Even better would be
support for macros in Ruby 1.9

-1

robert

 

[George Moschovitis]

Another comment: maybe instead of redefining 'require' you should just ....
strings or here-docs?

Hello Phil,

thanks for your comments. I 'll work on the code a bit and post a new
version. I 'll try to include your requests. I agree with you that
macros are usefull. For example i use them in a coll localization
scheme, or even for loging (using this simple macro implementation,
I can fully strip the logging/profiling code, much better than using
a block, and more flexible than using AOP techinques).

Anyway it would be nice to hear more ideas, comments!

regards,
George Moschovitis

 

[gabriele renzi @ google]

* I admit disclosing abstract syntax tree to Ruby programs can open
new possibilities. it's good for users at the moment, I guess.
but it is very bad in a long run unless I design it perfectly.
I'm sure I will change my mind in AST design and alas, here comes
another big field of incompatibility.

is this really a problem?
IMO a developer accessing the AST (or just the parser)
should be aware of what he's doing and expect some breakage.
IIRC python has modules to access, the AST, the parser and the VM
internals, and this has proved quite ufeful even if python changed
much over time..

 

[Joel VanderWerf]

For example, suppose you're writing a game and you have a lot of code
that selects behavior based on percentages (75% of hits are normal
blows, 20% are critical hits, and 5% kill your foe out-right for
example). Even in a language as expressive as Ruby this is a bit
awkward to write and not especially intentional. But with macros it can
be written very cleanly:

percent-case
0.75 case
normal_hit
end

0.20 case
critical_hit
end

otherwise
kill_foe
end
end

It's not really that hard to do something like this in ruby, but it is a
little less efficient than it would be with macros.

class PercentCase
def initialize block
@cases = []
instance_eval(&block)
end

def choose
n = rand(100)
@cases.each do |percent, block|
n -= percent
return block.call if n < 0
end
otherwise.call
end

def chance percent, &block
@cases << [percent, block]
end

def otherwise(&block)
@otherwise = block
end
end

def percent_case(&block)
PercentCase.new(block).choose
end

10.times do
percent_case do
chance 75 do
puts "normal_hit"
end

chance 20 do
puts "critical_hit"
end

otherwise do
puts "kill_foe"
end
end
end

To make it more efficient, you could keep the PercentCase object around
and reuse it each time.

 

[Ben Giddings]

For example, suppose you're writing a game and you have a lot of code
that selects behavior based on percentages (75% of hits are normal
blows, 20% are critical hits, and 5% kill your foe out-right for
example). Even in a language as expressive as Ruby this is a bit
awkward to write and not especially intentional. But with macros it can
be written very cleanly:

percent-case
0.75 case
normal_hit
end

0.20 case
critical_hit
end

otherwise
kill_foe
end
end

I'm sorry to say it, but I think you just proved Matz' point there. I
look at that, and it doesn't look like Ruby to me. Even with your
explanation, I don't understand how it works, or how to modify it. Is
"case" in your code the normal ruby "case" statement? It doesn't look
like it.

The one place I've really missed macros is in debugging. I'm really
used to C macros that are really simple to type 'dbg(val);' but can
produce detailed valuable output:

foo.c:132 do_it() val => 23

There are hackish ways to get something approaching this in Ruby, mainly
by twiddling with the 'caller' array, and using symbols to grab the name
of the symbol along with its value. That feels really hackish to me.
I'd feel slightly better about it if callers returned an array of
"SourceLine" objects, or something, and you could get the method name as
well. But it's still not quite up there with the ease of use of a C macro.

On the other hand, aside for this and a few other isolated cases, I
*hate* C macros, and *hate* the C preprocessor. It means that when
you're writing .c and .h files, you're not actually writing C code,
you're writing C/C-preprocessor code, and depending on who has touched
the codebase, you never quite know what's what.

I don't know if there's a middle ground, but I certainly lean towards
avoiding macros.

Ben

 

[Joel VanderWerf]

As far as I can tell Lisp lambda functions (ie closures) do everything
that Ruby blocks and proc objects do but with cleaner semantics and
fewer artifical distinctions.

What are some of the differences? I've been away from lisp for a while....

I'm not so sure, I experimented with defining FSMs in a dynamic
language with good support for anonymous functions/closures and the
macro based solution was a whole lot nicer to use.

I've found ruby blocks to be fairly good for defining declarative
constructs. For example, I based the syntax for a declarative language
for hybrid automata (~ FSMs + ODEs) on this technique. Here's a sample:

class Vehicle < Component
state :SpeedUp, :SlowDown

default do
self.x = 0
self.xDot = 20
start SlowDown
end

setup do
@timer = create(Timer)
end

flow SpeedUp, SlowDown do
diff "xDot' = xDDot"
diff "x' = xDot"
end

flow SpeedUp do
alg "xDDot = 3"
end

flow SlowDown do
alg "xDDot = -3"
end

transition SpeedUp => SlowDown, SlowDown => SpeedUp do
guard {@timer.tick}
action {puts "Tick!"}
end
end

(The parsing of all these blocks--using instance_eval plus manual
parsing of the differential/algebraic equations--happens at load time,
then there is a compile stage which generates a dynamic lib and loads
it, and at run time it's mostly native code executing, with an
occasional call to a ruby proc or method.)

The one respect in which ruby was less than perfectly suitable to this
syntactical problem is having to use "self.x = ..." all the time, but
there are good reasons for that in the bigger picture.

Yeah, that's one common application.

Hm? Blocks can be used to defer evaluation, using the &block or Proc.new
constructs. Or do you mean something else?

I actually come at this from a Dylan POV, not from a Lisp one and Dylan

I still wish Dylan hadn't been left to wither...

 

[Phil Tomson]

This seems hard to do well. A good macro system really needs to be
integrated into the language. If not, you wind up with abortions like
the C/C++ macro system which is both dangerous to use and
under-powered.

It might be possible to do with an open compiler (ie one that provided
hooks to allow third-party components to affect its behavior). I tend
to be a bit suspicious of open compilers though. They seem to either
lock users into particular implementations or significantly constrain
compiler implementations. I'll have to think about it some more, but it
seems like adding macros wouldn't be all that much more difficult to
implement than opening the parser up.


I don't see the connection. Static typing runs counter to the whole
concept of typing in Ruby. Macros, on the other hand, are a completely
orthogonal concept and, if anything, fit well into the Ruby philosophy
of providing a flexible and expressive language.

Read my later post. I was only using that debate as an example...

There is a camp of folks who think that Ruby needs to emulate
static-typing in some cases. There is another camp that thinks this is a
bad idea. Similarly, there is a camp that says that macros would be
useful (or at least interesting to explore - that's kind of where I'm at
since I've never used LISP macros) and there is another camp (which Matz
is in) that says that macros are just plain bad. Since Matz is in the
latter camp you're not likely to see macros included in Ruby
'out-of-the-box', but if you can some up with an addon to do macros (if
it's possible)...

I wasn't trying to tie static typing and macros together.

Phil

 

[Paul Brannan]

The one place I've really missed macros is in debugging. I'm really
used to C macros that are really simple to type 'dbg(val);' but can
produce detailed valuable output:

foo.c:132 do_it() val => 23

This looks more like logging than debugging. C macros typically make
debugging more difficult, not less, because most debuggers don't let you
"step into" a macro like you can with a function.

There are hackish ways to get something approaching this in Ruby, mainly
by twiddling with the 'caller' array, and using symbols to grab the name
of the symbol along with its value. That feels really hackish to me.
I'd feel slightly better about it if callers returned an array of
"SourceLine" objects, or something, and you could get the method name as
well. But it's still not quite up there with the ease of use of a C macro.

The caller() method certainly has its problems. The API could be easier
to use, but that doesn't make C macros better. You are comparing the
design of two APIs (the Ruby API and the C preprocessor API), not macros
versus no-macros.

I don't know if there's a middle ground, but I certainly lean towards
avoiding macros.

What experience do you have with lisp macros or other non-C-preprocessor
macros?

Paul

 

[Jim Weirich]

percent-case
0.75 case
normal_hit
end

0.20 case
critical_hit
end

otherwise
kill_foe
end
end

I spent about 10 minutes coding this in Ruby (and it turned out to be
very similar to Joel VanderWerf's version that he posted). I'm curious
what a macro version would be like. What would the implementation of
percent_case look like in Lisp or Dylan macros?

 

[Austin Ziegler]

I don't buy this argument at all. It's not like good Ruby
programmers would all of a sudden use macros to rewrite the Ruby
grammar. Instead they would augment the grammar with productions
that make sense for their applications.

For example, suppose you're writing a game and you have a lot of
code that selects behavior based on percentages (75% of hits are
normal blows, 20% are critical hits, and 5% kill your foe
out-right for example). Even in a language as expressive as Ruby
this is a bit awkward to write and not especially intentional. But
with macros it can be written very cleanly:

percent-case
0.75 case
normal_hit
end

0.20 case
critical_hit
end

otherwise
kill_foe
end
end

I'm sorry, but I don't see how this is more readable than:

case hit_roll
when (0.0 .. 0.75)
normal_hit
when (0.75 .. 0.95)
critical_hit
else
kill_foe
end
end

Yes, I've switched the percentages to Ranges, but there's no reason
that you couldn't build a similar mechanism into one of your own
classes:

class Attacker
def initialize
@total = 0
@results = {}
yield self if block_given?
end

def add_result(percentage, &block)
if (percentage > 100) or (@total + percentage > 100)
raise ArgumentError
end

percentage_range = (@total .. (@total + percentage))
@results[percentage_range] = block
@total += percentage
end

def hit(percentage)
if percentage.kind_of?(Float)
percentage = (percentage * 100).to_i
end
res = nil
@results.each_key do |rr|
if rr === percentage
res = @results[rr].call(percentage)
break
end
end
res
end
end

hero = Attacker.new do |hh|
hh.add_result(75) { |p| puts "Normal hit with #{p} roll." }
hh.add_result(20) { |p| puts "Critical hit with #{p} roll." }
hh.add_result(5) { |p| puts "Deadly hit with #{p} roll." }
end

(0 ... 20).each { hero.hit(rand) }

# Result:
Normal hit with 60 roll.
Normal hit with 5 roll.
Normal hit with 75 roll.
Normal hit with 74 roll.
Normal hit with 43 roll.
Normal hit with 10 roll.
Normal hit with 8 roll.
Normal hit with 21 roll.
Normal hit with 31 roll.
Deadly hit with 99 roll.
Normal hit with 50 roll.
Normal hit with 47 roll.
Normal hit with 48 roll.
Normal hit with 72 roll.
Normal hit with 3 roll.
Critical hit with 82 roll.
Normal hit with 48 roll.
Critical hit with 86 roll.
Critical hit with 76 roll.
Normal hit with 3 roll.

-austin

 

[Dennis Ranke]

For example, suppose you're writing a game and you have a lot of code
that selects behavior based on percentages (75% of hits are normal
blows, 20% are critical hits, and 5% kill your foe out-right for
example). Even in a language as expressive as Ruby this is a bit
awkward to write and not especially intentional. But with macros it can
be written very cleanly:
percent-case
0.75 case
normal_hit
end
0.20 case
critical_hit
end
otherwise
kill_foe
end
end

It's not really that hard to do something like this in ruby, but it is a
little less efficient than it would be with macros.

[...]

Or you could even do it like this, which looks (to me) even nicer than the
macro solution

class PercentCase
def initialize
@v = rand(100)
end

def test(o)
@v -= o
@v <= 0
end
end

class Integer
alias_method ercentCase, :===
def ===(o)
if o.is_a? PercentCase
o.test self
else
percentCase o
end
end
end

10.times do
case PercentCase.new
when 75
puts 'normal_hit'
when 20
puts 'critical_hit'
else
puts 'kill_foe'
end
end

 

[Robert Klemme]

Just some minor remark...

What are some of the differences? I've been away from lisp for a while....

I've found ruby blocks to be fairly good for defining declarative
constructs. For example, I based the syntax for a declarative language
for hybrid automata (~ FSMs + ODEs) on this technique. Here's a sample:

class Vehicle < Component
state :SpeedUp, :SlowDown

default do
self.x = 0
self.xDot = 20
start SlowDown
end

You could get rid of the "self" if you defined a method that receives a
hash along the lines of:

default do

values( :x => 0, :xDot => 20 )
start SlowDown
end

Or even remove the values setting from the block.

Since I don't know the inner workings I don't know whether there are other
factors that prevent this solution. Just an idea.

setup do
@timer = create(Timer)
end

flow SpeedUp, SlowDown do
diff "xDot' = xDDot"
diff "x' = xDot"
end

flow SpeedUp do
alg "xDDot = 3"
end

flow SlowDown do
alg "xDDot = -3"
end

transition SpeedUp => SlowDown, SlowDown => SpeedUp do
guard {@timer.tick}
action {puts "Tick!"}
end
end

(The parsing of all these blocks--using instance_eval plus manual
parsing of the differential/algebraic equations--happens at load time,
then there is a compile stage which generates a dynamic lib and loads
it, and at run time it's mostly native code executing, with an
occasional call to a ruby proc or method.)

So you create C code and compile it?

The one respect in which ruby was less than perfectly suitable to this
syntactical problem is having to use "self.x = ..." all the time, but
there are good reasons for that in the bigger picture.

See above.

Regards

robert

 

[Gavin Sinclair]

Lisp lambda functions *do* do everything that Ruby blocks do. But
doing is not the same as being. I wrote that they are not equivalent.
The difference is syntax. It's an artifical distinction that works
tremendously well in practice at expressing the solution in code.

Between Ruby and Lisp? To tell the truth I'm not entirely sure. The
pickaxe book does a lousy job explaining the differences between blocks
and proc objects. I'm still not sure why Ruby needs two concepts when
Lisp and other languages get by with one. I also don't know why Ruby
uses that weird yield syntax.

On the one hand you advocate macros to make syntax more expressive; on
the other you advocate rolling everything back into anonymous
functions with no visual distinctions. *shrug*

Gavin

 

[Austin Ziegler]

A macro (or some clever code like Joel's) is better in two ways:

1) It's much easier to glance at numbers like 0.75 and 0.20 and figure
out the relative weighting. For me at least, it's a fair amount harder
to subtract two sets of numbers and compare them. Of course, you could
add comments listing the percentages, but it's always preferable to
rewrite the code so that it doesn't need such comments.

Hmm. I disagree. I mean, it's better to deal with code like Joel's (or
my Attacker class that I presented, which even makes sure you can't go
over 100%), because then you've treated the cases as closures and
objectified the whole thing, so that you don't have to do the numbers,
but basically (0 .. 75) says very clearly that you'll accept any value
in that range. It's harder to be dynamic, but you'll note that my
Attacker class (the second possibility in the email to which you
responded) makes that easy and still uses the range comparison.

I don't think that a domain-specific language is needed for that case.
The only situation I can think of that came up recently was the one
where someone wanted to introduce the a new operator ":="; that could
be handled with a domain-specific language with ease.

-austin

 

[Steven Jenkins]

It's not Ruby. It's a little domain specific language (DSL) tailored to
a particular problem.

I don't have an opinion on macros per se, but DSLs are (in my opinion) a
poor application for macros. To paraphrase Dennis Ritchie, if you want
racc, you know where to find it.

I'm a big fan of DSLs, but I like to build them up from the grammar.

Steve

 

[Rando Christensen]

A macro (or some clever code like Joel's) is better in two ways:

1) It's much easier to glance at numbers like 0.75 and 0.20 and figure
out the relative weighting. For me at least, it's a fair amount harder
to subtract two sets of numbers and compare them. Of course, you could
add comments listing the percentages, but it's always preferable to
rewrite the code so that it doesn't need such comments.

I don't think you're right on this point. In the case that was
presented:

0.75 case
normal_hit
end

Does this mean 0.75 and above, or 0.75 and below? You won't know
this unless you look up the macro's definition. Here, however:

when (0.0 .. 0.75)
normal_hit

I don't think you can be confused on that point.

The macro version has made the code much less clear in my mind, which is
precisely the argument against macros in ruby.