LiveAST: a pure Ruby 1.9.2 library obtaining live abstract syntaxtrees

[James M. Lawrence]

= LiveAST

== Summary

A pure Ruby library for obtaining live abstract syntax trees of
methods and procs.

== Synopsis

require 'live_ast'

class Greet
def default
"hello"
end
end

#### ASTs of methods

m = Greet.instance_methoddefault)

p m.to_ast
# => sdefn, :default, sargs), sscope, sblock,
# sstr, "hello"))))

#### ASTs of lambdas, procs, blocks

f = lambda { "foo" }

p f.to_ast
# => siter, scall, nil, :lambda, sarglist)), nil,
# sstr, "foo"))

def query(&block)
p block.to_ast
# => siter, scall, nil, :query, sarglist)), nil,
# sstr, "bar"))
end

query do
"bar"
end

#### ASTs from dynamic code

f = ast_eval "lambda { 'dynamic' }", binding

p f.to_ast
# => siter, scall, nil, :lambda, sarglist)), nil,
# sstr, "dynamic"))

ast_eval "def g ; 'dynamic' ; end", binding
m = methodg)

p m.to_ast
# => sdefn, :g, sargs), sscope, sblock, sstr, "dynamic"))))

== Install

% gem install live_ast

== Description

LiveAST enables a program to find the ASTs of objects created by
dynamically generated code. It may be used in a strictly noninvasive
manner, where no standard classes or methods are modified, or it may
be transparently integrated into Ruby (experimental). The default
setting is in between.

RubyParser is responsible for parsing and building the ASTs, though
another parser may be easily substituted (in fact the name to_ast is
used instead of to_sexp because LiveAST has no understanding of what
the parser outputs).

LiveAST is thread-safe.

Ruby 1.9.2 or higher is required.

== Links

* Documentation: http://liveast.rubyforge.org
* Rubyforge home: http://rubyforge.org/projects/liveast/
* Repository: http://github.com/quix/live_ast

== to_ruby

Although the ruby2ruby package (gem install ruby2ruby) is
not an official dependency of LiveAST, for convenience

require 'live_ast/to_ruby'

will require ruby2ruby and define the to_ruby method for Method,
UnboundMethod, and Proc. These methods are one-liners which pass
the extracted ASTs to ruby2ruby.

require 'live_ast'
require 'live_ast/to_ruby'

p lambda { |x, y| x + y }.to_ruby # => "lambda { |x, y| (x + y) }"

class A
def f
"A#f"
end
end

p A.instance_methodf).to_ruby # => "def f\n \"A#f\"\nend"

==

Thanks to Ryan Davis for writing RubyParser, Ruby2Ruby, minitest, and
many other great tools.

LiveAST is 249 lines of code (reported by simplecov) which employs a
simple trick to get these ASTs. Unlike ParseTree, which was the real
thing, LiveAST achieves its results by faking them. But like any
sufficiently plausible knockoff, you may never know the difference. Had
I thought of it sooner, I would have called it FarceTree.

==

James M. Lawrence

 

[Kaspar Schiess]

good news!

 

[Stephen Prater]

how does this work exactly? It hooks into the require / eval
mechanism and then runs all the code through a separate parser?

 

[James M. lawrence]

Stephen Prater wrote in post #982526:

how does this work exactly? It hooks into the require / eval
mechanism and then runs all the code through a separate parser?

Kernel#require is not touched. In MRI eval _can_ be replaced, but at the
price of syntax restrictions surrounding its use. It would be needless
to run _all_ code through a separate parser; only code tied to an AST
request is parsed (with RubyParser, though the parser can be replaced).
For anyone who's curious, the readme should (I hope) contain enough
explanation.

 

[Sam Barton]

So this is like ParseTree for Ruby 1.9? From a practical standpoint
could you please explain how it differs from ParseTree? I don't care
about the implementation details.

 

[Marc Heiler]

Hello,

Can someone explain to me what one could do with this?

It sounds very useful to be able and access the ASTs of methods and so
forth, I am just not completely sure what I can do with this new
information afterwards.

 

[James Lawrence]

So this is like ParseTree for Ruby 1.9? From a practical standpoint
could you please explain how it differs from ParseTree? I don't care
about the implementation details.

Short answer:
* use ast_eval instead of eval when ASTs are needed
* use to_ast instead of to_sexp

There is a caveat outlined in the readme, but the situation to which
it applies is rather unusual (code which frequently/continually evals
strings or reloads files without calling to_ast may need a periodic
flush_cache).

Incidentally it is only eval's implicit binding parameter which
prevents a seamless integration with Ruby; hence the ast_eval method.
Unfortunately Ruby has no robust way to handle this case
(Binding.of_caller works but is hobbled).

I debated whether to use to_sexp for the sake of familiarity, however
I wanted to emphasize that LiveAST is a general tool which can hook
into any parser. LiveAST has no business demanding that the parser
return a sexp in particular (hence to_ast).

 

[Kaspar Schiess]

Can someone explain to me what one could do with this?

Just to give this a quick and incomplete answer:

- you could rewrite the tree before executing it:
http://rewrite.rubyforge.org/

- you could transform ruby into sql:
http://errtheblog.com/posts/63-full-of-ambition

- you could transform ruby into ruby (think serialisation):
http://seattlerb.rubyforge.org/ruby2ruby/

- or you could write a ruby vm on top of ruby (shameless plug!):
https://github.com/kschiess/verneuil

& much much more! For inspiration, read on Lisp and transpose to Ruby

kaspar

 

[Sam Barton]

Marc Heiler wrote in post #982809:

Hello,

Can someone explain to me what one could do with this?

It sounds very useful to be able and access the ASTs of methods and so
forth, I am just not completely sure what I can do with this new
information afterwards.

My favorite example is transforming

db.select { |job, city| job == "spelunker" and city == "Miami" }

into a database query. The ruby code you see does not actually execute
-- some generated code does instead. Ruby syntax is transformed into
SQL syntax.

Syntax abstraction is very powerful and has thousands of uses, and
though it is normally a lisp thing, any language can do it with sexp
tools.

There are some grumps out there who tut-tut these approaches -- they
want nothing beyond a certain level of abstraction. "No, no, no!" they
say when they encounter anything Lisp-like. Can't you see them wagging
their fingers at us now?

 

[John Mair]

The weird thing about this project is it requires Ruby 1.9.2 and above,
yet RubyParser can only parse 1.8 code....

just an observation

 

[James M. Lawrence]

The weird thing about this project is it requires Ruby 1.9.2 and above,
yet RubyParser can only parse 1.8 code....

just an observation

LiveAST merely takes a static Ruby parser and electrifies it with the
Ruby runtime. The weirdness you observe is the lack of a full-featured
cross-implementation parser for Ruby 1.9. This project is only a
messenger for that weirdness

Ruby needs a parser:
http://groups.google.com/group/ruby-core-google/browse_thread/thread/3d245807cb0b42b1/

(The current implementation of LiveAST is different from the hacky
prototype mentioned in that thread, though it is still small.)

 

[John Mair]

Why not use Ripper for 1.9 code and RubyParser for 1.8 code?

James Lawrence wrote in post #983253:

 

[James M. Lawrence]

Why not use Ripper for 1.9 code and RubyParser for 1.8 code?

Existing tools use ParseTree sexps, which are RubyParser sexps. The
toolchain is not complete without a compiler (ruby2ruby + eval). The
1.8 syntax restriction only applies to files containing ASTs which are
actually requested. All other files may be filled with stabby lambdas,
if you like.

Here is the LiveAST readme synopsis with RubyParser:

% ruby synopsis.rb
sdefn, :default, sargs), sscope, sblock, sstr, "hello"))))
siter, scall, nil, :lambda, sarglist)), nil, sstr, "foo"))
siter, scall, nil, :query, sarglist)), nil, sstr, "bar"))
siter, scall, nil, :lambda, sarglist)), nil, sstr, "dynamic"))
sdefn, :g, sargs), sscope, sblock, sstr, "dynamic"))))

Now with Ripper:

% ruby -r live_ast_ripper synopsis.rb
[:def, [ident, "default", [4, 6]], [arams, nil, nil, nil, nil,
nil], [:bodystmt, [[:string_literal, [:string_content,
[tstring_content, "hello", [5, 5]]]]], nil, nil, nil]]
[:method_add_block, [:method_add_arg, [:fcall, [ident, "lambda",
[18, 4]]], []], [:brace_block, nil, [[:string_literal,
[:string_content, [tstring_content, "foo", [18, 14]]]]]]]
[:method_add_block, [:method_add_arg, [:fcall, [ident, "query", [28,
0]]], []], [:do_block, nil, [[:string_literal, [:string_content,
[tstring_content, "bar", [29, 3]]]]]]]
[:method_add_block, [:method_add_arg, [:fcall, [ident, "lambda", [1,
0]]], []], [:brace_block, nil, [[:string_literal, [:string_content,
[tstring_content, "dynamic", [1, 10]]]]]]]
[:def, [ident, "g", [1, 4]], [arams, nil, nil, nil, nil, nil],
[:bodystmt, [[:string_literal, [:string_content, [tstring_content,
"dynamic", [1, 9]]]]], nil, nil, nil]]

I'm willing to bet that 4 out of 5 dentists prefer RubyParser. Notice
the distinction being made between {} and do/end for blocks
(brace_block & do_block). Ripper is a more general tool; it's a less
abstract AST. The live_ast_ripper plugin is still available -- it's
just not the default.

JL