Best gem to parse Ruby with?

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

I've been considering rewriting my require_all gem:

http://github.com/tarcieri/require_all/

...to parse the Ruby source code, extracting a list of constants a
particular file defines and a list of constants that are expected to be
defined before a file is loaded.

The gem can iterate through all files requested to be loaded, generate these
lists of constants per file, and from there resolve the proper order that
the files should be loaded in, provided all constants are resolvable.

This would solve a number of problems with require_all's approach.

What's the best gem to do this with? ruby_parser?

 

[Caleb Clausen]

I've been considering rewriting my require_all gem:

http://github.com/tarcieri/require_all/

...to parse the Ruby source code, extracting a list of constants a
particular file defines and a list of constants that are expected to be
defined before a file is loaded. [snip]
What's the best gem to do this with? ruby_parser?

No doubt its author will say it is, but I'd like to recommend my own
library, RedParse. If you can stand its painful slowness, it outputs
the nicest parsetrees.

see: http://github.com/coatl/redparse
or simply: gem install redparse

You should also consider ParseTree, which is likely to be faster than
either of the others.

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

No doubt its author will say it is, but I'd like to recommend my own
library, RedParse. If you can stand its painful slowness, it outputs
the nicest parsetrees.

I'm definitely looking for a pure Ruby solution, and in that regard RedParse
looks nice.

You should also consider ParseTree, which is likely to be faster than
either of the others.

ParseTree is great but it seems to only work with 1.8.x MRI which is
unacceptable to me. I guess I should've stated that outright. I'm fine
with an FFI solution but not something which needs a non-FFI native
extension.

 

[Luis Lavena]

[Note:  parts of this message were removed to make it a legal post.]

No doubt its author will say it is, but I'd like to recommend my own
library, RedParse. If you can stand its painful slowness, it outputs
the nicest parsetrees.

I'm definitely looking for a pure Ruby solution, and in that regard RedParse
looks nice.

You should also consider ParseTree, which is likely to be faster than
either of the others.

ParseTree is great but it seems to only work with 1.8.x MRI which is
unacceptable to me.  I guess I should've stated that outright.  I'm fine
with an FFI solution but not something which needs a non-FFI native
extension.

Pure ruby to parse Ruby doesn't fit you?

see ruby_parser:

http://rubyforge.org/projects/parsetree

Non-FFI extension is impossible. At the end you're going to need link
somehow to Ruby C level, and FFI *is* actually a C extension.

 

[Caleb Clausen]

ParseTree is great but it seems to only work with 1.8.x MRI which is
unacceptable to me. I guess I should've stated that outright. I'm fine
with an FFI solution but not something which needs a non-FFI native
extension.

RedParse doesn't work in 1.9 yet either, tho.

I forgot to mention RubyNode before. That fills out the list of ruby
parsing libraries I know of.

So you want something that:
works in 1.9
written in pure ruby
doesn't use extensions (except maybe thru FFI)

Taken together, I think these requirements eliminate all the
reasonable candidates.

Why do you care whether it uses FFI or the traditional extension api?
For that matter, why do you care what language it's written in?

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

I forgot to mention RubyNode before. That fills out the list of ruby
parsing libraries I know of.

So you want something that:
works in 1.9

1.9 support would be nice, but really I'm looking for something that works
on JRuby. If it doesn't work on 1.9 I'm not too worried for the time being.

written in pure ruby
doesn't use extensions (except maybe thru FFI)

Taken together, I think these requirements eliminate all the
reasonable candidates.

Why do you care whether it uses FFI or the traditional extension api?
For that matter, why do you care what language it's written in?

I don't care what language it's written in, however I don't want to depend
on the old extension API because I'd like for my gem to work on JRuby. I
plan on using this gem on JRuby in the very near future and in that regard
very much want to support it.

 

[Caleb Clausen]

1.9 support would be nice, but really I'm looking for something that works
on JRuby. If it doesn't work on 1.9 I'm not too worried for the time being.

Ah, ok. I don't know if RedParse works on JRuby or not; last I knew,
there were a number of strange things it (and its dependencies) were
doing that exposed bugs on JRuby. Those may well have been fixed by
now, tho.

In the JRuby world, there is JParseTree, the JRuby version of
ParseTree. Probably, you could write your code to use ParseTree on MRI
and JParseTree on JRuby. You should try it first, to see if it works
for you.

 

[Ryan Davis]

Ah, ok. I don't know if RedParse works on JRuby or not; last I knew,
there were a number of strange things it (and its dependencies) were
doing that exposed bugs on JRuby. Those may well have been fixed by
now, tho.

In the JRuby world, there is JParseTree, the JRuby version of
ParseTree. Probably, you could write your code to use ParseTree on MRI
and JParseTree on JRuby. You should try it first, to see if it works
for you.

Nope. Last I looked JParseTree wasn't compatible with the Unified Ruby
sexp format.

 

[Eric Hodel]

RedParse doesn't work in 1.9 yet either, tho.

I forgot to mention RubyNode before. That fills out the list of ruby
parsing libraries I know of.

So you want something that:
works in 1.9
written in pure ruby
doesn't use extensions (except maybe thru FFI)

Taken together, I think these requirements eliminate all the
reasonable candidates.

Um, ruby_parser works great on 1.9, is written in pure ruby and can
not use extensions. (It sits atop racc which is part of the standard
library and has a non-extension version. I imagine that racc would
ship with JRuby.)

 

[Caleb Clausen]

Um, ruby_parser works great on 1.9, is written in pure ruby and can
not use extensions. (It sits atop racc which is part of the standard
library and has a non-extension version. I imagine that racc would
ship with JRuby.)

I did not realize that. I thought the extension was required.

 

[Caleb Clausen]

Nope. Last I looked JParseTree wasn't compatible with the Unified Ruby
sexp format.

For what Tony is doing, I doubt he cares about the difference between
unified and raw trees.

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

For what Tony is doing, I doubt he cares about the difference between
unified and raw trees.

Interoperability is always nice

 

[Ryan Davis]

For what Tony is doing, I doubt he cares about the difference between
unified and raw trees.

Well the raw trees are incompatible... so unless they unify, there is
no way he can write code for ParseTree and then use it for jruby.

 

[Caleb Clausen]

Well the raw trees are incompatible... so unless they unify, there is
no way he can write code for ParseTree and then use it for jruby.

Tony is primarily interested in constants. It looks to me like
constants have the same representation in both variants. Correct me if
I'm wrong.