JRuby performance questions answered

[Charles Oliver Nutter]

Many folks still seem to be in the dark about JRuby performance relative
to Ruby 1.8.x. I figured a short post with some basic info could start a
discussion for anyone wanting to know more.

Long story short, JRuby 1.1 is now generally faster than Ruby 1.8.6, and
for most execution benchmarks it is the fastest 1.8-compatible Ruby
implementation available. This is based on the YARV benchmarks, the
Alioth benchmarks, a number of benchmarks we created for JRuby, and a
few other external benchmarks we run from time to time. Many (most?) of
these are microbenchmarks, but a number are nontrivial code.

For example, M. Ed. Borasky's MatrixBenchmark:

JRuby:
Hilbert matrix of dimension 64 times its inverse = identity? true
18.784000 0.000000 18.784000 ( 18.784000)

Ruby 1.8.6:
Hilbert matrix of dimension 64 times its inverse = identity? true
32.560000 0.110000 32.670000 ( 32.776915)

Or the YARV "pentomino" benchmark:

~/NetBeansProjects/jruby $ time jruby -J-server
test/bench/yarv/bm_app_pentomino.rb

real 1m31.770s
user 1m32.016s
sys 0m1.278s
~/NetBeansProjects/jruby $ time ruby test/bench/yarv/bm_app_pentomino.rb

real 1m48.100s
user 1m47.489s
sys 0m0.287s

Ola Bini of the JRuby team recently ran all the YARV benchmarks and
reported the results here:

http://ola-bini.blogspot.com/2007/10/mystery-expos-on-jruby-performance.html

And I periodically post about performance progress on my blog here:

http://headius.blogspot.com

I expect we'll see Alioth updated soon after the 1.1 or 1.1 beta 1 releases.

What about Rails?

One of the biggest uses for JRuby recently has been Rails. Rails
performance is a more complicated thing to measure; there's a lot of
non-execution bottlenecks that get in the way. But both Ola and Nick
Sieger (also of JRuby team) have found that JRuby on Rails performance
is either very near or exceeding Ruby 1.8.x on Rails:

http://ola-bini.blogspot.com/2007/10/current-state-of-regular-expressions.html
(toward the bottom of the post are updated Rails numbers)

http://blog.nicksieger.com/articles/2007/10/25/jruby-on-rails-fast-enough

Rails is probably the ultimate real-world benchmark, and we're excited
to see that JRuby's starting to pass Ruby 1.8.x performance here as well.

I won't drag this on any longer, but I'd encourage you to try out your
own Ruby code and let us know how it performs. We've spent a lot of time
on performance issues for the 1.1 release, and have many future plans to
continue increasing performance. We want your input.

Many people believed we'd never be faster than the C implementation, and
many still think we're slower. Now that I've set that record straight,
any questions?

- Charlie

 

[Paul Stickney]

Will JRuby support continuations (via callcc)?

I've been reading about Actors in Scala recently and the technical
documentation mentioned problems with trying to implement
continuations on the JVM due to lack of stack manipulation.

 

[Charles Oliver Nutter]

Will JRuby support continuations (via callcc)?

I've been reading about Actors in Scala recently and the technical
documentation mentioned problems with trying to implement
continuations on the JVM due to lack of stack manipulation.

Neither JRuby nor XRuby nor IronRuby nor Ruby.NET will support
continuations in the near future. However, there is research happening
now to support continuations on the JVM...if that takes, we'll piggy
back off that support.

- Charlie

 

[znmeb]

Many people believed we'd never be faster than the C implementation,
and many still think we're slower. Now that I've set that record
straight, any questions?

1. How long will it be before Alioth has some *reasonable* numbers for
jRuby? As of yesterday, they still have you significantly slower than
MRI. So I need to take jRuby out of my slides for RubyConf ... I

2. I haven't actually been benchmarking jRuby recently ... too busy
profiling MRI.

3. I'm using the "Pet Store" benchmark for Rails. It's as close to a
"real Rails benchmark" as anything I can find.

4. How's your performance on a SPARC relative to MRI?

 

[Charles Oliver Nutter]

1. How long will it be before Alioth has some *reasonable* numbers for
jRuby? As of yesterday, they still have you significantly slower than
MRI. So I need to take jRuby out of my slides for RubyConf ... I

The current published Alioth numbers are based on JRuby 1.0(ish), which
was generally 2-3x slower than MRI. I'm hoping the numbers will be
updated soon after the 1.1 releases...but it probably won't happen until
1.1 final comes out in December. If someone else wants to re-run them
for us, it would make us very happy

2. I haven't actually been benchmarking jRuby recently ... too busy
profiling MRI.

Well, we know there's still a lot we can do for JRuby
performance...including not only the 1.9 benchmarks, but a number of
additional ideas we have that won't affect 1.8 compatibility.

3. I'm using the "Pet Store" benchmark for Rails. It's as close to a
"real Rails benchmark" as anything I can find.

Yeah, that's what Ola's been using, and we're *real close* to matching
MRI on that one right now. Still hunting for bottlenecks.

4. How's your performance on a SPARC relative to MRI?

Haven't given it a try...I don't have access to a good sparc box at the
moment (and yes, I know...I work at Sun, so I can have a sparc by
snapping my fingers...it's a matter of having time to monkey with it).

- Charlie

 

[William James]

Neither JRuby nor XRuby nor IronRuby nor Ruby.NET will support

"Neither ... nor ..." is for two alternatives. Perhaps this will
suffice.

None of JRuby or XRuby or IronRuby or Ruby.NET will support

 

[vjoel]

"Neither ... nor ..." is for two alternatives. Perhaps this will
suffice.

None of JRuby or XRuby or IronRuby or Ruby.NET will support

None of snow or rain or heat or gloom of night stays these couriers from
the swift completion of their appointed rounds.

 

[charles.nutter]

"Neither ... nor ..." is for two alternatives. Perhaps this will
suffice.

None of JRuby or XRuby or IronRuby or Ruby.NET will support

Perhaps

JRuby, XRuby, IronRuby, Ruby.NET; none of them will support

Too poetic?

- Charlie

 

[znmeb]

The current published Alioth numbers are based on JRuby 1.0(ish), which
was generally 2-3x slower than MRI. I'm hoping the numbers will be
updated soon after the 1.1 releases...but it probably won't happen
until 1.1 final comes out in December. If someone else wants to re-run
them for us, it would make us very happy

I can't help you with that, but if you'll give me the numbers you do
have (time in seconds for each benchmark on the latest jRuby, MRI and
KRI on a common platform) I'll run the boxplots off and post them to
the list.

 

[mfp]

Perhaps

JRuby, XRuby, IronRuby, Ruby.NET; none of them will support

Too poetic?

The original polysyndeton conveyed the intended meaning effectively IMO.
How thin is the line between a deliberate figure of speech and a grammatical
accident!

This doesn't sound too bad to me, but I'm no native speaker:
"Neither JRuby nor XRuby, IronRuby or Ruby.NET will ..."

"Or" must bind more tightly than "nor" there in order for the corresponding
logical proposition to be correct:

AND(NOT(JRuby), NOT(OR(XRuby, IronRuby, Ruby.NET)))

"None of JRuby, XRuby, IronRuby or Ruby.NET will ..." and
"As for JRuby, XRuby, IronRuby and Ruby.NET, none [of them] will ..."
seem OK too.

 

[vjoel]

The original polysyndeton conveyed the intended meaning effectively IMO.
How thin is the line between a deliberate figure of speech and a grammatical
accident!

I agree, and thanks for teaching me a new word.

This doesn't sound too bad to me, but I'm no native speaker:
"Neither JRuby nor XRuby, IronRuby or Ruby.NET will ..."

That could sound like the "Neither ... nor ..." clause modifies
IronRuby. As in:

Neither fish nor fowl, greens or libertarians will be the first choice
of voters who are repelled by two-party politics.

It might not even be grammatical, but it is clear that we are not
talking about electing ducks and sturgeons. I'd rewrite it anyway.

"Or" must bind more tightly than "nor" there in order for the corresponding
logical proposition to be correct:

I don't think it works that way, at least as I hear it.

 

[mfp]

That could sound like the "Neither ... nor ..." clause modifies
IronRuby. As in:

Neither fish nor fowl, greens or libertarians will be the first choice
of voters who are repelled by two-party politics.

It might not even be grammatical, but it is clear that we are not
talking about electing ducks and sturgeons. I'd rewrite it anyway.

Yet another effect is achieved by reordering the phrase this way:
Neither fish nor greens, fowl or libertarians
This being a play on the standard idiom [1], the two parts are connected,
yielding a new meaning.

Is any sufficiently complex figure of speech undistinguishable from nonsense?

I don't think it works that way, at least as I hear it.

I had to repeat it a few times before I convinced myself that this is how
things should work, but the interpretation as a juxtaposition is very
attractive. Maybe I'm drawn to it because my mother tongue is closer to Latin?

A skilled writer could use the ambiguity for very effective double-entendres.

[1] I had to look it up. In Spanish, French and German the idiom is "neither
meat nor fish" (actually, "neither fish nor meat" in German; I wonder why).

 

[caduceass]

I had to repeat it a few times before I convinced myself that this is how
things should work, but the interpretation as a juxtaposition is very
attractive. Maybe I'm drawn to it because my mother tongue is closer to Latin?

A skilled writer could use the ambiguity for very effective double-entendres.

[1] I had to look it up. In Spanish, French and German the idiom is "neither
meat nor fish" (actually, "neither fish nor meat" in German; I wonder why).

The switching of nouns may have more to do with culture than language roots.

I couldn't resist an attempt at lolcat...

Ther be JRuby, XRuby, IronRuby, Ruby.NET, and no s'port continue stufs now.

Latin rooted languages allow a lot of latitude, certainly, and I think
if they lost that latitude, they would make a lot of poets bored, as
you suggested.

I could have used "but" instead of "and" in that last sentence and
still convey the same meaning, but it might have changed the intent if
your vernacular idioms differ from mine.

Todd

 

[rogerpack2005]

Long story short, JRuby 1.1 is now generally faster than Ruby 1.8.6, and
for most execution benchmarks it is the fastest 1.8-compatible Ruby
implementation available.

Congrats on such an important benchmark! What made the difference 1.0
to 1.1?
-Roger

 

[charles.nutter]

Congrats on such an important benchmark! What made the difference 1.0
to 1.1?

Mostly the compiler, but there's been a number of optimizations across
the whole runtime and most of the core classes.

- Charlie

 

[gga]

Many people believed we'd never be faster than the C implementation, and
many still think we're slower.

Isn't that still a correct impression and statement? Currently the C
implementation to benchmark against is 1.9, which is soon to become
the official release (early next year, if all goes to plan). 1.8 is
mostly 3 years-old code now or so.
Or is JRuby now faster than 1.9?

Now that I've set that record straight, any questions?

I do have one, yes. How much work does the JRuby team place in
actually improving Sun's JVM?

Recently there was a question of actually turning off ObjectSpace for
JRuby as a default, instead of actually taking the JVM and enhancing
it to support some sort of ObjectSpace, which seemed more like the
right answer to me.

 

[charles.nutter]

Isn't that still a correct impression and statement? Currently the C
implementation to benchmark against is 1.9, which is soon to become
the official release (early next year, if all goes to plan). 1.8 is
mostly 3 years-old code now or so.
Or is JRuby now faster than 1.9?

Ruby 1.9 is not likely to be the standard, widely-used version of Ruby
for some time. Matz estimated it probably wouldn't be the case until
late 2008. He also mentioned at the conference that while 1.9.1 might be
feature-stable, it's still a bit experimental and will probably not be a
production-ready VM for a while.

In my opinion, no, it's not really a direct comparison to put JRuby up
against Ruby 1.9 because 1.9 contains several compatibility-breaking
changes. Many of those changes enable faster performance at the cost of
backward compatibility. JRuby is an implementation of Ruby 1.8, and
under that definition, it's appropriate to compare it to other
implementations of Ruby 1.8. Some example of compatibility-breaking
changes in Ruby 1.9:

- case/when with literals will be constant time, but not invoke ===
- Fixnum math will be performed as direct operations, rather than
dispatch through Fixnum#+, Fixnum#-, and so on.
- block arguments can only be local variables

And so on. We plan to make all the same optimizations, but also make
them individually configurable, so that if you want to have fast integer
math but everything else 1.8 semantics, you'll be able to. Or if you
want to turn on 1.9 features as a whole, you can.

Since you ask, yes, there's a few benchmarks where we're already faster
than 1.9:

JRuby:
1m loops yielding three fixnums 10 times to block splatting and
accessing them
3.225000 0.000000 3.225000 ( 3.225000)
3.189000 0.000000 3.189000 ( 3.189000)
3.170000 0.000000 3.170000 ( 3.170000)
3.152000 0.000000 3.152000 ( 3.151000)
3.322000 0.000000 3.322000 ( 3.322000)

1.9:
1m loops yielding three fixnums 10 times to block splatting and
accessing them
4.940000 0.010000 4.950000 ( 4.973445)
4.950000 0.020000 4.970000 ( 4.971358)
4.950000 0.010000 4.960000 ( 4.984771)
4.950000 0.010000 4.960000 ( 4.980683)
4.950000 0.010000 4.960000 ( 4.973008)

~/NetBeansProjects/jruby $ jruby -J-server
test/bench/bench_fib_iterative.rb
11.610000 0.000000 11.610000 ( 11.611000)
11.481000 0.000000 11.481000 ( 11.481000)
~/NetBeansProjects/jruby $ ../ruby1.9/ruby -I ../ruby1.9/lib
test/bench/bench_fib_iterative.rb
13.110000 4.700000 17.810000 ( 17.928527)
12.930000 4.770000 17.700000 ( 17.818631)

1.9's eval performance is also considerably degraded due to the cost of
compilation...in some cases even slower than Ruby 1.8.

I do have one, yes. How much work does the JRuby team place in
actually improving Sun's JVM?

Recently there was a question of actually turning off ObjectSpace for
JRuby as a default, instead of actually taking the JVM and enhancing
it to support some sort of ObjectSpace, which seemed more like the
right answer to me.

The problems with ObjectSpace are not just problems you can flip a
switch and make go away; they're systemic issues that will affect any
modern GC. Look at it this way:

In Ruby, where Objects do not move around in memory, it's easy to just
walk through the entire set of objects in memory and return them in
turn. There's no parallel threads to create more garbage, and there's no
concurrent garbage collector moving objects around and compacting the
heap.

On the JVM, however, it's an entirely different situation. Not only are
threads actually running in parallel, creating their own garbage...the
GC itself runs in parallel on many versions of Java; so you'd not only
have to stop all other threads from running, you'd have to stop the
garbage collector itself.

The other answer here is that...you already can do these things; they're
used for profiling and debugging modes of the JVM. And oddly enough, the
most common applications of ObjectSpace.each_object are for runtime
profiling and debugging. The difference in the JVM (and in JRuby 1.1) is
that we're not going to saddle users with the performance hit of what's
typically a profiling/debugging feature that they'll never use on a
production server.

(and of course we're very keen on improving the JVM...but there's only
so many hours in a day)

- Charlie