question on bottleneck of ruby

[Roger Pack]

Question: if I made the statement "Ruby is slower than some other
interpreted languages" almost anyone could agree with it (slower
computationally, not programmer time-wise).
So my question is...where is the bottleneck? Is it in method lookup?
Is it that method lookup is more frequent, since everything is an
object? Is it in parsing? Storage? Just wondering. Thanks!
-Roger

 

[khaines]

Question: if I made the statement "Ruby is slower than some other
interpreted languages" almost anyone could agree with it (slower
computationally, not programmer time-wise).
So my question is...where is the bottleneck? Is it in method lookup?
Is it that method lookup is more frequent, since everything is an
object? Is it in parsing? Storage? Just wondering. Thanks!

The dynamic nature of the method dispatch. The overall implementation --
both the tree walker versus a vm, and some of the implementation details.
The GC can also be a bottle neck. It stops the world when it runs, and it
has to touch every object when it runs, so as the program gets bigger (in
object count), the GC burden gets higher.


Kirk Haines

 

[Roger Pack]

Thanks Kirk.

Follow-up question: it seems like there have been a lot of 'replacement
attempts' of the Ruby interpreter, but none of them seem to have really
really sped it up (jruby, rubinius, etc.) They do add cool features like
thread safeness or what not, and in certain instances are faster, but
not like 'we've overcome the bottleneck!' faster.
Does this mean that they suffer from the same bottlenecks? Is Ruby just
a hard language to optimize for? Do they all tree walk and have slow
method dispatch and a slow GC or something? Hmm.

Stay well.
-Roger

 

[Richard Conroy]

Thanks Kirk.

Follow-up question: it seems like there have been a lot of 'replacement
attempts' of the Ruby interpreter, but none of them seem to have really
really sped it up (jruby, rubinius, etc.)

I don't think that the developers of the alternatives would consider themselves
as 'replacements' or 'competitors'. With the possible exception of Rubinius.
But that's its intent - to be the replacement MRI for Ruby 2.0; in the meantime
YARV (1.9) is the Ruby Interpreter based on the current code base. There have
been a number of synthetic benchmarks that show interesting performance
improvements in YARV currently. I am not sure how it stacks up in real
world use though. (Not being critical here, real-world benchmarks are
notoriously
difficult to do right).

Does this mean that they suffer from the same bottlenecks? Is Ruby just
a hard language to optimize for? Do they all tree walk and have slow
method dispatch and a slow GC or something? Hmm.

You should definitely keep up to date with the JRuby guy's blogs (Ola Bini
and Charles Oliver Nutter). They have been quite detailed on this topic
over the last few months. While their solutions for performance are to make
JRuby very compatible with the JVM, they articulate the particular reasons
for Ruby's weak performance.

Basically, Ruby is hard to optimise for. It's super dynamic nature (Open
Classes) mean that traditional compiler/interpreter tricks cannot be used.

 

[Markus Schirp]

I don't think that the developers of the alternatives would consider
themselves as 'replacements' or 'competitors'. With the possible
exception of Rubinius. But that's its intent - to be the replacement
MRI for Ruby 2.0; in the meantime YARV (1.9) is the Ruby Interpreter
based on the current code base. There have been a number of synthetic
benchmarks that show interesting performance improvements in YARV
currently. I am not sure how it stacks up in real world use though.
(Not being critical here, real-world benchmarks are notoriously
difficult to do right).


You should definitely keep up to date with the JRuby guy's blogs (Ola
Bini and Charles Oliver Nutter). They have been quite detailed on
this topic over the last few months. While their solutions for
performance are to make JRuby very compatible with the JVM, they
articulate the particular reasons for Ruby's weak performance.

Basically, Ruby is hard to optimise for. It's super dynamic nature
(Open Classes) mean that traditional compiler/interpreter tricks
cannot be used.

So why not add a "class freeze method", so that these traditional
compiler/interpreter tricks can be used?

During development you just not activate this "frozen class state", for
releases you can freeze some "performance penalty" classes.
So we not loose ruby's super-dynamic character (during development),
and got the benefits from more agressive optimisations.

 

[Richard Conroy]

So why not add a "class freeze method", so that these traditional
compiler/interpreter tricks can be used?

IIRC you can set how open your classes are at runtime. Though
we *like* the fact that Ruby classes are open at runtime. Its not
a development feature by any means - Rails makes huge use of
this.

Interpreter writers know this and design accordingly. So they use
non-traditional tricks instead.

Mostly I just don't see Ruby performance as a significant issue.
Memory management and GC would be higher on my priority list,
and are arguably easier to solve anyway.

JRuby now adds another option for dropping down to system
language performance for specific cases. .NET users will get
similar treatment eventually, and yet another system language
option will be added to the mix (and .NET users will get a
proper scripting option on the CLR as well ;-P )

The only area where I have noticed very poor Ruby performance
is in REXML. Though REXML is very much an entry-level option,
and the authors themselves would redirect you to alternatives if
your needs are greater. My observations were probably multiplied
by my absolutely rubbish XML skills too....

 

[Charles Oliver Nutter]

I agree with Kirk. Ruby's inherent performance bottleneck is also its most
distinctive feature: the open classes and data structures. There's always
going to be a limit to how much faster you can do name-resolution at
runtime, since it's costly to begin with, and you can't deterministically
predict it. (There are approaches to adaptively predict name-bindings at
runtime, notably from Smalltalk, and I expect JRuby will be able to leverage
them. But they can ameliorate, not solve, the fundamental issue.)

The fact that classes are open doesn't really impact Ruby performance
all that much. Ultimately the largest reasons for poor performance are
the fact that methods, instance variables, and constants can't be bound
during a compile phase and must be looked up at runtime. But the same
techniques used to speed up other dynamic languages can apply equally
well here.

Garbage collection and poor memory usage, on the other hand, are problems
that can and should be solved.

Very true, and I hope these are solved some time in future 1.9.1
releases, because I worry that with YARV running faster and generating
garbage faster, the GC bottlenecks are going to become more pronounced.

- Charlie

 

[Clifford Heath]

The fact that classes are open doesn't really impact Ruby performance
all that much.

That's true for single-process benchmarks where you don't implement
sharing of compiled modules. In a compiler-interpreter I once worked
on, we serialised the bytecode to a memory-mappable file for each
cluster of modules, which did a lot to improve startup performance,
especially for subsequent instances. Not possible for Ruby however,
doe to the open classes - QED.

The other performance factor (related to a different discussion) that
makes byte-code interpretation faster than AST-interpretation is that
with byte-code, you get much better locality of reference, so your
cache+memory system works much better. This is a *very* significant
factor that justifies some recent complaints. It's also quite possible
for Ruby interpreters to implement, despite open classes.

Clifford Heath.

 

[M. Edward (Ed) Borasky]

I agree with Kirk. Ruby's inherent performance bottleneck is also its most
distinctive feature: the open classes and data structures. There's always
going to be a limit to how much faster you can do name-resolution at
runtime, since it's costly to begin with, and you can't deterministically
predict it. (There are approaches to adaptively predict name-bindings at
runtime, notably from Smalltalk, and I expect JRuby will be able to leverage
them. But they can ameliorate, not solve, the fundamental issue.)

Which is why they teach data structures in computer science class. It's
all about fast search, I think. That's one of the big gripes I have with
"lazy" interpretation. If you don't do stuff until you have to do it, it
only pays off if you end up *never* having to do it.

[snip]

I used to believe that large Ruby programs could be constructed as
independent coarse-grained modules running in their own processes,
communicating through message-passing. Having done quite a bit of this now,
I think the approach helps quite a bit, but still suffers from memory usage.
It's hard to get the memory-usage of even a small Ruby program down to a
level that would permit (perhaps) hundreds of cooperative processes to be
running at once.

And unless the Ruby "inner interpreter" is highly optimized, even if you
have only one copy of the text segment and only one copy of all the
libraries in RAM, you're *really* unlikely to have all the "good stuff"
in the tiny caches processors have.

Bottom line: using Ruby will always be characterized by a tradeoff between
performance and programmer productivity. This is not a criticism of Ruby in
any way, shape or form! Productivity is a fundamental engineering value, and
time-to-market is a fundamental quality dimension. Ruby therefore has, and
will continue to have, a unique value proposition.

I'm not sure this is a valid tradeoff. The economics of *development*
and the economics of *operating* a large code are two entirely different
subjects. People have "always" prototyped in "slow but productive"
languages, like Lisp, Perl, PHP and Ruby, and then reached a point where
the economics dictated a complete rewrite for speed into C, C++ or Java.
I can think of more examples of this than I can of something that was
developed and prototyped rapidly and then grew by "just throwing more
hardware at inefficient software."

So ... just like a startup should plan for the day when a big company
offers them the choice of selling out or being crushed like a bug, when
you implement a great idea in some rapid prototyping framework like
Rails, plan for the day when you are offered the choice of rewriting
completely in a compiled language or going bankrupt buying hardware.

 

[Charles Oliver Nutter]

That's true for single-process benchmarks where you don't implement
sharing of compiled modules. In a compiler-interpreter I once worked
on, we serialised the bytecode to a memory-mappable file for each
cluster of modules, which did a lot to improve startup performance,
especially for subsequent instances. Not possible for Ruby however,
doe to the open classes - QED.

Startup time does not general performance make. Just ask Java

The other performance factor (related to a different discussion) that
makes byte-code interpretation faster than AST-interpretation is that
with byte-code, you get much better locality of reference, so your
cache+memory system works much better. This is a *very* significant
factor that justifies some recent complaints. It's also quite possible
for Ruby interpreters to implement, despite open classes.

JRuby has successfully implemented a full Ruby 1.8 to Java bytecode
compiler (as of about ten minutes ago), so it is indeed possible...and
nicely fast.

- Charlie

 

[M. Edward (Ed) Borasky]

I'm not sure this is a valid tradeoff. The economics of *development*
and the economics of *operating* a large code are two entirely different
subjects. People have "always" prototyped in "slow but productive"
languages, like Lisp, Perl, PHP and Ruby, and then reached a point where
the economics dictated a complete rewrite for speed into C, C++ or Java.
I can think of more examples of this than I can of something that was
developed and prototyped rapidly and then grew by "just throwing more
hardware at inefficient software."

So ... just like a startup should plan for the day when a big company
offers them the choice of selling out or being crushed like a bug, when
you implement a great idea in some rapid prototyping framework like
Rails, plan for the day when you are offered the choice of rewriting
completely in a compiled language or going bankrupt buying hardware.

OK ... so ... is Twitter in trouble?

http://www.scripting.com/stories/2007/09/27/twitterIsTakingAShowerToni.html

"I have seen the future, and it's just like the present, only longer."
-- Kehlog Albran

 

[Clifford Heath]

Startup time does not general performance make. Just ask Java

True, but lack of memory pressure enables you to keep the whole
working set in memory, and that does affect runtime.

Plus there is a large class of applications for which startup time
dominates, otherwise we'd be writing /bin/cat in Java .

Clifford Heath.

 

[Byung-Hee HWANG]

Which is why they teach data structures in computer science class. It's
all about fast search, I think. That's one of the big gripes I have with
"lazy" interpretation. If you don't do stuff until you have to do it, it
only pays off if you end up *never* having to do it.

[...snip...]
I've found in my businesses that such fleeting opportunities come up all the
time. If you're a person who believes in proper software engineering, and
well-controlled processes, you're probably climbing the walls and getting
ready to throw things at me right now! But this is why I was talking about a
value-tradeoff. If I'm right, then there are a lot of opportunities to
create capturable business value that traditional methodologies (including
fast-prototype-followed-by-extensive-rewrite) simply can't touch.

For these cases, Ruby is uniquely valuable.

It seems like you are angry. I can feel that you like Ruby very much.

[...snip...]

 

[Robert Dober]

In some respects, guilty as charged. (I deny the point about
non-understandability. If you're going to develop like this, then writing
documentation and unit tests *must* dominate the development effort, perhaps
by 10-to-1. Otherwise, you end up with nothing usable.)

This however (although 10 might be an exaggeration) is a good thing
more often than not.

Robert

 

[Chad Perrin]

Plus there is a large class of applications for which startup time
dominates, otherwise we'd be writing /bin/cat in Java .

I second the motion.

For most of what I write in Ruby, startup time is *much* more important
than long-running performance. I'd prefer to avoid having to go back to
Perl for some of that, to avoid extremely long waits (for some definition
of "extremely long") for basic sysadmin utilities.

 

[Chad Perrin]

Which is why they teach data structures in computer science class. It's
all about fast search, I think. That's one of the big gripes I have with
"lazy" interpretation. If you don't do stuff until you have to do it, it
only pays off if you end up *never* having to do it.

[...snip...]
I've found in my businesses that such fleeting opportunities come up all the
time. If you're a person who believes in proper software engineering, and
well-controlled processes, you're probably climbing the walls and getting
ready to throw things at me right now! But this is why I was talking about a
value-tradeoff. If I'm right, then there are a lot of opportunities to
create capturable business value that traditional methodologies (including
fast-prototype-followed-by-extensive-rewrite) simply can't touch.

For these cases, Ruby is uniquely valuable.

It seems like you are angry. I can feel that you like Ruby very much.

[...snip...]

I rather strongly suspect that you are not a psychologist -- and that,
even if you were, you would not be trying to diagnose people over the
Internet. Please stick to the discussion topic rather than attempting to
assign motivations and emotions to others involved in the discussion as
an alternative to making a salient point.

 

[Charles Oliver Nutter]

I second the motion.

For most of what I write in Ruby, startup time is *much* more important
than long-running performance. I'd prefer to avoid having to go back to
Perl for some of that, to avoid extremely long waits (for some definition
of "extremely long") for basic sysadmin utilities.

I third it. I don't like the slow startup time of JRuby any more than
you all would. We'll do what we can to fix that.

- Charlie

 

[Chad Perrin]

Which is why they teach data structures in computer science class. It's
all about fast search, I think. That's one of the big gripes I have with
"lazy" interpretation. If you don't do stuff until you have to do it, it
only pays off if you end up *never* having to do it.

My understanding is that lazy evaluation can actually be of distinct
benefit to simplifying attempts to code for concurrency. I haven't
really investigated the matter personally, having little call to write
software that would benefit from concurrency, but I imagine that will
change in time. That being the case, I will surely enjoy the benefits of
lazy evaluation at that time, should my understanding of its benefits to
concurrency not prove to be based on faulty information.

I'm not sure this is a valid tradeoff. The economics of *development*
and the economics of *operating* a large code are two entirely different
subjects. People have "always" prototyped in "slow but productive"
languages, like Lisp, Perl, PHP and Ruby, and then reached a point where
the economics dictated a complete rewrite for speed into C, C++ or Java.
I can think of more examples of this than I can of something that was
developed and prototyped rapidly and then grew by "just throwing more
hardware at inefficient software."

So ... just like a startup should plan for the day when a big company
offers them the choice of selling out or being crushed like a bug, when
you implement a great idea in some rapid prototyping framework like
Rails, plan for the day when you are offered the choice of rewriting
completely in a compiled language or going bankrupt buying hardware.

There are a great many use cases for Ruby where there will *never* come a
time that runtime performance is that important. Probably 80% of the
code I write, minimum, falls into that category. Under such
circumstances, a reasonably quick startup time and a decent algorithm
make much more of a difference than long-running performance and a binary
or bytecode compiled language with a reputation for performance.

That doesn't mean I wouldn't like to see Ruby's performance improved
significantly in the future. It just means that if Ruby never approaches
the performance of C, or the long-running performance characteristics of
an optimizing VM like Java's, it will in no way hamper my ability to put
Ruby to good use without having to plan for the day when I have to
rewrite everything -- because that day will never come in at least the
vast majority of cases.

In fact, in cases where rapid coding up front in a way that requires a
high level language is very important, and high performance software will
become very important given time, my preference would not be to prototype
in Ruby (or Perl, or UCBLogo, or whatever) anyway. It'd be to use
something like OCaml, with excellent performance characteristics in
binary compiled form, decent long-running performance in bytecode
compiled form running on its VM, and convenient source code access using
the interpreter with the ability to test stuff on the fly using its
"toplevel" interactive interpreter. Use a tool to suit the job at hand.

A lot of the time, in my work and play, that tool is Ruby -- and will
never require a rewrite in a "faster" language.

 

[Rick DeNatale]

The other performance factor (related to a different discussion) that
makes byte-code interpretation faster than AST-interpretation is that
with byte-code, you get much better locality of reference, so your
cache+memory system works much better. This is a *very* significant
factor that justifies some recent complaints.

Depending on circumstances byte-codes can actually be faster than machine code.

An example.

Many years ago, Digitalk produced a Smalltalk implementation for PCs.
They continually had to answer questions about the performance of
byte-code interpretation. When they came out with a version 32-bit
PCs they decided to expand everything to x86 machine code, so that
they could say that their Smaltalk was compiled instead of
interpreted.

What they learned was that the machine code version actually ran
slower due to locality of reference and paging.

Of course tuning the performance of anything, and in particular a
dynamic language implementation, is as much of an art as a science,
and requires constant experimentation and willingness to overcome
one's assumptions.