Ruby and E.V.E. Paradox

[GD]

I have been using Ruby for several years. Most importantly I have been using
this amazing language as a support language for E.V.E. Paradox, a game suite I
have been working on for much of the last three years. Ruby is used extensively
in the tools used to build and test E.V.E. Paradox. It has literally saved me
hundreds of hours. I use Ruby to generate many thousands of lines of C++ source
from configuration files, automate builds, automatically fix problems with other
software, generate dynamic web content, manage data, test, maintain the
codebase, and so forth. It is reliable, easy to use, intuitive, and a fantastic
base for rapid development of tools. To everyone involved in the development of
this amazing language, thankyou.

Recently I have upgraded my development machine and with it, all of the support
libraries I use with my software. I presently use LUA for embedded scripting in
E.V.E. Paradox. After the upgrade I found that several key API calls had been
renamed (for no reason AFAICT), and the existing scripts, even when fixed, no
longer works. I have found LUA generally difficult to work with, and only use
the basics of the language anyway. The documentation is less than ideal. I
started entertaining the idea that it would be very, *very* nice if I could use
Ruby as my scripting language instead; basically Ruby is solid and powerful, and
I have years of experience with it.

This is what I needed to replace LUA:
- Ability to load a script *from*memory*, not from a file (important).
- Run any init code in the script.
- Hook a bunch of local functions (C++) so that the script can call them for
information from the main code body.
- Call functions in the script both periodically and in response to events (eg.
an update call every 1/60 of a second, in response to mouse clicks, etc).
- Clean up everything entirely when the level is over.
- Be prepared to run everything again later when the next level needing
scripting is loaded.
- Ideally have separate contexts so that two separate games running at once (in
the same exe) can work in their own scripting environment.

I noticed that Ruby is built with many functions available in a separate dynamic
library, which builds easily both under Linux and through MinGW, and has some
kind of embeddable interface. I figured there was a chance I could get things
going, so I gave myself two days to figure things out and get the basics running.

The process has been a complete nightmare. I have been plagued by constant
crashes and uncertain documentation. I have been at this for close to two full
days. I am using ruby-1.8.5-p12 built from source, LD_LIBRARY_PATH is fine, and
other instances of Ruby have been completely removed from the machine.

This is what I have encountered:

- There doesn't appear to be any way to indicate that Ruby should clean
everything up and get to a state where I can start again. I have attempted to
work around this by making each script contain a single unique class that I use,
but this is a terrible hack.
- How do I load a script from memory? rb_eval_string()? rb_compile_string()?
Something else?
- The documentation is flawed. It indicates calling rb_run(), but that can
eventually call exit, which is the last thing you want from an embedded
language. It mentions rb_apply() as having an argc parameter.
- I am getting inexplicable crashes when calling the garbage collector. Fearing
it was my mistake, I eliminated calls repeatedly until I got down to ruby_init()
and two calls to rb_gc(). This crashed. I have searched around and noticed
people with comparable problems, and did the same thing. I have seen
condescending replies with no indication as to why a simple code sequence such
as this could cause a crash. Not calling the garbage collector is NOT a
solution, as it could be called outside of my control and I don't want my code
randomly crashing when it does.
- I managed to mitigate some crashes by moving the Ruby initialisation from
inside a function to near the code initialisation such that the stack present on
creation will be in place by the time other calls are made. This made a
difference! Is the ruby interpreter performing some operations on the real stack
that I don't know about? Shouldn't something like this be mentioned somewhere?-
- I wrapped rb_gc() in a rb_protect() call, didn't help.
- The documentation is not at all clear how you protect something from the
garbage collector, or indicate it is no longer in use.
- How does one call "Foo.new(args)" on a class from outside of Ruby, where the
args have been generated from outside of Ruby (hence no name) meaning
rv_eval_string is unavailable, and I'm not sure how rb_funcall()/rb_apply() can
be used to call "new" on a class to get an object?
- rb_protect() is a complete mess. Rather than being able to call an arbitrary
function with a void pointer, you are stuck calling something that takes a
single VALUE arg. This means that if you want to call a general function, you
need to resort to tricks like building up a ruby array with the receiver, method
string, and real args (also in their own array), pass it through, and
deconstruct it on the other end, feeding it into rb_apply() rather than using
the more convenient rb_funcall() calls. Writing multiple dedicated callbacks is
not an option, as I lose the local context (eg. "this") in C++ when I do that,
unless I go to the extra effort to wrap it. Is there some reason it is like this?
- The code runs fine for a while if I don't call the garbage collector.
Unfortunately, I get stack depth errors and eventual segfaults if I make
repeated calls to the Ruby script (via rb_apply()). Since I need to call an
"update" function sixty times a second, minimising my call use is not an option.
- When searching for errors that come up during development (eg. stack size
problems, crashes on garbage collection) I get a huge number of hits. I am not
alone here, and it seems people are having genuine trouble finding a solution.
- I can't find one example that completely and correctly demonstrates the whole
process involved in having C call Ruby then call C, return values through the
chain neatly, and clean up. If there was such a reference, it would be outstanding.
- There seem to be external efforts to clean things up, but they aren't making
it back into the official code. Is the embedded interface not really supposed to
be used for anything but extensions to Ruby?

I wonder if Ruby is at all suitable as a general-purpose embedded scripting
language? I am beginning to think that it is not.

The embedded interface and documentation are terrible! I don't mean any offense,
and I maintain my stance that Ruby is an outstanding language. I love it. And I
do realise that sometimes code takes some time to mature- I've been using Ruby
since the 90's and it wasn't always a smooth ride. Is this the problem, it just
isn't quite ready for embedding yet? Or I'm not using the right documentation?
Or I'm asking too much? Am I doing the wrong thing?

It probably seems odd for someone with such a high opinion of the language to
have such a low opinion of the embedded interface. Perhaps it is because Ruby
has set my expectations so high that I am that much more easily disappointed.

Some projects seem to have managed it, and they may work, to some degree. But
their use of it seems to be fairly basic, rather than counting on it as a core
component of the program. VIM has a nice example, although it doesn't seem to
worry about garbage collection or cleaning up. As such it doesn't seem complete.

Am I alone in thinking this? Should I be looking at other embeddable languages
(Python, for example), or just go back and get the LUA code working again? Am I
doing something wrong?

Is there anything I can do to successfully and reliably add support for my
favourite scripting language into my project? Or is it simply an unrealistic
prospect?

Garth

 

[Devin Mullins]

- Ideally have separate contexts so that two separate games running at
once (in the same exe) can work in their own scripting environment.

AFAIK, this is going to be your difficult one. Others with actual
knowledge can comment on why, and on workarounds.

Other than that, I'm clueless, and WOW that was a long post.

Devin

 

[GD]

Hey Devin,

AFAIK, this is going to be your difficult one. Others with actual
knowledge can comment on why, and on workarounds.

Looking at the Ruby source, I can see this being difficult. There
are also workarounds on the E.V.E. Paradox side (script merging,
limiting access to one scripted level at a time etc) if needed.
Hence "ideally" rather than "realistically". The other stuff
is pretty critical. Not crashing is also nice!

I've considered forking and using pipe IPC to solve the problem,
but I'd lose the tight integration and also have to learn a lot
more about Windows IPC. I might look into this if the current
approach is infeasible.

Other than that, I'm clueless, and WOW that was a long post.

Indeed, and let it not be said that I didn't give it a darn good
shot. Would you believe that I actually left out a lot of
detail in my post?

Garth

 

[gga]

GD ha escrito:

This is what I needed to replace LUA:
- Ability to load a script *from*memory*, not from a file (important).

Use rb_eval_string_protect().

- Run any init code in the script.

BEGIN {} calls or rb_eval_string_protect() of whatever you need.

- Hook a bunch of local functions (C++) so that the script can call them for
information from the main code body.

Ideally, handle this using SWIG, as either a ruby extension or a pseudo
extension you link in statically.
Avoid doing the binding yourself manually. Binding functions for most
basic C constructs (ints, chars, etc) is VERY easy.
Doing proper binding of classes, it is not as easy, as the GC requires
you to be careful. Best thing for you to do is look at SWIG's output.
SWIG also deals properly now with inheritance of C++ classes into ruby,
including multiple inheritance.

- Call functions in the script both periodically and in response to events (eg.
an update call every 1/60 of a second, in response to mouse clicks, etc).

This may be much trickier with the current ruby C interpreter,
depending on what you are looking for. You can start a C++ thread to
do it, but the ruby C interpreter is not thread safe. So, if you have
ruby code running in two different threads at the same time, bad things
WILL happen.
You can also try to rely on a single thread and let ruby's own light
threads handle this. However, this may not be as efficient as actual
multi-threaded code.
Mind you, ruby is not the only language with issues here. Python is
also pretty bad at this.
LUA and TCL are in a class of their own as far as multithreading
interpreters go.

- Clean up everything entirely when the level is over.


This is what I have encountered:

- There doesn't appear to be any way to indicate that Ruby should clean
everything up and get to a state where I can start again. I have attempted to
work around this by making each script contain a single unique class that I use,
but this is a terrible hack.

ruby_cleanup(0); SHOULD clean up the interpreter. Problem is not so
much cleaning the interpreter but restarting it again.
I did find issues when I did this (back in 1.8.0). I think now issues
are better, but I won't vouch for it being 100% solved.
If all fails, best thing you can do is what you did. Encapsulate all
in its own class and clean that instead. That will still leave the
class/module constants around, but it is better than nothing.

- How do I load a script from memory? rb_eval_string()? rb_compile_string()?
Something else?

Use rb_eval_string_protect().

- The documentation is flawed. It indicates calling rb_run(), but that can
eventually call exit, which is the last thing you want from an embedded
language. It mentions rb_apply() as having an argc parameter.

Again, rb_eval_string_protect() is what you probably want.

- I am getting inexplicable crashes when calling the garbage collector.

This is a probably a bug in your code. Take a look at rubyMEL at
highend3d.com for a relatively simple embedding of MEL into a 3d
application (Maya).
Depending on platform (windows, mainly) you need other initialization
code besides ruby_init().

- The documentation is not at all clear how you protect something from the
garbage collector, or indicate it is no longer in use.

You need to use a mark and a free function respectively, and use ruby's
mysterious Data_Wrap_Struct stuff. This is the pretty hairy part of
exposing a C++ api.
Suggestion: look at SWIG's code. It has a bunch of wrapper functions
to handle it nicely and it is now pretty mature.
For any questions regarding SWIG not clear from the SWIG manual, refer
to FXRuby's excellent wrapping of the Fox library.

- How does one call "Foo.new(args)" on a class from outside of Ruby, where the
args have been generated from outside of Ruby (hence no name) meaning

You call it like any other method, with rb_funcall* like:

VALUE cFoo; // defined somewhere else...
int nargs = 1;
VALUE args[] = { INT2NUM( 30 ) };
ID new_id = rb_intern("new");
rb_funcall2( cFoo, new_id, nargs, args );

- rb_protect() is a complete mess. Rather than being able to call an arbitrary
function with a void pointer, you are stuck calling something that takes a
single VALUE arg.

No, you are wrong. It is extremely well thought out.
No need for ugly void*, thank god.
Think of VALUE as being a void* (it is really an unsigned long). Ruby
requires sizeof(unsigned long) == sizeof(void*) -- see ruby.h's check.
Just do a simple casting of your pointer to VALUE in your caller and do
the opposite cast from VALUE to whatever in your receiver function.
See how all the RCAST() and similar macros work in ruby.h.

- The code runs fine for a while if I don't call the garbage collector.
Unfortunately, I get stack depth errors and eventual segfaults if I make
repeated calls to the Ruby script (via rb_apply()).

This likely means you wrapped your C++ classes incorrectly or you are
trying to run ruby in a multi-threaded environment.
If the first is true, look and learn from SWIG. If the second is true,
remove the multi-threaded code, create a global mutex that all your
ruby functions should call before doing anything or use a different
scripting language at this time.
Ruby is not thread safe, unlike LUA.

- I can't find one example that completely and correctly demonstrates the whole
process involved in having C call Ruby then call C, return values through the
chain neatly, and clean up. If there was such a reference, it would be outstanding.

There's no real trick to it.
Ruby is fine as an embedded language, just as you remember it is not
safe as a multi-threaded embedded language. YARV may eventually solve
this somewhere in the future. The unofficial fork of Ruby that is
Sydney (and now rubinius) supposedly is multi-thread safe, but is
32-bits only (and I won't vouch for it).
Python is just as bad as Ruby, btw, albeit it avoids some threading
issues by using a global lock every time two threads try to run some
code simultaneously, effectively running only a single thread at a
time. In ruby, you can do the same, but you need to create this global
mutex yourself.
LUA and TCL are different and better in this aspect, as they both can
use different (and very light) interpreter instances to execute code.

Am I alone in thinking this? Should I be looking at other embeddable languages
(Python, for example), or just go back and get the LUA code working again? Am I
doing something wrong?

Truth is... you are looking at the two best languages for embedding.
LUA is admittedly better to embed due its multi-threading aspects,
while Ruby is admittedly better for embedding due to its nice syntax.

Is there anything I can do to successfully and reliably add support for my
favourite scripting language into my project? Or is it simply an unrealistic
prospect?

Follow the advices I gave you and you should be able to get ruby as an
embedded language. Wrapping classes for the garbage collector is the
hairy part, but you can steal all of SWIG's source code for that now
(or use SWIG itself for doing it).
However, if your game engine does require heavy multi-threading in the
scripting language, you are indeed looking at the wrong language.
LUA is still the best language for this, unfortunately.

 

[gga]

GD ha escrito:

This is what I needed to replace LUA:
- Ability to load a script *from*memory*, not from a file (important).

Use rb_eval_string_protect().

- Run any init code in the script.

BEGIN {} calls or rb_eval_string_protect() of whatever you need.

- Hook a bunch of local functions (C++) so that the script can call them for
information from the main code body.

Ideally, handle this using SWIG, as either a ruby extension or a pseudo
extension you link in statically.
Avoid doing the binding yourself manually. Binding functions for most
basic C constructs (ints, chars, etc) is VERY easy.
Doing proper binding of classes, it is not as easy, as the GC requires
you to be careful. Best thing for you to do is look at SWIG's output.
SWIG also deals properly now with inheritance of C++ classes into ruby,
including multiple inheritance.

- Call functions in the script both periodically and in response to events (eg.
an update call every 1/60 of a second, in response to mouse clicks, etc).

This may be much trickier with the current ruby C interpreter,
depending on what you are looking for. You can start a C++ thread to
do it, but the ruby C interpreter is not thread safe. So, if you have
ruby code running in two different threads at the same time, bad things
WILL happen.
You can also try to rely on a single thread and let ruby's own light
threads handle this. However, this may not be as efficient as actual
multi-threaded code.
Mind you, ruby is not the only language with issues here. Python is
also pretty bad at this.
LUA and TCL are in a class of their own as far as multithreading
interpreters go.

- Clean up everything entirely when the level is over.


This is what I have encountered:

- There doesn't appear to be any way to indicate that Ruby should clean
everything up and get to a state where I can start again. I have attempted to
work around this by making each script contain a single unique class that I use,
but this is a terrible hack.

ruby_cleanup(0); SHOULD clean up the interpreter. Problem is not so
much cleaning the interpreter but restarting it again.
I did find issues when I did this (back in 1.8.0). I think now issues
are better, but I won't vouch for it being 100% solved.
If all fails, best thing you can do is what you did. Encapsulate all
in its own class and clean that instead. That will still leave the
class/module constants around, but it is better than nothing.

- How do I load a script from memory? rb_eval_string()? rb_compile_string()?
Something else?

Use rb_eval_string_protect().

- The documentation is flawed. It indicates calling rb_run(), but that can
eventually call exit, which is the last thing you want from an embedded
language. It mentions rb_apply() as having an argc parameter.

Again, rb_eval_string_protect() is what you probably want.

- I am getting inexplicable crashes when calling the garbage collector.

This is a probably a bug in your code. Take a look at rubyMEL at
highend3d.com for a relatively simple embedding of MEL into a 3d
application (Maya).
Depending on platform (windows, mainly) you need other initialization
code besides ruby_init().

- The documentation is not at all clear how you protect something from the
garbage collector, or indicate it is no longer in use.

You need to use a mark and a free function respectively, and use ruby's
mysterious Data_Wrap_Struct stuff. This is the pretty hairy part of
exposing a C++ api.
Suggestion: look at SWIG's code. It has a bunch of wrapper functions
to handle it nicely and it is now pretty mature.
For any questions regarding SWIG not clear from the SWIG manual, refer
to FXRuby's excellent wrapping of the Fox library.

- How does one call "Foo.new(args)" on a class from outside of Ruby, where the
args have been generated from outside of Ruby (hence no name) meaning

You call it like any other method, with rb_funcall* like:

VALUE cFoo; // defined somewhere else...
int nargs = 1;
VALUE args[] = { INT2NUM( 30 ) };
ID new_id = rb_intern("new");
rb_funcall2( cFoo, new_id, nargs, args );

- rb_protect() is a complete mess. Rather than being able to call an arbitrary
function with a void pointer, you are stuck calling something that takes a
single VALUE arg.

No, you are wrong. It is extremely well thought out.
No need for ugly void*, thank god.
Think of VALUE as being a void* (it is really an unsigned long). Ruby
requires sizeof(unsigned long) == sizeof(void*) -- see ruby.h's check.
Just do a simple casting of your pointer to VALUE in your caller and do
the opposite cast from VALUE to whatever in your receiver function.
See how all the RCAST() and similar macros work in ruby.h.

- The code runs fine for a while if I don't call the garbage collector.
Unfortunately, I get stack depth errors and eventual segfaults if I make
repeated calls to the Ruby script (via rb_apply()).

This likely means you wrapped your C++ classes incorrectly or you are
trying to run ruby in a multi-threaded environment.
If the first is true, look and learn from SWIG. If the second is true,
remove the multi-threaded code, create a global mutex that all your
ruby functions should call before doing anything or use a different
scripting language at this time.
Ruby is not thread safe, unlike LUA.

- I can't find one example that completely and correctly demonstrates the whole
process involved in having C call Ruby then call C, return values through the
chain neatly, and clean up. If there was such a reference, it would be outstanding.

There's no real trick to it.
Ruby is fine as an embedded language, just as you remember it is not
safe as a multi-threaded embedded language. YARV may eventually solve
this somewhere in the future. The unofficial fork of Ruby that is
Sydney (and now rubinius) supposedly is multi-thread safe, but is
32-bits only (and I won't vouch for it).
Python is just as bad as Ruby, btw, albeit it avoids some threading
issues by using a global lock every time two threads try to run some
code simultaneously, effectively running only a single thread at a
time. In ruby, you can do the same, but you need to create this global
mutex yourself.
LUA and TCL are different and better in this aspect, as they both can
use different (and very light) interpreter instances to execute code.

Am I alone in thinking this? Should I be looking at other embeddable languages
(Python, for example), or just go back and get the LUA code working again? Am I
doing something wrong?

Truth is... you are looking at the two best languages for embedding.
LUA is admittedly better to embed due its multi-threading aspects,
while Ruby is admittedly better for embedding due to its nice syntax.

Is there anything I can do to successfully and reliably add support for my
favourite scripting language into my project? Or is it simply an unrealistic
prospect?

Follow the advices I gave you and you should be able to get ruby as an
embedded language. Wrapping classes for the garbage collector is the
hairy part, but you can steal all of SWIG's source code for that now
(or use SWIG itself for doing it).
However, if your game engine does require heavy multi-threading in the
scripting language, you are indeed looking at the wrong language.
LUA is still the best language for this, unfortunately.
Another thing you might find out an issue when embedding another
language is performance.
LUA (and specially LuaJIT) is damn fast and is several orders of
magnitude faster than ruby (LuaJIT is about 10x-15x faster for most
stuff, and that includes YARV).

 

[M. Edward (Ed) Borasky]

GD wrote:

[snip]

I wonder if Ruby is at all suitable as a general-purpose embedded
scripting language? I am beginning to think that it is not.

The embedded interface and documentation are terrible! I don't mean
any offense, and I maintain my stance that Ruby is an outstanding
language. I love it. And I do realise that sometimes code takes some
time to mature- I've been using Ruby since the 90's and it wasn't
always a smooth ride. Is this the problem, it just isn't quite ready
for embedding yet? Or I'm not using the right documentation? Or I'm
asking too much? Am I doing the wrong thing?

It probably seems odd for someone with such a high opinion of the
language to have such a low opinion of the embedded interface. Perhaps
it is because Ruby has set my expectations so high that I am that much
more easily disappointed.

Some projects seem to have managed it, and they may work, to some
degree. But their use of it seems to be fairly basic, rather than
counting on it as a core component of the program. VIM has a nice
example, although it doesn't seem to worry about garbage collection or
cleaning up. As such it doesn't seem complete.

Am I alone in thinking this? Should I be looking at other embeddable
languages (Python, for example), or just go back and get the LUA code
working again? Am I doing something wrong?

Is there anything I can do to successfully and reliably add support
for my favourite scripting language into my project? Or is it simply
an unrealistic prospect?

Well ... first of all -- I suspect your project is rather large, and you
seem to have factored it into a Ruby piece, a C++ piece, and a Lua
piece, with each of those languages doing what it is good at. So,
barring a complete re-write in a *single* language, I'd say the most
expeditious thing to do would be to fix the Lua code that's broken in
Lua, rather than trying to rewrite it in Ruby. I don't know what your
time and money constraints are -- if you need to earn a living with
E.V.E. Paradox, I'd do whatever gets you on the air in the most "agile"
manner.

But if you've got the time, the ambition, the money, etc. and want to
rewrite the project in a single language, I think, given today's
high-speed low-cost systems, almost any language will do the job, so
pick one that you can work productively in. With a little tweaking and
maybe a little C, you ought to be able to do everything you're doing now
in Ruby instead of a mix of languages. Ruby has a nice set of
concurrency primitives, such as monitors, threads, distributed objects
and Linda-like tuple spaces.

 

[Martin DeMello]

However, if your game engine does require heavy multi-threading in the
scripting language, you are indeed looking at the wrong language.
LUA is still the best language for this, unfortunately.

What about Io? I haven't used either Io or Lua myself, but they both
seem to offer similar embedding-oriented features, and Io seems like a
more attractive language to work in.

martin

 

[GD]

Hey gga,

Thankyou for the reply and information.

GD ha escrito:



Use rb_eval_string_protect().

Excellent, guessed that one at least. Maybe worth a mention in the docs
if that is the "official" solution; it wasn't too clear.

Ideally, handle this using SWIG, as either a ruby extension or a pseudo
extension you link in statically.
Avoid doing the binding yourself manually. Binding functions for most
basic C constructs (ints, chars, etc) is VERY easy.
Doing proper binding of classes, it is not as easy, as the GC requires
you to be careful. Best thing for you to do is look at SWIG's output.
SWIG also deals properly now with inheritance of C++ classes into ruby,
including multiple inheritance.

I'm keeping things fairly simple; I mostly just bind functions/methods
and wrap pointers in opaque data structures (eg. using Data_Wrap_Struct
in the Ruby version). My plan was to store a pointer to the calling class
in an opaque structure and store it in the corresponding Ruby class, so
I can reference it when needed.

This may be much trickier with the current ruby C interpreter,
depending on what you are looking for. You can start a C++ thread to
do it, but the ruby C interpreter is not thread safe. So, if you have
ruby code running in two different threads at the same time, bad things
WILL happen.
You can also try to rely on a single thread and let ruby's own light
threads handle this. However, this may not be as efficient as actual
multi-threaded code.
Mind you, ruby is not the only language with issues here. Python is
also pretty bad at this.
LUA and TCL are in a class of their own as far as multithreading
interpreters go.

Everything is single-threaded, or at least, the other threads don't even
know Ruby exists. Events and updates are handled in the one thread and
can be fed in in a safe and controlled manner. I'm not asking too much
of an embedded language, I'm keeping things fairly gentle. The nasty
stuff is handled in the C++ code.

ruby_cleanup(0); SHOULD clean up the interpreter. Problem is not so
much cleaning the interpreter but restarting it again.
I did find issues when I did this (back in 1.8.0). I think now issues
are better, but I won't vouch for it being 100% solved.
If all fails, best thing you can do is what you did. Encapsulate all
in its own class and clean that instead. That will still leave the
class/module constants around, but it is better than nothing.

Indeed. Wasn't sure about ruby_cleanup, and I've seen a lot of examples
floating around, along with comments that things end up crashing shortly
afterward.

Again, rb_eval_string_protect() is what you probably want.

I have data structures in the C++ side, no names, so rb_eval_string and
company can't be used.

This is a probably a bug in your code. Take a look at rubyMEL at
highend3d.com for a relatively simple embedding of MEL into a 3d
application (Maya).
Depending on platform (windows, mainly) you need other initialization
code besides ruby_init().

I won't be so arrogant to suggest my code has no bugs (haha) but we're
talking about code that has been run through days of heavy automated
testing each release with the current LUA setup being changed to use
Ruby instead, minimally reduced to using only two calls (ruby_init and
rb_gc) being called from multiple different locations and crashing
in different but consistent ways each time. It is entirely possible
that my code is interfering with it in some way, but it is manifesting
as predictable crashes in Ruby only, not in any of the other libraries
I am using. Short version: Looks more like a bug brought out by my
circumstances rather than because of the code itself.

I'm currently trying to get things going under Linux, so no tricky
init stuff yet.

You need to use a mark and a free function respectively, and use ruby's
mysterious Data_Wrap_Struct stuff. This is the pretty hairy part of
exposing a C++ api.
Suggestion: look at SWIG's code. It has a bunch of wrapper functions
to handle it nicely and it is now pretty mature.
For any questions regarding SWIG not clear from the SWIG manual, refer
to FXRuby's excellent wrapping of the Fox library.

I hadn't thought of going back into SWIG. Good idea, thankyou. I had a
play around with it a week or so ago, seemed to work well. No doubt
the generated code would grant some clues.

Should some of the tricks used by SWIG be mentioned on the official
Ruby site, perhaps? Seems odd that an offsite source is considered
(mildly) authoritative.

- How does one call "Foo.new(args)" on a class from outside of Ruby, where the
args have been generated from outside of Ruby (hence no name) meaning

You call it like any other method, with rb_funcall* like:

VALUE cFoo; // defined somewhere else...
int nargs = 1;
VALUE args[] = { INT2NUM( 30 ) };
ID new_id = rb_intern("new");
rb_funcall2( cFoo, new_id, nargs, args );

Okay, so just like an object. Good news. Very elegant and Ruby-ish.
Any way to grab the class from the Ruby script? What I really want is
to replace your first line with something like:

VALUE cFoo = rb_grab_class_from_script("ClassDefinedInScript")

Each level defines its own class and I need to create an object of
the correct type. I can't find a call to lookup a class name and
return a VALUE. There is probably one, I just don't know what it is.

No, you are wrong. It is extremely well thought out.
No need for ugly void*, thank god.
Think of VALUE as being a void* (it is really an unsigned long). Ruby
requires sizeof(unsigned long) == sizeof(void*) -- see ruby.h's check.
Just do a simple casting of your pointer to VALUE in your caller and do
the opposite cast from VALUE to whatever in your receiver function.
See how all the RCAST() and similar macros work in ruby.h.

The best way to illustrate would be with an example:

VALUE GlueFunc(void *v)
{
Glue *glue = (Glue *)v;
return rb_funcall(glue->game->getLevel(), glue->method, glue->arg0);
}

struct Glue
{
Game *game;
ID method;
VALUE arg0;
};

void Game::runWithProtect1(VALUE level, ID method, VALUE arg0)
{
Glue glue;
glue.game = this;
glue.method = method;
glue.arg0 = arg0;
VALUE rv = rb_protect_that_uses_void(GlueFunc, &glue, &status);
if (status)
reportError();
return rv;
)


In main code:

ID id_update;
VALUE current_frame;

....

runWithProtect1(id_update, current_frame);

(NB: There would be a large number of calls to runWithProtect(), so I need
to keep the call format simple).

If the glue function must be a VALUE which is an array, what steps
would you suggest were used to ensure the relevant data, include a C++
pointer, were passed through the callback to reach the glue function
GlueFunc? If you're bored, code up a solution that works. I managed to
get something going, minus the "this" pointer, and it was messy. It
involved converting everything into Ruby objects and putting them into
an array, and deconstructing at the other end. Perhaps you have a better
solution, and I'm open to improved ideas.

This likely means you wrapped your C++ classes incorrectly or you are
trying to run ruby in a multi-threaded environment.
If the first is true, look and learn from SWIG. If the second is true,
remove the multi-threaded code, create a global mutex that all your
ruby functions should call before doing anything or use a different
scripting language at this time.
Ruby is not thread safe, unlike LUA.

Only using Ruby from a single thread. I am likely making mistakes
with creation and marking of Ruby objects/classes as I'm not familiar
enough with it nor does the documentation explain it. Thus manually
calling the GC, which has led to crashes even when all I do is init
and call the GC, hence my problems. I can't be making mistakes with
Ruby if I'm not actually loading any scripts or creating any objects
and classes.

outstanding.


There's no real trick to it.

I must disagree. If it is easy, there is a link _somewhere_ that
shows a concrete official example of how it should be done. Maybe
I have not found it. If it doesn't exist, if a Guru who intimately
understands the code could put one together and put it in the official
documentation, it would be an enormous help to people using the
embedded interface.

Ruby is fine as an embedded language, just as you remember it is not
safe as a multi-threaded embedded language. YARV may eventually solve
this somewhere in the future. The unofficial fork of Ruby that is
Sydney (and now rubinius) supposedly is multi-thread safe, but is
32-bits only (and I won't vouch for it).
Python is just as bad as Ruby, btw, albeit it avoids some threading
issues by using a global lock every time two threads try to run some
code simultaneously, effectively running only a single thread at a
time. In ruby, you can do the same, but you need to create this global
mutex yourself.
LUA and TCL are different and better in this aspect, as they both can
use different (and very light) interpreter instances to execute code.

I'm not using Ruby as a multithreaded language; only one thread will
ever talk to it and being MT safe is unimportant to me.

The Python docs look decent, I'd just rather not relearn Python again,
I chose Ruby years ago, I'm good with it, and I like it better than
Python! But I'd rather move on from LUA if I can...

LUA has some very nice elements; there is a call to get a context, a
call to tear it down, so forth. It works. The interface to the controlling
code, however, is less than ideal and poorly documented. Using it is
painful. Error-handling makes me want to gouge my eyes out. The language
is a mere flicker to the brilliance of Ruby. However, it is very well
suited for the task; its design has always been around being an embedded
language. The embedded interface works. It is a well-constructed embedded
language.

My knowledge of TCL is non-existent.

Truth is... you are looking at the two best languages for embedding.
LUA is admittedly better to embed due its multi-threading aspects,
while Ruby is admittedly better for embedding due to its nice syntax.

Actually, I got LUA up and running in an hour or so. It was very basic,
but a good start that evolved into the current codebase. The documentation
was poor but I got it eventually. I'm not even using the MT stuff. I didn't
even know it handled it well.

I've spent close to two days on embedding Ruby and can't get the basic
things I've described working. The closest I've got is short runs followed
by crashes I can't diagnose. The embedding documentation is all over the
place, lacks examples, and has mistakes. When someone with twenty years
of development experience is having troubles this bad it does suggest there
might be a problem that needs addressing...

It is enormously frustrating as the Ruby language is significantly superior
to the LUA language, but the case is the exact reverse with the embedding
interface- and the LUA embedding interface could be much better! It breaks
my heart to have to struggle so hard to get a powerful language like Ruby
- that I use throughout my project extensively- to work reliably as an
embedded scripting language for the runtime itself. If I wasn't such a
huge fan of Ruby- and I promise you I am difficult to impress with a
language- then I would have given up after the first few frustrating
hours and said to myself "language is not ready yet", then moved on to
something else. I dearly want to be completely wrong. If I can get Ruby
going I can increase the significance of scripting in level design, and
add some really cool features I can't be bothered implementing in
LUA.

Follow the advices I gave you and you should be able to get ruby as an
embedded language. Wrapping classes for the garbage collector is the
hairy part, but you can steal all of SWIG's source code for that now
(or use SWIG itself for doing it).
However, if your game engine does require heavy multi-threading in the
scripting language, you are indeed looking at the wrong language.
LUA is still the best language for this, unfortunately.

I'll dig around in the SWIG stuff, this may give me some ideas.

Thanks for the suggestions and input, much appreciated.

Garth

 

[GD]

Hey Ed,

Well ... first of all -- I suspect your project is rather large, and you seem

to have factored it into a Ruby piece, a C++ piece, and a Lua piece, with each
of those languages doing what it is good at. So, barring a complete re-write in
a *single* language,

The existing Ruby stuff is involved in the build process, and involves
running Ruby scripts- no embedded Ruby. It works brilliantly. I am
considering replacing the LUA piece with an embedded Ruby piece. Rewriting
the existing scripts would take a day at most, once I get the embedded Ruby
interface and hooks to my code working fine. The "working fine" bit is
the issue.

You are correct regarding each language doing what it is best at, so
hopefully you'll understand when I say a rewrite is simply never going
to happen. The low-level C++ code is optimised to death and the flexible
Ruby configuration and build tools would be overwhelmingly painful to write
in C++. So yeah, I need the best of all worlds. I am using multiple tools
in ways that emphasise their strengths. I just need them to play nice
together.

I'd say the most expeditious thing to do would be to fix the Lua code that's

broken in Lua, rather than trying to rewrite it in Ruby. I don't know what your
time and money constraints are -- if you need to earn a living with E.V.E.
Paradox, I'd do whatever gets you on the air in the most "agile" manner.

Yes. Can't see that taking more than a day. BUT if Ruby was my scripting
language I can make huge improvements to the scope of scripting in
E.V.E. Paradox. I can move game features out into it. There is a lot of
incredible stuff I could do. I'd love to have Ruby scripting, and the
payoff in time may be worth it. But if the outcome is unattainable, I need
to consider other options.

Garth

 

[GD]

Hey Martin,

Never used it. Will look into it, thanks for the suggestion.

Garth

 

[gga]

GD ha escrito:

I'm keeping things fairly simple; I mostly just bind functions/methods
and wrap pointers in opaque data structures (eg. using Data_Wrap_Struct
in the Ruby version). My plan was to store a pointer to the calling class
in an opaque structure and store it in the corresponding Ruby class, so
I can reference it when needed.

That's possible, but it can get ugly. Do what SWIG does. Create a
hash that stores a mapping between the ruby object and the C++ object
(and/or viceversa). Every time you allocate a new class, you enter it
in the hash. Every time the free function for the class is called, the
mapping is removed from the hash.
You can even use ruby's own hashes to do so.
That way, you don't need to pass any pointer, and can access any class
at any time from any function.
Also, if your classes contain or return pointers to other C++ classes,
you need to keep telling the GC about it. See the SWIG Zoo/Animal
tutorial in the Ruby section.

I have data structures in the C++ side, no names, so rb_eval_string and
company can't be used.

This makes no sense to me.

Should some of the tricks used by SWIG be mentioned on the official
Ruby site, perhaps? Seems odd that an offsite source is considered
(mildly) authoritative.

Well, all that stuff IS properly documented in the PickAxe, in the
extending Ruby section. The PickAxe is somewhat poor in that it does
cover properly wrapping methods that return new C++ objects (the Zoo
and Animal tutorial in SWIG is a top-notch example of all the headaches
you can run into on wrapping a complex library).

You also have this, if you haven't read it.

http://metaeditor.sourceforge.net/embed/

I found it somewhat confusing, but it is probably another source.

Okay, so just like an object. Good news. Very elegant and Ruby-ish.
Any way to grab the class from the Ruby script? What I really want is
to replace your first line with something like:

VALUE cFoo = rb_grab_class_from_script("ClassDefinedInScript")

Class names are just constants.
If your class is in the global namespace, use:

VALUE cFoo = rb_const_get( rb_cObject, rb_intern("YourClass") );

If not, replace rb_cObject for the proper module name.

Each level defines its own class and I need to create an object of
the correct type. I can't find a call to lookup a class name and
return a VALUE. There is probably one, I just don't know what it is.

//--------------------------------------------------------------------------------------------
// g++ -fPIC -shared -I/usr/include/ -I /usr/include/ruby/\
// -I/usr/lib/ruby/1.8/x86_64-linux/ callback.cpp -o callback.so
//
// > irb
// irb> require 'callback.so'
// irb> wrap()
// irb> wrap 'crap', 'hello'
//


#include <ruby.h>
#include <iostream>

#define VALUEFUNC(f) ((VALUE (*)(ANYARGS)) f)

struct CallbackData_t
{
void* crapola;
VALUE method;
VALUE arg0;
};

VALUE wrap_callback( VALUE v )
{
using namespace std;
CallbackData_t* data = static_cast< CallbackData_t* >((void*)v);

cerr << "wrap callback got: " << data->crapola << endl;
// do any rb_funcall2() or rb_apply() here

if ( rb_obj_is_kind_of( data->arg0, rb_cString ) )
cerr << "arg0 is " << StringValueCStr( data->arg0 ) << endl;

return Qnil;
}

VALUE wrap_protect(int argc, VALUE *argv, VALUE self)
{
CallbackData_t data;
data.crapola = (void*)0xDEADBEEF;

// 0 mandatory arguments, 2 optional
rb_scan_args( argc, argv, "02", &data.method, &data.arg0 );

int error;
VALUE result = rb_protect( wrap_callback, VALUE(&data), &error );
if ( error )
rb_raise( rb_eRuntimeError, "Something crappy happened" );
return result;
}

extern "C" void Init_callback(void)
{
rb_define_method(rb_cObject, "wrap", VALUEFUNC(wrap_protect), -1);
}
//--------------------------------------------------------------------------------

The Python docs look decent, I'd just rather not relearn Python again,
I chose Ruby years ago, I'm good with it, and I like it better than
Python! But I'd rather move on from LUA if I can...

Python is worse at embedding than Ruby is. So if you have troubles
with Ruby, good luck with Python! On python you don't need to just
take care of garbage collecting, but of reference counting. It is a
royal pain if you are doing it manually like you are.

LUA has some very nice elements; there is a call to get a context, a
call to tear it down, so forth. It works.
Indeed.

My knowledge of TCL is non-existent.

Keep it that way. If you don't like Lua, TCL will scare the shit out
of you. TCL syntax can look pretty but it often does not. It does
have one of the best embedding interface for embedding of any language
(and 20+ years of knowledge you can rely on).

I've spent close to two days on embedding Ruby and can't get the basic
things I've described working.

You should be able to get ruby up and running in about a couple of
hours (that's all it took me). Wrapping complex classes that return
pointers to other classes and the like, however, is hairy. But SWIG is
there to help now.

I dearly want to be completely wrong.

You are.

 

[gga]

Io is, at this point in time, too immature, compared to Lua or Ruby.
Their current "stable" release won't even compile.

 

[gga]

Io is, at this point in time, too immature, compared to Lua or Ruby.
Their current "stable" release won't even compile.
And, unfortunately, their developers are a tad insane and use darcs for
revision control, which means you can spend almost an hour retrieving
their current unstable release the first time you check it out.

 

[Vincent Fourmond]

Am I alone in thinking this? Should I be looking at other embeddable
languages (Python, for example), or just go back and get the LUA code
working again? Am I doing something wrong?

Just an idea - have you ever looked at scheme ? It seems a good choice
for embedded scripts, although I never had to work with it so far.

Vince

 

[benjohn]

Io is, at this point in time, too immature, compared to Lua or Ruby.
Their current "stable" release won't even compile.
And, unfortunately, their developers are a tad insane and use darcs for
revision control, which means you can spend almost an hour retrieving
their current unstable release the first time you check it out.

I thought Darks seemed pretty awesome when I looked, but very
annoying needing to get hold of ocaml or hascal, or what ever it was to
compile it up. Now, if it were written in Ruby...

 

[M. Edward (Ed) Borasky]

Just an idea - have you ever looked at scheme ? It seems a good choice
for embedded scripts, although I never had to work with it so far.

Vince

Yeah, there are a couple of Scheme-C half-breeds (Scheme interpreter
plus Scheme-to-C compilers, Scheme embeddable in C, etc.) that are
pretty good and high-performance. Chicken and Gambit are the two that I
know the most about. Gambit has an experimental Erlang-like lightweight
process capability that might be good for game programming. I know next
to nothing about game programming. I don't play the games and the last
time I even attempted to write one, the hardware was on the order of 64
kilobyte 6502s and the language of choice was Forth.

 

[Brian Mitchell]

I thought Darks seemed pretty awesome when I looked, but very
annoying needing to get hold of ocaml or hascal, or what ever it was to
compile it up. Now, if it were written in Ruby...

Replying to the general content above...

Io developers are very smart and also very novel. They have managed to
reinvent a good portion of language ideas out there. This is a sign of
good levels of insanity. Sane people would probably fall in line and
do with the less than perfect tools available. Insane people take on
invisibly huge projects and responsibility to create things that some
people just haven't thought possible or realistic.

The result? Io is a great language that is combining a lot of healthy
features into a pretty usable package (co-routines, prototype object
system, message tree as objects as code -- hello lisp, etc... see
iolanguage.com or check out freenode's #io). It is regrettably
unstable though. I've fallen out of the active crowd, being so busy
with work, but last time I was involved it was mostly an unplanned
project with some rough ideas. I hope it has moved beyond project
management difficulties.

The use of darcs was actually one of the more key portions of the Io
community IMO. I have to admit I am biased though. I was one of the
people who originally suggested that darcs be used (there was a time
when there was nothing but rsync :S -- Steve is happily into the scm
thing now). What it let us do was experiment and detach things from
the main repository at any time and with anyone. Things like bikeshed
arguments quickly became show-me-a-darcs-patch. This really helped us
evolve things much faster than just a simple ML would have done.

The barrier to entry was also pretty high because of darcs and the use
of IRC over the ML for serious conversations. This meant you had to
put the time in to be heard. Is it fair? I don't know. It seems to
keep trolls out but does it prevent valid work? It probably is just a
trade off like anything. I've had success (great active collaboration)
and failure (a few bug fixes floating around that took over a week to
get applied when it isn't mentioned actively enough) with this system.

Haskell needs no major mention as it is pretty popular topic on social
networks like reddit. It is worth noting that it can be a pain to take
on such a large monster just to get an SCM installed. All I can say is
that I would rather have more stable projects using something other
than C (good bye seg-faults -- mostly).

As far as a good scm in Ruby... now there is something I would love to
see. I've honestly been thinking of porting Mercurial, as I have
mentioned many, many times. One other interesting option would be to
wrap git up into a ruby extension and then write some code with that.
Of course, like most of these stories go, I don't have time right
now... but if someone does, I would love to help out in smaller ways.

Brian.

 

[M. Edward (Ed) Borasky]

As far as a good scm in Ruby... now there is something I would love to
see. I've honestly been thinking of porting Mercurial, as I have
mentioned many, many times. One other interesting option would be to
wrap git up into a ruby extension and then write some code with that.
Of course, like most of these stories go, I don't have time right
now... but if someone does, I would love to help out in smaller ways.

Brian.

Bah! A new FAQ -- we've done editors, IDEs and GUI Toolkits, so let's
have a "what's the best version control system for Ruby" contest now.

But seriously, folks, why does a version control system need to be
written in any particular language? Shouldn't Rubyists focus on tools
that increase productivity in Ruby, like tools that deal with the
specifics of Ruby syntax and semantics, rather than trying to re-invent
a version control system, an editor, an IDE, etc.?

Speaking of which, I decided to have a bake-off between KDevelop and
Komodo 4 Beta for a Ruby IDE. I think in the end KDevelop is going to
win, because it does more languages than Komodo, free is a very good
price, and I don't need Windows, MacOS or Solaris compatibility. But
Komodo 4 is a pretty good tool from what I've seen, and if I have to do
a lot of Perl development in my "day job" on Windows/ActiveState Perl,
I'll probably ask for a copy and take the Ruby IDE as a "bonus".
KDevelop has some nice integration with QtRuby as well, and I *think*
it's possible to develop QtRuby (QT4) on KDevelop that can be deployed
on a Windows system. Komodo has some nice integration with Tk.

 

[Brian Mitchell]

Bah! A new FAQ -- we've done editors, IDEs and GUI Toolkits, so let's
have a "what's the best version control system for Ruby" contest now.

But seriously, folks, why does a version control system need to be
written in any particular language? Shouldn't Rubyists focus on tools
that increase productivity in Ruby, like tools that deal with the
specifics of Ruby syntax and semantics, rather than trying to re-invent
a version control system, an editor, an IDE, etc.?

Speaking of which, I decided to have a bake-off between KDevelop and
Komodo 4 Beta for a Ruby IDE. I think in the end KDevelop is going to
win, because it does more languages than Komodo, free is a very good
price, and I don't need Windows, MacOS or Solaris compatibility. But
Komodo 4 is a pretty good tool from what I've seen, and if I have to do
a lot of Perl development in my "day job" on Windows/ActiveState Perl,
I'll probably ask for a copy and take the Ruby IDE as a "bonus".
KDevelop has some nice integration with QtRuby as well, and I *think*
it's possible to develop QtRuby (QT4) on KDevelop that can be deployed
on a Windows system. Komodo has some nice integration with Tk.

Why not. Not a single word in that told you to go out of your way to
use it. Thanks goes wholly to OSS. Some of us like to innovate rather
than sit around. I don't think I would be productive trying to modify
Subversion's code base to do what I want out of an SCM tool. It just
comes down to the fact that I am better with Ruby.

sarcasm do

For that, why are we even speaking multiple of anything? Why don't we
all use the same language, the same OS, the same hardware... and while
we are at it, we can have the same attitude ideas and personalities.

end

Brian.

 

[M. Edward (Ed) Borasky]

Why not. Not a single word in that told you to go out of your way to
use it. Thanks goes wholly to OSS. Some of us like to innovate rather
than sit around. I don't think I would be productive trying to modify
Subversion's code base to do what I want out of an SCM tool. It just
comes down to the fact that I am better with Ruby.

Yes ... I understand the need to create and need to innovate as much as
anyone. However, there are lots of worthwhile projects that are begging
for developers to enhance and maintain them while other developers are
re-inventing wheels, levers, inclined planes and pulleys. And there's
even more "worthwhile" open source software that is languishing because
only other programmers can use it!

Now in the specific case of version control systems, the main reason
there seem to be so many of them is that different project management
schemes have different requirements, and only the heavy-duty industrial
strength (and expensive) ones like ClearCase have any hope of covering
lots of those bases. And before you go beating up on ClearCase, I don't
happen to be a big fan of it, I know it has warts, but it's what's
mandated in a lot of places.

CVS is still "good enough" for lots of SCM users, and between CVS and
SVN, there's not much need for more innovation as far as I'm concerned.
One of the big principles of "agile" and "pragmatic" development is
YAGNI -- You Ain't Gonna Need It!

sarcasm do

For that, why are we even speaking multiple of anything? Why don't we
all use the same language, the same OS, the same hardware... and while
we are at it, we can have the same attitude ideas and personalities.

end

Well, actually, we pretty much do all use the same language -- C and
dialects of C -- and we pretty much all do use the same hardware -- x86
and x86-64. We pretty much all have a "File" menu in the upper left
corner, we pretty much all have a mouse and a keyboard and a color
graphical display, we pretty much all talk to each other using IPV4 and
dozens of other standard protocols, etc., etc. Innovations need to be
*compelling* to get someone to use something else.

I don't really think of Ruby as innovative -- one of the charming things
about Ruby is that it is a collection of good ideas that have stood the
test of time. Regular expressions, classes, objects and methods, dynamic
"typing", flexible syntax, "it's always run time", continuations,
arbitrary precision integer arithmetic, flexible arrays and hashes ---
these are all wheels that Ruby didn't re-invent!