Seven new VMs, all in a row

[Peter Suk]

Hello everyone,

I thought I'd talk about my new project here, since there is a good
chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs. Ruby and Smalltalk are very similar
under the covers, so Smalltalk VMs are a very good match for the
language. This will give Ruby a much faster execution environment
(perhaps 30X), VMs which are capable of incremental garbage collection,
generational garbage collection that is so fast your progra still works
even with an infinite loop allocating new objects (I do this as a lark
sometimes), a wonderful debugger which will let programmers modify
methods on the fly & continue execution, a "workspace" window where you
can execute arbitrary code, a visual "inspect", a powerful "Refactoring
Browser," an industrial strength OODB (Gemstone) with objects and
methods you can define in Ruby, and a readily accessible meta-level
which will allow Rubyists to readily modify their own language. (For
example, you could then use Method wrappers to very quickly implement
an Aspect-Oriented Ruby.)

My strategy for doing this involves writing a Ruby parser (or, rather,
translating the existing one in JRuby to Ruby) then writing a Smalltalk
Parser object to request parsing of Ruby code into an AST from a Ruby
program outside of Smalltalk. We then reify the AST inside the image
and use it to compile Ruby methods into bytecodes. (Multiple syntaxes
can coexist in one Smalltalk image.) Once this is done, we can then
compile the external Ruby parser and bring it into Smalltalk.
Afterwards, we can use the Refactoring Browser Smalltalk parser plus a
little runtime type inferencing to incrementally transform the image
into pure Ruby.

We can do all of this without changing the structure of Ruby files &
Modules or requiring Rubyists to do Smalltalk style image oriented
development. And for those of you sharp enough to wonder: yes, we can
handle Modules, Mixins, and fully qualified Method names without
changing the Smalltalk VMs. (At least those that have Namespaces.)

If anyone is interested, please drop me a line.

--Peter

 

[R. Mark Volkmann]

I think it would be really cool if you could do this on top of the free Squeak
VM.

 

[Peter Suk]

I think it would be really cool if you could do this on top of the
free Squeak
VM.

Yes, I was thinking along these lines. However, no real Namespaces
(yet) in Squeak. We could finagle this by referencing classes with
Module-name prefixes. (This has already been proposed.) When real
Namespaces appear in Squeak, we could then properly place the classes
in modules/namespaces and rename them programatically.

The VMs I am targeting first are commercial ones, but they have free
versions/licenses. (VisualWorks and Gemstone) In addition, they are
very attractive for implementing servers. (VisualWorks is damn fast,
and Gemstone is a mature, rock-solid OODB.)

 

[Lothar Scholz]

Hello Peter,

PS> Hello everyone,

PS> I thought I'd talk about my new project here, since there is a good
PS> chance that someone might be interested in it. I'm planning to put

Is this just brainstorming or have you already done something ?

 

[Peter Suk]

Hello Peter,

PS> Hello everyone,

PS> I thought I'd talk about my new project here, since there is a good
PS> chance that someone might be interested in it. I'm planning to put

Is this just brainstorming or have you already done something ?

Lothar,

I have my hands on a working copy of Racc, and the JRuby source. I've
begun the port of the RubyLexer into Smalltalk, but I've decided to
abandon that and use the bootstrap from outside the image technique
instead. I've also started the port of DefaultRubyParser.y from the
JRuby source. I have a simple Ruby script that uses Regexps to
transform the DefaultRubyParser.ry into Ruby. (Regexps are not
powerful enough to tackle the job generically -- that would take a
Context Free Grammar -- but I am including enough specific context in
the regexps that it will work in the specific cases needed.)

I have also worked out and demonstrated the manipulation of method
dictionaries so that we can compile fully qualified Ruby method names
for the VisualWorks VM. I have sketched out a design for handling Ruby
Mixins and assignment-implicit accessors.

I have also recruited a helper for translating the AST node classes.
(There is something to be said for little languages. It's a lot easier
to build syntax driven tools for them!)

Next, I will be taking a very simplified Ruby subset (the calculator
example from the "Dragon Book" which comes with Racc, and the ability
to define methods in Ruby classes) and working out the
Parser-Outside-The-Image scheme with that. Hopefully I'll have that by
this weekend.

--Peter

 

[jc]

Hi Peter,

- Would this support programming both within or without an image,
choice up to the user?

- From what I understand Smalltalk has a small VM written in C, with
the rest written in Smalltalk itself. Would your version be the Ruby
equivalent of this (VM with everything else in Ruby), or would there
still be some Smalltalk behind the Ruby?

-jc

 

[pat eyler]

Hi Peter,

- Would this support programming both within or without an image,
choice up to the user?

- From what I understand Smalltalk has a small VM written in C, with
the rest written in Smalltalk itself. Would your version be the Ruby
equivalent of this (VM with everything else in Ruby), or would there
still be some Smalltalk behind the Ruby?

See the MetaRuby project:

http://blog.zenspider.com/archives/metaruby/index.html (among other places)


(Although this is about Ruby in Ruby, not Ruby in smalltalk)

 

[Florian Groß]

I thought I'd talk about my new project here, since there is a good
chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs.

This does indeed sound promising. Having a self-hosted Ruby would have
certain benefits as well.

a wonderful debugger which will let programmers modify methods on the
fly & continue execution

That is already possible with ruby-breakpoint which spawns an IRB
session at a specific place of your code. See
http://ruby-breakpoint.rubyforge.org/ if you are interested in this sort
of things.

It sure can't hurt to have more options, though.

 

[Peter Suk]

Hi Peter,

- Would this support programming both within or without an image,
choice up to the user?

Basically, you could treat the image as just a very powerful
debugger/IDE. The image would spit out new versions of your module
files, which you could then edit/diff to your liking. I have devised a
very powerful algorithm to respect comments, based on diffs of token
sequences.

- From what I understand Smalltalk has a small VM written in C, with
the rest written in Smalltalk itself. Would your version be the Ruby
equivalent of this (VM with everything else in Ruby), or would there
still be some Smalltalk behind the Ruby?

Initially, there would be some Smalltalk behind the Ruby, but we'd fix
it so you'd never see it. Eventually, we can develop a completely Ruby
meta-layer.

 

[Peter Suk]

See the MetaRuby project:

http://blog.zenspider.com/archives/metaruby/index.html (among other
places)

(Although this is about Ruby in Ruby, not Ruby in smalltalk)

I am aware of MetaRuby. This project is also about Ruby in Ruby.
(Smalltalk is in Smalltalk. But the VM doesn't care which
language/meta-language it is running!)

However, this meta-Ruby will run two orders of magnitude faster!

 

[Peter Suk]

This does indeed sound promising. Having a self-hosted Ruby would have
certain benefits as well.


That is already possible with ruby-breakpoint which spawns an IRB
session at a specific place of your code. See
http://ruby-breakpoint.rubyforge.org/ if you are interested in this
sort
of things.

It sure can't hurt to have more options, though.

In the Smalltalk debuggers, you can spawn off as many debugger sessions
as you want by highlighting arbitrary code in the debugger as well as
any browser window, as well as visually "inspect" any object (with each
one appearing in a window), with the context of every window sensitive
to the class being displayed. The effect is basically an
"irb-anywhere" which you can arbitrarily interleave with any number of
visual inspects. That, and a self-hosted language, faster VM, more
garbage collection options, etc...

 

[Lothar Scholz]

Hello Peter,

PS> to the class being displayed. The effect is basically an
PS> "irb-anywhere" which you can arbitrarily interleave with any number of

So it can crash anywhere SCNR.

Yes maybe its possible to transfer part of the language (all?) into an
smalltalk VM. But you can never get all the binary extensions to work,
an emulation layer would kill all your speed benefits if possible
at all. So maybe you should simply announce a ruby branch called SmallRuby,
since this would not have so much todo with the current ruby anymore.

But i'm courious whats your result in the next years.
Is this your Ph.D thesis ?

 

[Glenn Parker]

So, what are the potential platforms for Ruby on a Smalltalk VM?

 

[Thomas E Enebo]

Yes maybe its possible to transfer part of the language (all?) into an
smalltalk VM. But you can never get all the binary extensions to work,
an emulation layer would kill all your speed benefits if possible
at all. So maybe you should simply announce a ruby branch called SmallRuby,
since this would not have so much todo with the current ruby anymore.

OTOH, if it is 30x faster and makes an alternate implementation of socket
(which likely would not have any speed deficiency) and a few other extensions
it may become an attractive branch to use for many people.

Seems like an idea worth pursuing to me...Proofs in the pudding though.
It will be interesting to see what gains materialize...

-Tom

 

[Eric Hodel]

--Apple-Mail-5--686032734
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII; format=flowed

Yes maybe its possible to transfer part of the language (all?) into an
smalltalk VM. But you can never get all the binary extensions to work,
an emulation layer would kill all your speed benefits if possible
at all. So maybe you should simply announce a ruby branch called
SmallRuby,

Wow, could you possibly be more negative?

This kind of dismissive overlording behavior does not reflect the usual
spirit of acceptance and togetherness found on this list, especially
since so many projects of the same spirit are in progress at the same
time (YARV, Rubidium, JRuby, MetaRuby and a few less-publicized ones).

This is a time for offering constructive criticism and helpful insight
into the difficult problems ahead, not a time to say things are
impossible!

--
Eric Hodel - (e-mail address removed) - http://segment7.net
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

--Apple-Mail-5--686032734
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iEYEARECAAYFAkJVsv0ACgkQMypVHHlsnwSl7QCgiWPtrQE1fpi1gMCszr7493Ri
xUgAoL+NE847GDe5uL0Um6TCSVri/Yij
=s6kT
-----END PGP SIGNATURE-----

--Apple-Mail-5--686032734--

 

[Peter Suk]

Hello Peter,

PS> to the class being displayed. The effect is basically an
PS> "irb-anywhere" which you can arbitrarily interleave with any
number of

So it can crash anywhere SCNR.

In general this sort of thing doesn't crash under Smalltalk. In fact,
the debugger is just a normal application doing a few funky things with
Processes. It's just a browser for the reified context stack. In
fact, you can write your own feature-poor, but working debugger in
Smalltalk and have it running in a matter or minutes.

Yes maybe its possible to transfer part of the language (all?) into an
smalltalk VM.

Not what I'm trying to do. The Smalltalk VM has basically *none* of
the *language* in it.

But you can never get all the binary extensions to work,

Why not? There are plenty of binary extensions to the various
Smalltalks. They are designed to make it easy to build in binary
extensions.

an emulation layer would kill all your speed benefits if possible
at all. So maybe you should simply announce a ruby branch called
SmallRuby,
since this would not have so much todo with the current ruby anymore.

You missed something in my previous posts, or do not have some pieces
of background concerning how Smalltalk works. There is no emulation
going on, except stepping during debugging, and that happens at the
level of the bytecodes so all VM/Object semantics are preserved. Ruby
will be running on top of a Just-In-Time compiler, translating the
bytecodes into native machine instructions. The VisualWorks JIT has
been in development since 1989, so is one of the most mature JIT
interpreters around. Its speed rivals Hotspot in real-world situations
where GC is significant. Object allocation will be native -- Objects
will just be Objects to the VM/garbage collector, with the
Ruby/Smalltalk implementation language being moot.

But i'm courious whats your result in the next years.
Is this your Ph.D thesis ?

No. I want to make the world safe for Pure-OO. A way I can help
Pure-OO development is to vastly increase the power of the Ruby
community by giving it access to the great technology developed for
Smalltalk.

Other things that putting Ruby on top of Smalltalk VMs could garner for
the Ruby community:

The Smalltalk MT environment can produce native Windows DLLs that are
indistinguishable from C++ DLLs. We would theoretically be able to do
the same for Ruby when we can host Ruby on that VM. There could be a
Gemstone Ruby -- with a name like that there *should* be -- giving the
community access to a great OODB. There is also a fast Smalltalk VM
for .NET which could be used for Ruby. If we can overcome the lack of
real Namespaces in Squeak, then we'd have a Ruby VM that runs
completely bit-identical on 32 platforms.

Also, I'd like to do automated syntax + type-inference Ruby <-->
Smalltalk translations so that Ruby can benefit from the wealth of
great stuff done in Smalltalk, and Smalltalkers can be invigorated by
contact with a larger, very active community.

--Peter

 

[Peter Suk]

So, what are the potential platforms for Ruby on a Smalltalk VM?

Squeak alone has been ported to 32 platforms, running in all of them
bit-identically, including Acorn Risc, WinCE, Mac OSX, Mac System 9,
Windows, Linux, various Unixes. There is a Smalltalk VM for PalmOS.
(slow, though) The first target, VisualWorks, runs on 8 platforms,
including Various Windows, WinCE, Mac 8, Mac 9, Mac OSX, HPUX, AIX,
IRIX, Solaris-SPARC, Linux x86, Linux SPARC, Linux PPC.

--Peter

 

[Avi Bryant]

Hello everyone,

I thought I'd talk about my new project here, since there is a good
chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs.

Hi Peter,

I'm definitely interested in this; please keep me posted. Actually, a
couple of people are aware that I was looking into the possibility of
doing this as a commercial product - since, from my benchmarking, we
can get about a 20x to 30x speed increase by hosting Ruby on a
commercial Smalltalk VM like VisualWorks, it seemed like it might be
worthwhile. However, the interest wasn't great enough for me to pursue
it, so I'm very glad that someone is.

A small note: you say "at least those that have namespaces", but
there's no need to be concerned about that. Namespaces are an
image-level thing, not a VM-level thing; any Smalltalk VM worth its
salt can support namespaces just fine (and in fact constructs like pool
variables are effectively namespaces already, they just have somewhat
different conventions than Ruby's).

Avi

 

[Phil Tomson]

OTOH, if it is 30x faster and makes an alternate implementation of socket

I'm very skeptical about the 30x faster claim.

Phil

 

[Phil Tomson]

Hi Peter,

I'm definitely interested in this; please keep me posted. Actually, a
couple of people are aware that I was looking into the possibility of
doing this as a commercial product - since, from my benchmarking, we
can get about a 20x to 30x speed increase by hosting Ruby on a
commercial Smalltalk VM like VisualWorks, it seemed like it might be

This "20x to 30x speed increase" stuff is interesting. I would have
thought it would be closer to something like 5x. Can anyone elaborate on
how those increases are being acheived in the SmallTalk VMs? I'm wondering
if it might also be worthwhile to incorporate some of these ideas into YARV -
parhaps that would get us to high-speed Ruby even faster than trying to
put Ruby on top of a SmallTalk VM?

Phil