[ANN] slave-1.1.0

[Ara.T.Howard]

SYNOPSIS

the Slave class forks a process and starts a drb server in the child using
any object as the server. the process is detached so it is not required
(nor possible) to wait on the child pid. a Heartbeat is set up between the
parent and child processes so that the child will exit of the parent exits
for any reason - preventing orphaned slaves from running indefinitely. the
purpose of Slaves is to be able to easily set up a collection of objects
communicating via drb protocols instead of having to use IPC.

typical usage:

slave = Slave::new{ AnyObject.new }

slave.object #=> handle on drb object
slave.uri #=> uri of the drb object
slave.socket #=> unix domain socket path for drb object
slave.psname #=> title shown in ps/top

object = slave.object

value = object.any_method #=> use the object normally

slaves may be configured via the environment, the Slave class, or via the
ctor for object itself. attributes which may be configured include

* object : specify the slave object. otherwise block value is used.
* socket_creation_attempts : specify how many attempts to create a unix domain socket will be made
* debug : turn on some logging to STDERR
* psname : specify the name that will appear in 'top' ($0)
* at_exit : specify a lambda to be called in the *parent* when the child dies
* dumped : specify that the slave object should *not* be DRbUndumped (default is DRbUndumped)
* threadsafe : wrap the slave object with ThreadSafe to implement gross thread safety

URIS

http://rubyforge.org/projects/codeforpeople/
http://codeforpeople.com/lib/ruby/slave

HISTORY

1.1.0:
- replaced HeartBeat class with LifeLine.

- __HUGE__ cleanup of file descriptor/fork management with tons of help
from skaar and ezra. thanks guys!

- introduced Slave.object method used to return any object directory from
a child process. see samples/g.rb.

- indroduced keyword to automatically make slave objects threadsafe.
remember that your slave object must be threadsafe because they are
being served via DRb!!!

SAMPLES

<========< samples/a.rb >========>

~ > cat samples/a.rb

require 'slave'
#
# simple usage is simply to stand up a server object as a slave. you do not
# need to wait for the server, join it, etc. it will die when the parent
# process dies - even under 'kill -9' conditions
#
class Server
def add_two n
n + 2
end
end

slave = Slave.new bject => Server.new

server = slave.object
p server.add_two(40) #=> 42

slave.shutdown

~ > ruby samples/a.rb

42


<========< samples/b.rb >========>

~ > cat samples/b.rb

require 'slave'
#
# if certain operations need to take place in the child only a block can be
# used
#
class Server
def connect_to_db
"we only want to do this in the child process!"
@connection = ostgresql
end
attr :connection
end

slave = Slave.new('object' => Server.new){|s| s.connect_to_db}

server = slave.object

p server.connection #=> ostgresql
#
# errors in the child are detected and raised in the parent
#
slave = Slave.new('object' => Server.new){|s| s.typo} #=> raises an error!

~ > ruby samples/b.rb

ostgresql
./lib/slave.rb:460:in `initialize': undefined method `typo' for #<Server:0xb7565694> (NoMethodError)
from samples/b.rb:22:in `new'
from samples/b.rb:22


<========< samples/c.rb >========>

~ > cat samples/c.rb

require 'slave'
#
# if no slave object is given the block itself is used to contruct it
#
class Server
def initialize
"this is run only in the child"
@pid = Process.pid
end
attr 'pid'
end

slave = Slave.new{ Server.new }
server = slave.object

p Process.pid
p server.pid # not going to be the same as parents!
#
# errors are still detected though
#
slave = Slave.new{ fubar } # raises error in parent

~ > ruby samples/c.rb

14387
14388
./lib/slave.rb:460:in `initialize': undefined local variable or method `fubar' for main:Object (NameError)
from samples/c.rb:21:in `new'
from samples/c.rb:21


<========< samples/d.rb >========>

~ > cat samples/d.rb

require 'slave'
#
# at_exit hanlders are handled correctly in both child and parent
#
at_exit{ p 'parent' }
slave = Slave.new{ at_exit{ p 'child' }; 'the server is this string' }
#
# this will print 'child', then 'parent'
#

~ > ruby samples/d.rb

"child"
"parent"


<========< samples/e.rb >========>

~ > cat samples/e.rb

require 'slave'
#
# slaves never outlive their parent. if the parent exits, even under kill -9,
# the child will die.
#
slave = Slave.new{ at_exit{ p 'child' }; 'the server is this string' }

Process.kill brutal=9, the_parent_pid=Process.pid
#
# even though parent dies a nasty death the child will still print 'child'
#

~ > ruby samples/e.rb

"child"


<========< samples/f.rb >========>

~ > cat samples/f.rb

require 'slave'
#
# slaves created previously are visible to newly created slaves - in this
# example the child process of slave_a communicates directly with the child
# process of slave_a
#
slave_a = Slave.new{ Array.new }
slave_b = Slave.new{ slave_a.object }

a, b = slave_b.object, slave_a.object

b << 42
puts a #=> 42

~ > ruby samples/f.rb

42


<========< samples/g.rb >========>

~ > cat samples/g.rb

require 'slave'
#
# Slave.object can used when you want to construct an object in another
# process. in otherwords you want to fork a process and retrieve a single
# returned object from that process as opposed to setting up a server.
#
this = Process.pid
that = Slave.object{ Process.pid }

p 'this' => this, 'that' => that

#
# any object can be returned and it can be returned asychronously via a thread
#
thread = Slave.objectasync => true){ sleep 2 and [ Process.pid, Time.now ] }
this = [ Process.pid, Time.now ]
that = thread.value

p 'this' => this, 'that' => that

~ > ruby samples/g.rb

{"that"=>14406, "this"=>14405}
{"that"=>[14407, Tue Nov 28 09:47:31 MST 2006], "this"=>[14405, Tue Nov 28 09:47:29 MST 2006]}


enjoy.

-a

 

[Tim Pease]

SYNOPSIS

the Slave class forks a process and starts a drb server in the child using
any object as the server. the process is detached so it is not required
(nor possible) to wait on the child pid. a Heartbeat is set up between the
parent and child processes so that the child will exit of the parent exits
for any reason - preventing orphaned slaves from running indefinitely. the
purpose of Slaves is to be able to easily set up a collection of objects
communicating via drb protocols instead of having to use IPC.

Does Slave work on the Win32 platform, or is this a UNIX only gem?

TwP

 

[cremes.devlist]

Does Slave work on the Win32 platform, or is this a UNIX only gem?

Slave is UNIX only since it relies on fork(). I've looked at the
code. Ara does a nice job of keeping the fork stuff encapsulated. If
you have a Win32 mechanism that could be bolted on to Slave and
replace fork(), you'd get the rest of the code for free.

cr

 

[ara.t.howard]

Does Slave work on the Win32 platform, or is this a UNIX only gem?

what is this "windows" you speak of?

;-)

-a

 

[ara.t.howard]

Does Slave work on the Win32 platform, or is this a UNIX only gem?

in all seriousness i'd be happy if it could be made to work - have a look and
let me know what you think if you get a chance.

-a

 

[Tim Pease]

in all seriousness i'd be happy if it could be made to work - have a look and
let me know what you think if you get a chance.

I think Daniel Berger is your man for getting this to work on Windows.
Just one of those things that would be "nice to have" in the future.

TwP

 

[Daniel Berger]

I think Daniel Berger is your man for getting this to work on Windows.
Just one of those things that would be "nice to have" in the future.

It's probably doable, but I'll have to look at the source code.

I don't know much about slave, but one question I have right off the
bat is why you would want one drb server per object, instead of just
one drb server that stored objects in a hash table that you could push
and pop at will. Maybe I need to go back and see Ara's use case for
this...

Regards,

Dan

 

[Tim Pease]

It's probably doable, but I'll have to look at the source code.

I don't know much about slave, but one question I have right off the
bat is why you would want one drb server per object, instead of just
one drb server that stored objects in a hash table that you could push
and pop at will. Maybe I need to go back and see Ara's use case for
this...

Slave forks off a ruby object in a separate process and then uses DRb
for communication between the processes. Each forked process has its
own DRb server.

TwP

 

[Daniel Berger]

Tim Pease wrote:

Slave forks off a ruby object in a separate process and then uses DRb
for communication between the processes. Each forked process has its
own DRb server.

TwP

Yes, but why? I assume the purpose is IPC. If so, again I ask why you
would need 1 drb server per object instead of one drb server with lots
of objects. If its purspose isn't IPC, what is it?

Thanks,

Dan

 

[ara.t.howard]

Tim Pease wrote:



Yes, but why? I assume the purpose is IPC. If so, again I ask why you
would need 1 drb server per object instead of one drb server with lots
of objects. If its purspose isn't IPC, what is it?

the purpose is two-fold:

1) easy ipc
2) scalling expensive operations across multiple processors easily

you certainly could setup a hashtable of objects. but a few things to
consider

1) drop-in scalibility

object = ExpensiveCalcuation.new

# 1000 lines of code

change to

object = Slave.new{ ExpensiveCalcuation.new }

# exact same 1000 lines of code

and, whammo. you scale to multi-processor machines.


2) DRbUndumped nuances

let's say you build a table of objects like this:


harp:~ > cat a.rb
require "slave"
require "yaml"

slave = Slave.new{ Hash.new }
table = slave.object

array = []

table["array"] = array

table["array"] << Process.pid

y "slave.pid" => slave.pid
y "array" => array
y "table['array']" => table["array"]


what will happen?


harp:~ > ruby a.rb
slave.pid: 5258
array: []
table['array']: []


yikes!!! what went wrong? well, consider:

array = [] # array is in parent

table["array"] = array # array is marshaled across to child as a copy

table["array"] << Process.pid # array is marshaled back across to parent and this __local_copy__ is appended too

p "array" => array # local copy empty

p "table['hash']" => table["hash"] # remote copy empty


here is a fix

array = []
array.extend DRbUndumped # the important part


now, running again


harp:~ > ruby a.rb
slave.pid: 5302

array:
- 5301

table['array']:
- 5301


but look carefully this example is even __more__ fubar: the array lives in
the parent! we setup a table and sent only the proxy (DRbUndumped) across
to the child! we've completely negated the entire benefit of running
objects in another process - all we have is a lookup table for objects in
our own address space.


ok. take two. this time we'll try this:

harp:~ > cat a.rb
require "slave"
require "yaml"

slave = Slave.new{ # build table in child
table = Hash.new
array = []
table["array"] = array
table
}

table = slave.object

table["array"] << Process.pid

y "slave.pid" => slave.pid
y "table['array']" => table["array"]


we give a whirl:


harp:~ > ruby a.rb
slave.pid: 5358
table['array']: []


sigh. now our table exists only in the child, and so does all it's
elements. why doesn't it work. DRbUndumped is screwing us again. here's
why:


table["array"] << Process.pid # here the child marshals a copy back to
# parent. we append to the copy

y "table['array']" => table["array"] # and here we get a fresh __empty__ copy


dang. we know the fix though, in the child we do:

array.extend DRbUndumped


trying again

harp:~ > ruby a.rb
slave.pid: 5405
table['array']: !ruby/objectRb:RbObject
ref: -609755446
uri: drbunix:///tmp/hash_-609755436_5404_5405_0

wtf? well, since our object now lives in the child and we only get a drb
proxy back some things work, like '<<', but other methods, like 'to_yaml', and
'inspect' are handled locally by drb. to get around this 'to_yaml' bug we have
to force a copy


harp:~ > cat a.rb
require "slave"
require "yaml"

slave = Slave.new{ # build table in child
table = Hash.new
array = []
array.extend DRbUndumped
table["array"] = array
table
}

table = slave.object

table["array"] << Process.pid

y "slave.pid" => slave.pid
y "table['array']" => table["array"].map # force local copy, neither dup nor clone will cut it!



harp:~ > ruby a.rb
slave.pid: 5465
table['array']:
- 5464


ah, finally.


so, you see, managing a hierarchy of object across drb is tricky. not that it
can't be done - i just happen to think that managing __one__ object and it's
DRbUndumped nuances is about all most hackers can manage - thus that's the
default usage. nothing will stop you from making your server object a factory
or table though - it just gets confusing fast.


kind regards.

-a

 

[Bil Kleb]

what is this "windows" you speak of?

I think it might be what these GNU folks are talking about
in their coding standards document,

If you do support Windows, please do not abbreviate it as “win”.
In hacker terminology, calling something a “win” is a form of
praise. You're free to praise Microsoft Windows on your own
if you want, but please don't do this in GNU packages. Instead
of abbreviating “Windows” to “un”, you can write it in full or
abbreviate it to “woe” or “w”. In GNU Emacs, for instance, we
use ‘w32’ in file names of Windows-specific files, but the macro
for Windows conditionals is called WINDOWSNT.

http://www.gnu.org/prep/standards/standards.html#System-Portability

Later,

 

[ara.t.howard]

I think it might be what these GNU folks are talking about
in their coding standards document,

If you do support Windows, please do not abbreviate it as “win”.
In hacker terminology, calling something a “win” is a form of
praise. You're free to praise Microsoft Windows on your own
if you want, but please don't do this in GNU packages. Instead
of abbreviating “Windows” to “un”, you can write it in full or
abbreviate it to “woe” or “w”. In GNU Emacs, for instance, we
use ‘w32’ in file names of Windows-specific files, but the macro
for Windows conditionals is called WINDOWSNT.

http://www.gnu.org/prep/standards/standards.html#System-Portability

that's __awesome__. more humour in docs i say!

-a