sharing memory in ruby

[Ara.T.Howard]

since you seem to be online matz ;-)

any thoughts on a good approach for people that want to do some memory sharing
between ruby objects? say strings and mmaps and narrays and images all
sharing some memory regions. any do's and dont's or other thoughts?

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[Yukihiro Matsumoto]

Hi,

In message "Re: sharing memory in ruby"

|any thoughts on a good approach for people that want to do some memory sharing
|between ruby objects? say strings and mmaps and narrays and images all
|sharing some memory regions. any do's and dont's or other thoughts?

I don't thing you need to share memory within same processes. Just
refer to a same object. mmap seems reasonable solution to share
memory between processes. The another idea is using some kind of RPC,
such as dRuby instead of sharing memory.

matz.

 

[Ara.T.Howard]

Hi,

In message "Re: sharing memory in ruby"

|any thoughts on a good approach for people that want to do some memory sharing
|between ruby objects? say strings and mmaps and narrays and images all
|sharing some memory regions. any do's and dont's or other thoughts?

I don't thing you need to share memory within same processes. Just
refer to a same object. mmap seems reasonable solution to share
memory between processes. The another idea is using some kind of RPC,
such as dRuby instead of sharing memory.

attm. i'm writing some code to process images for katrina, check out

http://dmsp.ngdc.noaa.gov/interest/katrina.html

i sometimes must shared memory between an mmap, ruby string, and an narray to
i can work on it in memory. our machines have 8gb, but a couple channels and
some duplicate data and i run out... bascially i have an extension to narray
that allows it to share it's storage with an mmap in the same process but,
apparently, this is frowned upon? bascially i allow any string type object to
be passed to an narray and it uses that string's memory without copying it -
in this way i can

na = NArray::str mmap.to_s, samples, lines, NArray::SINT

any comments on this approach?

my narray.c patch is below...

the really critical bit is this:

+ ary->ptr = RSTRING(str)->ptr;
+ ary->ref = Qfalse;

the 'ary->ref' means it will not be free'd...

so now the NArray and String, which is itself an Mmap are all sharing memory.
the neat thing is that i can do this:

na = na + 1

and a the file backing is affected with no io on my part. also i can load
image larger than memory and, therfore, many large images at once. is this a
terrible idea? if not what alternative might i use? perhaps a generic
Pointer (shudder) object that behaves like a String might be useful...

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

--- narray.c.org 2004-10-15 20:57:34.000000000 -0600
+++ narray.c 2004-10-15 21:40:21.000000000 -0600
@@ -757,6 +757,54 @@


/* singleton method:
+ NArray.str( string, type, size1,size2,...,sizeN )
+*/
+static VALUE
+ na_s_str(int argc, VALUE *argv, VALUE klass)
+{
+ struct NARRAY *ary;
+ VALUE v;
+ VALUE str;
+ int i, type, len=1, str_len, *shape, rank=argc-2;
+
+ if (argc < 1)
+ rb_raise(rb_eArgError, "String Argument required");
+
+ if (argc < 2)
+ rb_raise(rb_eArgError, "Type and Size Arguments required");
+
+ type = na_get_typecode(argv[1]);
+
+ str = rb_funcall(argv[0], rb_intern("to_str"), 0);
+
+ str_len = RSTRING(str)->len;
+
+ if (argc == 2) {
+ rank = 1;
+ shape = ALLOCA_N(int,rank);
+ if ( str_len % na_sizeof[type] != 0 )
+ rb_raise(rb_eArgError, "string size mismatch");
+ shape[0] = str_len / na_sizeof[type];
+ }
+ else {
+ shape = ALLOCA_N(int,rank);
+ for (i=0; i<rank; i++)
+ len *= shape = NUM2INT(argv[i+2]);
+ len *= na_sizeof[type];
+ if ( len != str_len )
+ rb_raise(rb_eArgError, "size mismatch");
+ }
+
+ v = na_make_object( type, rank, shape, cNArray );
+ GetNArray(v,ary);
+ ary->ptr = RSTRING(str)->ptr;
+ ary->ref = Qfalse;
+
+ return v;
+}
+
+
+/* singleton method:
NArray[object]
*/
static VALUE
@@ -1189,6 +1237,7 @@
rb_define_singleton_method(cNArray,"to_na",na_s_to_na,-1);
rb_define_singleton_method(cNArray,"to_narray",na_s_to_na,-1);
rb_define_singleton_method(cNArray,"[]",na_s_bracket,-1);
+ rb_define_singleton_method(cNArray,"str",na_s_str,-1);

/* methods */
rb_define_method(cNArray, "[]", na_aref,-1);

 

[Masahiro TANAKA]

Hi,

Ara wrote:
+ ary->ref = Qfalse;

the 'ary->ref' means it will not be free'd...

This is not what I have intended.
'ary->ref' is an object that is referred from NArray object
so that the original String object is not garbage-collected.
So here should be `ary->ref = str;'.

+ ary->ptr = RSTRING(str)->ptr;

This has a serious caveat.

s = "0000"
na = NArray.str(s,2,NArray::SINT)
s << "1"*100 # The memory of 's' will be reallocated.
puts na[0] # accesses an invalid memory.

However, I think it would be quite useful if Ruby have something like
a common memory framework, which Ara suggested, for handling huge memory.

Masahiro Tanaka

 

[Yukihiro Matsumoto]

Hi,

In message "Re: sharing memory in ruby"

|However, I think it would be quite useful if Ruby have something like
|a common memory framework, which Ara suggested, for handling huge memory.

Perhaps. If you have any requirement/proposal/suggestion for that
kind of memory framework, I'd love to hear. I am not the right person
to make up the memory sharing, since I am no expert on the field.

matz.

 

[Ara.T.Howard]

Hi,


This is not what I have intended.
'ary->ref' is an object that is referred from NArray object
so that the original String object is not garbage-collected.
So here should be `ary->ref = str;'.

hmmm. i did not understand that. i'll make the mod and report back...
thanks for the insight.

+ ary->ptr = RSTRING(str)->ptr;

This has a serious caveat.

s = "0000"
na = NArray.str(s,2,NArray::SINT)
s << "1"*100 # The memory of 's' will be reallocated.
puts na[0] # accesses an invalid memory.

yes i'm aware of that. but, in my case, the string is actually a memory map
using guy's mmap and i hold it as a private variable. true, it could be
changed, but in fact it never is since the map is pinned to a specific
region/length of memory and i never tough the mmap object. i think the danger
is worth it since programs which normally could not even run run easily and
can be written in only a few lines. so, in my case the alternative has a
serious caveat - namely NoMemoryError ;-) it __is__ important to realize
this danger however.

However, I think it would be quite useful if Ruby have something like a
common memory framework, which Ara suggested, for handling huge memory.

any thoughts on what the ruby would look like?

kind regards.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

 

[Ara.T.Howard]

Hi,


This is not what I have intended.
'ary->ref' is an object that is referred from NArray object
so that the original String object is not garbage-collected.
So here should be `ary->ref = str;'.

hi masa-

here's the result:

jib:~/build/narray-0.5.7p2 > cat a.rb
#
# the narray lib below is patched with your suggested mod
#
require 'mmap'
require './narray'

data, format = './data', 'c*'
#
# create a file
#
open(data, 'w'){|f| f.write [42,0,0,42].flatten.pack(format)}
#
# map in in
#
mmap = Mmap::new 'data', 'rw', Mmap::MAP_SHARED
#
# modify mmap via mmap
#
na = NArray::str mmap.to_str, NArray::BYTE
na[] = 42
#
# original backing file is changed
#
p IO::read(data).unpack(format)


jib:~/build/narray-0.5.7p2 > ruby a.rb
[42, 42, 42, 42]


looks good. how would you feel about making a release with the patch?

it's inlined below.

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze. --Nagarjuna
===============================================================================

--- narray.c.org 2004-10-15 20:57:34.000000000 -0600
+++ narray.c 2004-10-15 21:40:21.000000000 -0600
@@ -757,6 +757,54 @@


/* singleton method:
+ NArray.str( string, type, size1,size2,...,sizeN )
+*/
+static VALUE
+ na_s_str(int argc, VALUE *argv, VALUE klass)
+{
+ struct NARRAY *ary;
+ VALUE v;
+ VALUE str;
+ int i, type, len=1, str_len, *shape, rank=argc-2;
+
+ if (argc < 1)
+ rb_raise(rb_eArgError, "String Argument required");
+
+ if (argc < 2)
+ rb_raise(rb_eArgError, "Type and Size Arguments required");
+
+ type = na_get_typecode(argv[1]);
+
+ str = rb_funcall(argv[0], rb_intern("to_str"), 0);
+
+ str_len = RSTRING(str)->len;
+
+ if (argc == 2) {
+ rank = 1;
+ shape = ALLOCA_N(int,rank);
+ if ( str_len % na_sizeof[type] != 0 )
+ rb_raise(rb_eArgError, "string size mismatch");
+ shape[0] = str_len / na_sizeof[type];
+ }
+ else {
+ shape = ALLOCA_N(int,rank);
+ for (i=0; i<rank; i++)
+ len *= shape = NUM2INT(argv[i+2]);
+ len *= na_sizeof[type];
+ if ( len != str_len )
+ rb_raise(rb_eArgError, "size mismatch");
+ }
+
+ v = na_make_object( type, rank, shape, cNArray );
+ GetNArray(v,ary);
+ ary->ptr = RSTRING(str)->ptr;
+ ary->ref = str;
+
+ return v;
+}
+
+
+/* singleton method:
NArray[object]
*/
static VALUE
@@ -1189,6 +1237,7 @@
rb_define_singleton_method(cNArray,"to_na",na_s_to_na,-1);
rb_define_singleton_method(cNArray,"to_narray",na_s_to_na,-1);
rb_define_singleton_method(cNArray,"[]",na_s_bracket,-1);
+ rb_define_singleton_method(cNArray,"str",na_s_str,-1);

/* methods */
rb_define_method(cNArray, "[]", na_aref,-1);