Fast way to move C array to ruby (rb_ary_new4)?

[Ville Sipola]

Good day

I have a number of large arrays in a C module, and I'd need to move them
to ruby as quickly as possible.

Currently I use the following type of an approach:

instance_array_in_ruby = rb_iv_get(calling_class, "@x");
double c_array[1000000];

for(i=0;i<1000000;i++)
{
rb_ary_store(instance_array_in_ruby, i, rb_float_new(c_array));
}

But that's slow, apparently due to the large number of rb_float_new:s.

There is a function called rb_ary_new4 that would seem to fit
the purpose of quickly moving an entire C array to ruby. Only I'm unable
to get it work, as it, too requires VALUE:s.
I've tried many things including mumbo-jumbo like
rb_iv_set(calling_class,"@x",rb_ary_new4(1000000, (VALUE)c_array));, but
I'm only managing to get segfaults or solutions that still need a
million rb_something_new:s.

Would you know an efficient way of moving a large data from C array to
ruby array?

 

[Robert Klemme]

Good day

I have a number of large arrays in a C module, and I'd need to move them
to ruby as quickly as possible.

Currently I use the following type of an approach:

instance_array_in_ruby =3D rb_iv_get(calling_class, "@x");
double c_array[1000000];

for(i=3D0;i<1000000;i++)
{
=A0 rb_ary_store(instance_array_in_ruby, i, rb_float_new(c_array));
}

But that's slow, apparently due to the large number of rb_float_new:s.

There is a function called rb_ary_new4 that would seem to fit
the purpose of quickly moving an entire C array to ruby. Only I'm unable
to get it work, as it, too requires VALUE:s.
I've tried many things including mumbo-jumbo like
rb_iv_set(calling_class,"@x",rb_ary_new4(1000000, (VALUE)c_array));, but
I'm only managing to get segfaults or solutions that still need a
million rb_something_new:s.

Would you know an efficient way of moving a large data from C array to
ruby array?

What about leaving it in C, i.e. wrapping in a specific type which
presents an Array (or Enumerable) like API and uses your C array
internally? A similar approach is taken by ENV (you can find the code
in hash.c).

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Robert Klemme]

What about leaving it in C, i.e. wrapping in a specific type which
presents an Array (or Enumerable) like API and uses your C array
internally? =A0A similar approach is taken by ENV (you can find the code
in hash.c).

PS: Of course whether that is more or less efficient than your
approach totally depends on the usage pattern. If only few items in
those arrays are accessed from Ruby land the wrapping approach is
likely cheaper. If all items are accessed in the majority of cases
then probably your original approach is better.

I'd also look into NArray which might help you here.

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Brian Candler]

Ville Sipola wrote in post #995293:

instance_array_in_ruby = rb_iv_get(calling_class, "@x");
double c_array[1000000];

for(i=0;i<1000000;i++)
{
rb_ary_store(instance_array_in_ruby, i, rb_float_new(c_array));
}

But that's slow, apparently due to the large number of rb_float_new:s.

I'm afraid that's bound to be the case, given that Floats are not
immediate values and therefore each one has to be allocated on the heap
(unless your data happens to have many instances of identical floats,
and could share the references)

Robert's wrapping sounds like the best way forward, perhaps with some
memoizing (i.e. lazy creation of Float objects)

It might also help a little if you pre-allocate the array:

a = Array.new(1000000)

For more of a hack, try:

a = Array.new(1000000) { 0.0+0.0 }

Then maybe you could walk the array and overwrite the RFloat structures
directly (ugh). Note that the 0.0+0.0 frig was required so that the same
object_id doesn't get used in each element, and that's not guaranteed to
work in future (e.g. if Ruby did more aggressive constant folding)

 

[Cameron McBride]

I'd also look into NArray which might help you here.

I second this suggestion. If you don't mind the NArray dependence,
this is extremely easy using NArray as it keeps it in the C array
structure.

Cameron

 

[Ville Sipola]

That's a lot of answers in a short time. Thank you all!

The kind of hack that Brian Candler mentioned (writing directly to
struct RFloat) was one of the ways that I already tried to accomplish,
but if it's likely to get broken in future, then I guess it's not a
solution to fight for.

And for Robert's second answer: yes, I need the whole data back, I'm
sending it all to other functions.

I guess I'll try them all, starting with the NArray.

Now I only have to figure out how to use that. From narray.c I get the
impression that they can be read into C by creating a struct NARRAY and
then using the command GetNArray, right? But how would you write into it
in C?

Thank you for the help.

Ville

 

[Ville Sipola]

Now I only have to figure out how to use that.

Now that I have, I thought it a good idea to describe how it was done,
so that others reading the post might be able to use the information.

So this was my method.

The writing part is a bit of a hack. Masahiro Tanaka himself advices
against this approach in here: https://www.ruby-forum.com/topic/83484,
but I'm yiet to find a better solution. If there is a function for
writing c data in an narray in a safe way, I'd like to hear about it.

First, in Ruby I did something like

@x=NArray.float(@number_of_samples).fill(some_value)

to make sure that the narray is of correct size.

Then in C:
static VALUE writingfunction(VALUE class,VALUE x)
{
struct NARRAY *xarray;
GetNArray(x, xarray);
double *floatpointerx = xarray->ptr;

double c_value;

for (counter = 0; counter < number_of_samples; counter++)
{
(do the actual calculation)
floatpointerx[samplecounter] = c_value; (This is the illegal
hack)
}
}

To read in C (this I believe is completely ok, please correct if I'm
wrong.)

static VALUE readingfunction(VALUE class, VALUE x)
{
struct NARRAY *xarray;
GetNArray(x, xarray);
double *floatpointerx = xarray->ptr;
for (counter = 0; counter < xarray->total; counter++)
{
a_function_that_takes_c_double(floatpointerx[counter1]);
}
}