Extension question

[Joe Van Dyk]

Hi,

I'm generating a fairly large image file (can be 10000x10000) pixels
by reading some binary data and setting each pixel. As you can
imagine, pure Ruby is REALLY slow, I think because of the loops. I've
written the program in C and it's literally hundreds to thousands of
times faster (depending on image size).

So, I'd like to write my first C extension. I'm using the GD library
for image manipulation.

Are there any general rules of thumb that you use when writing extensions?

My first problem was trying to figure out what should remain Ruby code
and what should be moved out to C. For example, if I opened the file
that contained binary data in Ruby, how could I "get" to that data
using a FILE pointer in C? Or should I just pass the filename to the
C extension?

Also, I couldn't really figure out how to only have one function in C,
the documentation seemed to say that I needed to have a class?

Thanks,
Joe

 

[Tom Copeland]

Hi,

I'm generating a fairly large image file (can be 10000x10000) pixels
by reading some binary data and setting each pixel. As you can
imagine, pure Ruby is REALLY slow, I think because of the loops. I've
written the program in C and it's literally hundreds to thousands of
times faster (depending on image size).

So, I'd like to write my first C extension. I'm using the GD library
for image manipulation.

You might be interested in this:

http://raa.ruby-lang.org/list.rhtml?name=ruby-gd

From the example scripts:

require "GD"
im = GD::Image.new(100,100)
red = im.colorAllocate(255,0,0)
im.rectangle(0,0,99,99,red)
im.png STDOUT

Yours,

tom

 

[Tom Copeland]

Also, I couldn't really figure out how to only have one function in C,
the documentation seemed to say that I needed to have a class?

You can attach the function to Object:

===============================
$ cat example.c
#include <stdio.h>
#include "ruby.h"
static VALUE test(VALUE self) {
printf("HI!\n");
return Qnil;
}
void Init_example() {
rb_define_method(rb_cObject, "test", test, 0);
}
$ cat extconf.rb
require 'mkmf'
create_makefile("example")
$ ruby extconf.rb && make
[... some gcc output ...]
$ irb irb
irb(main):001:0> require 'example'
=> true
irb(main):002:0> test
HI!
=> nil
irb(main):003:0>
===============================

Yours,

Tom

 

[Tom Copeland]

My first problem was trying to figure out what should remain Ruby code
and what should be moved out to C. For example, if I opened the file
that contained binary data in Ruby, how could I "get" to that data
using a FILE pointer in C? Or should I just pass the filename to the
C extension?

I wonder about this as well. So far I've been sort of passing data
objects back and forth between C and Ruby, the same way that I would if
I were doing an out-of-process method call. But I think that's mostly
just because I'm an extensions new bee....

Yours,

Tom

 

[Joe Van Dyk]

You might be interested in this:

http://raa.ruby-lang.org/list.rhtml?name=ruby-gd


require "GD"
im = GD::Image.new(100,100)
red = im.colorAllocate(255,0,0)
im.rectangle(0,0,99,99,red)
im.png STDOUT

Yours,

Hi,

I was using Ruby-GD before. The slowness is not the library, but the
looping 10 million times. I have to set each pixel, so I can't use
the rectangle function.

 

[Robert Klemme]

You can attach the function to Object:

IMHO the appropriate place would be a private singleton method of Kernel
wouldn't it? At least all other "functions" (gsub, gsub! etc.) are placed
there...

Kind regards

robert

 

[Charles Mills]

Are there any general rules of thumb that you use when writing

extensions?

http://www.ruby-talk.org/cgi-bin/scat.rb/ruby/ruby-talk/47700

You shouldn't be able to cause your extension to seg fault from Ruby.
That may sound like a joke, but I just mean using StringValue,
NUM2LONG, etc before you starting using Ruby objects in C.

My first problem was trying to figure out what should remain Ruby code
and what should be moved out to C. For example, if I opened the file
that contained binary data in Ruby, how could I "get" to that data
using a FILE pointer in C? Or should I just pass the filename to the
C extension?

You can do either. IMO it depends on the layout of the file. If your
reading line by line you can do this easily using the rb_io_* functions
found in intern.h. If your going to be moving the file position
around, using fgetc/ungetc, or reading into C structs it may be easier
to use the stdio functions directly.

-Charlie

 

[Tom Copeland]

I was using Ruby-GD before.

Ah, OK, so you were already familiar with that one, sorry.

Yours,

Tom

 

[Tom Copeland]

IMHO the appropriate place would be a private singleton method of Kernel
wouldn't it? At least all other "functions" (gsub, gsub! etc.) are placed
there...

Right you are, that's much better...

Yours,

Tom

 

[Tim Hunter]

Hi,

I'm generating a fairly large image file (can be 10000x10000) pixels
by reading some binary data and setting each pixel. As you can
imagine, pure Ruby is REALLY slow, I think because of the loops. I've
written the program in C and it's literally hundreds to thousands of
times faster (depending on image size).

Have you tried RMagick? http://rmagick.rubyforge.org.

The constitute method may be useful:
http://www.simplesystems.org/RMagick/doc/image1.html#constitute

That method requires you to have all the pixel data in memory at once,
though, which could be a problem unless you have a LOT of RAM.

You could use import_pixels to set just a subset of pixels at a time:
http://www.simplesystems.org/RMagick/doc/image2.html#import_pixels

That would minimize the memory requirements.

In both cases you'd have to convert your binary data to a Ruby numeric
type. Don't know whether that would be too slow or not.

 

[Charles Mills]

Right you are, that's much better...

Yours,

Tom

you can also put them in a module quite easily:

void
Init_myext(void)
{
VALUE m = rb_define_module("M");
rb_define_singleton_method(m, "test", some_func, -1);
}

now you have M.test()

-Charlie

 

[Joe Van Dyk]

extensions?

http://www.ruby-talk.org/cgi-bin/scat.rb/ruby/ruby-talk/47700

You shouldn't be able to cause your extension to seg fault from Ruby.
That may sound like a joke, but I just mean using StringValue,
NUM2LONG, etc before you starting using Ruby objects in C.

You can do either. IMO it depends on the layout of the file. If your
reading line by line you can do this easily using the rb_io_* functions
found in intern.h. If your going to be moving the file position
around, using fgetc/ungetc, or reading into C structs it may be easier
to use the stdio functions directly.

-Charlie

So, if I have a IO object in Ruby, how could I use C's stdio functions
on it? And how could I have figured it out without asking the mailing
list? Cuz I'm sure I'll have lots more questions.

Thanks,
Joe

 

[Joe Van Dyk]

you can also put them in a module quite easily:

void
Init_myext(void)
{
VALUE m = rb_define_module("M");
rb_define_singleton_method(m, "test", some_func, -1);
}

now you have M.test()

-Charlie

Yeah, that's probably what I'm going to do.

 

[Tim Hunter]

So, if I have a IO object in Ruby, how could I use C's stdio functions
on it? And how could I have figured it out without asking the mailing
list? Cuz I'm sure I'll have lots more questions.

Check out the GetOpenFile, GetWriteFile, etc. macros in rubyio.h, and
the rb_io_xxxxx functions in io.h.

I found these by perusing the Ruby source code and asking questions on
this list.

 

[Joe Van Dyk]

So, if I have a IO object in Ruby, how could I use C's stdio functions
on it? And how could I have figured it out without asking the mailing
list? Cuz I'm sure I'll have lots more questions.

Thanks,
Joe

And to expand the question...

I'm using the GD library to create the image. I'm creating the image
by reading binary data from a file and looping through some
predetermined width and height and setting each pixel of the picture
based on the data from the file.

Which parts of this should go in the C extension and which parts
should stay Ruby? I'm really scratching my head over this one.

Thanks,
Joe

 

[Joe Van Dyk]

Have you tried RMagick? http://rmagick.rubyforge.org.

The constitute method may be useful:
http://www.simplesystems.org/RMagick/doc/image1.html#constitute

That method requires you to have all the pixel data in memory at once,
though, which could be a problem unless you have a LOT of RAM.

You could use import_pixels to set just a subset of pixels at a time:
http://www.simplesystems.org/RMagick/doc/image2.html#import_pixels

That would minimize the memory requirements.

In both cases you'd have to convert your binary data to a Ruby numeric
type. Don't know whether that would be too slow or not.

Ooh, that looks nice. The image can be up to (and probably more than)
10,000x10,000 pixels though... that's a heck of a lot of memory. And
converting the data in Ruby would take a long time, I believe. Too
many iterations.

 

[Steven Jenkins]

And to expand the question...

I'm using the GD library to create the image. I'm creating the image
by reading binary data from a file and looping through some
predetermined width and height and setting each pixel of the picture
based on the data from the file.

Which parts of this should go in the C extension and which parts
should stay Ruby? I'm really scratching my head over this one.

If you're writing the extension to solve a performance problem, then I
recommend the general approach of putting in the extension *only* what
is required to solve the performance problem.

If your Ruby code doesn't need to do anything with the file itself, I'd
just hand the filename to the extension and let it
open/read/write/close, whatever. Alternately, you could ope the file in
Ruby and use IO#fileno to pass the open file descriptor to the
extension. Then you could operate on that fd with read()/write()/etc.
Doing actual IO in both Ruby and the extension may require care.

Steve

 

[Joe Van Dyk]

extensions?

http://www.ruby-talk.org/cgi-bin/scat.rb/ruby/ruby-talk/47700

You shouldn't be able to cause your extension to seg fault from Ruby.
That may sound like a joke, but I just mean using StringValue,
NUM2LONG, etc before you starting using Ruby objects in C.

How should errors be handled in C extensions? For example, my C
extension was expecting a Ruby string, but it was passed an IO object.

Or, what should I do when I try to open a file in C (the filename
being passed to the function) and the file doesn't exist?

 

[Charles Mills]

You can't.

Although you could get the FILE ptr from the Ruby IO Object it is
definately not a good idea. Also the next version of Ruby will not use
stdio at all, so you code will break.

So basically you can do one or the other - use a Ruby IO Object (and
the rb_io_* methods) or use fopen and friends.

And to expand the question...

I'm using the GD library to create the image. I'm creating the image
by reading binary data from a file and looping through some
predetermined width and height and setting each pixel of the picture
based on the data from the file.

Which parts of this should go in the C extension and which parts
should stay Ruby? I'm really scratching my head over this one.

Thanks,
Joe

Seems like you already decided the looping should be done in C. Seems
like you have two options:
1) Open the file and read a buffer (String) in Ruby using IO#read and
pass it to your C extension which then loops through the buffer and
does the pixel stuff.
2) Open the file and read a buffer in C, then loop though it.
(Probably should use fread() to read the buffer since there is no
rb_io_read function, but if you wanted you could use a Ruby IO object
and rb_funcall.)

Option one seems more modular, but it sounds like the size of the
buffer and the range of the loop are coupled so maybe not.
Option two will probably be faster because if you can reuse the buffer
and / or declare it on the stack.

-Charlie

 

[Tim Hunter]

How should errors be handled in C extensions? For example, my C
extension was expecting a Ruby string, but it was passed an IO object.

Generally the Ruby functions that convert Ruby types to C types will
raise an exception if they're given invalid input.

Or, what should I do when I try to open a file in C (the filename
being passed to the function) and the file doesn't exist?

Call rb_raise()

Check out chapter 21 in the new Pickaxe.