[ANN] FFI 0.1.1 (Foreign Function Interface) for Ruby 1.8.6/7 and1.9

[Charles Oliver Nutter]

We (JRuby and Rubinius contributors) designed in the ability to specify
calling convention for a given library bound through FFI. Largely this
was needed to support Win32's stdcall, but if Linux/PPC has a different
convention and libffi supports it, then FFi will too (though we may have
to add it to the list of accepted conventions).

I'll check on the syntax we're supporting and get back to ya.

In the body of the module you're binding:

ffi_convention :stdcall # :default is for normal C convention

We could add whatever convention Linux/PPC needs.

- Charlie

 

[Charles Oliver Nutter]

In the body of the module you're binding:

ffi_convention :stdcall # :default is for normal C convention

We could add whatever convention Linux/PPC needs.

Or, duh, have FFI use the appropriate default convention on the target
platform. Either way, it should be simple to deal with.

- Charlie

 

[Sylvain Joyeux]

Or, duh, have FFI use the appropriate default convention on the target
platform. Either way, it should be simple to deal with.

Thanks Charlie, but I have already my own library which does more than Ruby-FFI
(typelib.sf.net and typelib.sf.net/html/ruby) so I'll probably be using libffi
directly.

Sylvain

 

[Charles Oliver Nutter]

Thanks Charlie, but I have already my own library which does more than Ruby-FFI
(typelib.sf.net and typelib.sf.net/html/ruby) so I'll probably be using libffi
directly.

Is your library compatible with JRuby? I think that's a key point for
Ruby FFI.

- Charlie

 

[Sylvain Joyeux]

Is your library compatible with JRuby? I think that's a key point for
Ruby FFI.

It is not -- it is mainly C++ -- but given that most of the functionality is
written in C++ *and* I use that C++ interface in other C++ programs, moving to
Ruby-FFI is not practical for me. So ...

Sylvain

 

[Ken Bloom]

The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7
and 1.9!

FFI (gem install ffi) is a library for programmatically loading dynamic
libraries, binding functions within them, and calling those functions
from Ruby code. Here's a quick sample of binding and calling the getpid
C library function:


require 'ffi'

module GetPid
extend FFI::Library

attach_function :getpid, [], :uint
end

puts GetPid.getpid


Here's another, calling qsort and passing a Ruby block as a C callback:


require 'ffi'

module LibC
extend FFI::Library
callback :qsort_cmp, [ ointer, ointer ], :int attach_function
:qsort, [ ointer, :int, :int, :qsort_cmp ], :int
end

p = MemoryPointer.newint, 2)
p.put_array_of_int32(0, [ 2, 1 ])
puts "Before qsort #{p.get_array_of_int32(0, 2).join(', ')}"
LibC.qsort(p, 2, 4) do |p1, p2|
i1 = p1.get_int32(0)
i2 = p2.get_int32(0)
i1 < i2 ? -1 : i1 > i2 ? 1 : 0
end
puts "After qsort #{p.get_array_of_int32(0, 2).join(', ')}"


I posted a blog entry with a longer description of the library,
additional examples, and links to some other documentation and posts.
Docs are a little slim at this point, so feel free to experiment and
update the JRuby wiki page:

http://wiki.jruby.org/wiki/Calling_C_from_JRuby

I'm sure docs from here will filter back into the library and out into
the general cosmos.

Finally, there's no need to write a C extension to call C libraries, and
the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby 1.1.4+, and
Rubinius (though Rubinius has no callback support yet).

Don't be an extension stooge! Use FFI!

- Charlie

Based on all of the examples I've seen, there's something that's still
not clear to me. How does ruby-ffi know which library to load to find the
specified function?

--Ken

 

[Sean O'Halpin]

Based on all of the examples I've seen, there's something that's still
not clear to me. How does ruby-ffi know which library to load to find the
specified function?

--Ken

Use ffi_lib in the extended module, e.g.

require 'rubygems'
require 'ffi'

module NCursesFFI
extend FFI::Library
ffi_lib 'ncurses'
attach_function 'clear', [], :int
attach_function 'endwin', [], :int
attach_function 'getch', [], :int
attach_function 'initscr', [], :int
attach_function 'printw', [ :string ], :int
attach_function 'refresh', [], :int
end

NCursesFFI.initscr
NCursesFFI.printw("Hello again")
NCursesFFI.refresh
NCursesFFI.getch
NCursesFFI.endwin

Regards,
Sean

 

[Nit Khair]

module NCursesFFI
extend FFI::Library
ffi_lib 'ncurses'
attach_function 'clear', [], :int
attach_function 'endwin', [], :int
attach_function 'getch', [], :int
attach_function 'initscr', [], :int
attach_function 'printw', [ :string ], :int
attach_function 'refresh', [], :int
end

Sean,
Going by the above, and the samples of structs on the blog, for a medium
sized library such as ncurses, there can be some effort in getting this
module ready. Will there be some kind of repository of such modules, so
we are not duplicating each others effort ?

I'd like to also add that the above sample is working on osx
10.5.5/darwin/ppc after a small modification in platform.rb above line
28.

ARCH = case CPU.downcase
when /i?86|x86|i86pc|powerpc/
"i386"
when /amd64|x86_64/
"x86_64"

I have added "powerpc" to the first "when". btw, i had a clean gem
install.
Thanks.

 

[Sean O'Halpin]

module NCursesFFI
extend FFI::Library
ffi_lib 'ncurses'
attach_function 'clear', [], :int
attach_function 'endwin', [], :int
attach_function 'getch', [], :int
attach_function 'initscr', [], :int
attach_function 'printw', [ :string ], :int
attach_function 'refresh', [], :int
end

Sean,
Going by the above, and the samples of structs on the blog, for a medium
sized library such as ncurses, there can be some effort in getting this
module ready. Will there be some kind of repository of such modules, so
we are not duplicating each others effort ?

It's a fairly tedious (and largely thankless) task to convert a lib
like ncurses. However, I'm in the mood right now so expect a github
repo in the near future Your point about avoiding duplication of
effort is a good one. The main problem in tackling something like this
is fatigue setting in when you've got enough for your immediate
requirement and so ending up with a half-baked conversion (I speak
from experience). I believe another pitfall is trying to rubyify the
interface. I think it's better to provide a thin bridge and then
abstract on top of that. Then you get the benefit of being able to use
existing experience and documentation (with the usual translations)
and others can take or leave your abstraction as they see fit. (I do
think Ruby needs a portable console library but I'd rather build that
on top of ruby versions of ncurses and Win32 console than try to make
ncurses work on Windows. I'm hoping FFI will make that possible.)

I'm wondering if it would be useful to have these FFI-enabled libs
under an ffi/ namespace, e.g. you'd require 'ffi/ncurses' and possibly
include FFI::NCurses. We could put shared structs, constants, etc.
under ffi/include. Any thoughts anyone?

I'd like to also add that the above sample is working on osx
10.5.5/darwin/ppc after a small modification in platform.rb above line
28.

ARCH = case CPU.downcase
when /i?86|x86|i86pc|powerpc/
"i386"
when /amd64|x86_64/
"x86_64"

I have added "powerpc" to the first "when". btw, i had a clean gem
install.

Excellent news.

Regards,
Sean

 

[Luis Lavena]

module NCursesFFI
 extend FFI::Library
 ffi_lib 'ncurses'
 attach_function 'clear', [], :int
 attach_function 'endwin', [], :int
 attach_function 'getch', [], :int
 attach_function 'initscr', [], :int
 attach_function 'printw', [ :string ], :int
 attach_function 'refresh', [], :int
end

Sean,
Going by the above, and the samples of structs on the blog, for a medium
sized library such as ncurses, there can be some effort in getting this
module ready. Will there be some kind of repository of such modules, so
we are not duplicating each others effort ?

It's a fairly tedious (and largely thankless) task to convert a lib
like ncurses. However, I'm in the mood right now so expect a github
repo in the near future Your point about avoiding duplication of
effort is a good one. The main problem in tackling something like this
is fatigue setting in when you've got enough for your immediate
requirement and so ending up with a half-baked conversion (I speak
from experience). I believe another pitfall is trying to rubyify the
interface. I think it's better to provide a thin bridge and then
abstract on top of that. Then you get the benefit of being able to use
existing experience and documentation (with the usual translations)
and others can take or leave your abstraction as they see fit. (I do
think Ruby needs a portable console library but I'd rather build that
on top of ruby versions of ncurses and Win32 console than try to make
ncurses work on Windows. I'm hoping FFI will make that possible.)

Yeah, first layer of implementation and then abstraction (Rubify) is
the way to go

I'm wondering if it would be useful to have these FFI-enabled libs
under an ffi/ namespace, e.g. you'd require 'ffi/ncurses' and possibly
include FFI::NCurses. We could put shared structs, constants, etc.
under ffi/include. Any thoughts anyone?

Something like the win32utils team does:

gem: win32-service
gem win32-eventlog

require 'win32/service'
require 'win32/eventlog'

All these live inside Win32 namespace

I think these could be named ffi-ncourses but the name implies "FFI
implementation of ncurses" where I will love to have "cross platform
curses" (note the lack of 'n').

These should depend on ruby-ffi gem and we are set to go

My to cents

 

[Dominic Sisneros]

[Note: parts of this message were removed to make it a legal post.]

On windows, I can't get it to work for jruby or ruby

C:\jruby-1.1.5>jruby -v
jruby 1.1.5 (ruby 1.8.6 patchlevel 114) (2008-11-03 rev 7996) [x86-java]

C:\jruby-1.1.5>cd samples\ffi

C:\jruby-1.1.5\samples\ffi>dir
Volume in drive C has no label.
Volume Serial Number is F4EA-F50A

Directory of C:\jruby-1.1.5\samples\ffi

11/03/2008 05:04 PM <DIR> .
11/03/2008 05:04 PM <DIR> ..
11/03/2008 04:42 PM 301 ffi.rb
11/03/2008 04:42 PM 367 gettimeofday.rb
11/03/2008 04:42 PM 2,340 pty.rb
11/03/2008 04:42 PM 476 qsort.rb
11/03/2008 04:42 PM 2,078 win32api.rb
5 File(s) 5,562 bytes
2 Dir(s) 18,840,158,208 bytes free

C:\jruby-1.1.5\samples\ffi>jruby win32api.rb
null:1:in `const_missing': uninitialized constant POSIX::FFI (NameError)
from ffi.rb:4
from ffi.rb:1:in `require'
from win32api.rb:1

C:\jruby-1.1.5\samples\ffi>

For MRI, it tries to compile

C:\jruby-1.1.5\samples\ffi>gem install ffi
Building native extensions. This could take a while...
ERROR: Error installing ffi:
ERROR: Failed to build gem native extension.

c:/ruby/bin/ruby.exe extconf.rb install ffi
creating Makefile

nmake
'nmake' is not recognized as an internal or external command,
operable program or batch file.


Gem files will remain installed in c:/ruby/lib/ruby/gems/1.8/gems/ffi-0.1.1
for inspection.
Results logged to c:/ruby/lib/ruby/gems/1.8/gems/ffi-0.1.1/ext/gem_make.out

C:\jruby-1.1.5\samples\ffi>

 

[Nit Khair]

attach_function 'initscr', [], :int
attach_function 'printw', [ :string ], :int
attach_function 'refresh', [], :int
end

Sean,
Going by the above, and the samples of structs on the blog, for a medium
sized library such as ncurses, there can be some effort in getting this
module ready. Will there be some kind of repository of such modules, so
we are not duplicating each others effort ?

It's a fairly tedious (and largely thankless) task to convert a lib
like ncurses. However, I'm in the mood right now so expect a github
repo in the near future Your point about avoiding duplication of
effort is a good one. The main problem in tackling something like this
is fatigue setting in when you've got enough for your immediate
requirement and so ending up with a half-baked conversion (I speak
from experience). I believe another pitfall is trying to rubyify the
interface.

I suppose some of these conversions/wrappers would be "approved" and
sort of become a standard, so we don't have a plethora of them. Then
they could be placed in one repo (after some sort of review). Just a
thought.

Would hate to have find 5 different ncurses wrappers and have to study
each to figure out which to use.

 

[Charles Oliver Nutter]

module NCursesFFI
extend FFI::Library
ffi_lib 'ncurses'
attach_function 'clear', [], :int
attach_function 'endwin', [], :int
attach_function 'getch', [], :int
attach_function 'initscr', [], :int
attach_function 'printw', [ :string ], :int
attach_function 'refresh', [], :int
end

Sean,
Going by the above, and the samples of structs on the blog, for a medium
sized library such as ncurses, there can be some effort in getting this
module ready. Will there be some kind of repository of such modules, so
we are not duplicating each others effort ?

I'd recommend folks start releasing FFI-based "thin wrappers" for each
library they come across, and then depend on that gem. Don't go over the
top...just get the basic bindings in there, structs, tests to make sure
it's all working, and release a gem. Then build nice APIs on top of
that, separately.

One of my biggest gripes about C extensions in Ruby is that everyone
feels the need to release their own wrapper PLUS Ruby-like API in a
single gem, when just releasing the wrapper would allow a lot more
experimentation. So you end up with multiple people writing their own
extensions with their own APIs and little sharing.

If we just get a whole ton of, simple, clean wrappers out there, the
APIs will surely follow.

- Charlie

 

[Charles Oliver Nutter]

C:\jruby-1.1.5\samples\ffi>jruby win32api.rb
null:1:in `const_missing': uninitialized constant POSIX::FFI (NameError)
from ffi.rb:4
from ffi.rb:1:in `require'
from win32api.rb:1

That sample could be behind the times...play with it a bit and see what
you can see. None of the core JRuby devs use Windows on a regular
basis...I think this was just a quick experiment created early in FFI
dev cycle.

FWIW, I believe someone (Dan Berger, probably) is working on a full FFI
win32api impl. He could use some help.

For MRI, it tries to compile

C:\jruby-1.1.5\samples\ffi>gem install ffi
Building native extensions. This could take a while...
ERROR: Error installing ffi:
ERROR: Failed to build gem native extension.

c:/ruby/bin/ruby.exe extconf.rb install ffi
creating Makefile

nmake
'nmake' is not recognized as an internal or external command,
operable program or batch file.

We need to get a win32 binary FFI gem released. I think one or two
people are working on that. Jump on Ruby FFI mailing lists if you are
interested in helping.

- Charlie

 

[Charles Oliver Nutter]

I posted a blog entry with a longer description of the library,
additional examples, and links to some other documentation and posts.
Docs are a little slim at this point, so feel free to experiment and
update the JRuby wiki page:

http://wiki.jruby.org/wiki/Calling_C_from_JRuby

I'm sure docs from here will filter back into the library and out into
the general cosmos.

FYI, I've moved the content of the page above to the Ruby FFI wiki here:

http://kenai.com/projects/ruby-ffi/pages/Home

Please make updates and additions to that copy. I'll modify the JRuby
version to link to Ruby FFI's wiki.

- Charlie

 

[Ken Bloom]

module NCursesFFI
extend FFI::Library
ffi_lib 'ncurses'
attach_function 'clear', [], :int
attach_function 'endwin', [], :int
attach_function 'getch', [], :int
attach_function 'initscr', [], :int
attach_function 'printw', [ :string ], :int attach_function
'refresh', [], :int
end

Sean,
Going by the above, and the samples of structs on the blog, for a
medium sized library such as ncurses, there can be some effort in
getting this module ready. Will there be some kind of repository of
such modules, so we are not duplicating each others effort ?

I'd recommend folks start releasing FFI-based "thin wrappers" for each
library they come across, and then depend on that gem. Don't go over the
top...just get the basic bindings in there, structs, tests to make sure
it's all working, and release a gem. Then build nice APIs on top of
that, separately.

One of my biggest gripes about C extensions in Ruby is that everyone
feels the need to release their own wrapper PLUS Ruby-like API in a
single gem, when just releasing the wrapper would allow a lot more
experimentation. So you end up with multiple people writing their own
extensions with their own APIs and little sharing.

If we just get a whole ton of, simple, clean wrappers out there, the
APIs will surely follow.

- Charlie

The good news is that unlike C libraries, if someone bundles their own
wrapper with their own API, you can still get at the wrapper, by finding
the module that imports FFI and just using that.

 

[Sean O'Halpin]

I'd recommend folks start releasing FFI-based "thin wrappers" for each
library they come across, and then depend on that gem. Don't go over the
top...just get the basic bindings in there, structs, tests to make sure it's
all working, and release a gem. Then build nice APIs on top of that,
separately.

One of my biggest gripes about C extensions in Ruby is that everyone feels
the need to release their own wrapper PLUS Ruby-like API in a single gem,
when just releasing the wrapper would allow a lot more experimentation. So
you end up with multiple people writing their own extensions with their own
APIs and little sharing.

If we just get a whole ton of, simple, clean wrappers out there, the APIs
will surely follow.

- Charlie

Maybe I wasn't clear - what you're proposing is exactly what I
intended to convey. Thin wrapper in a single gem. Any abstractions
should be separate gems.

Regards,
Sean

 

[Charles Oliver Nutter]

Maybe I wasn't clear - what you're proposing is exactly what I
intended to convey. Thin wrapper in a single gem. Any abstractions
should be separate gems.

Perfect then

Tom Enebo actually suggested just naming them the library name, as in
libncurses, libtommath, whatever.

- Charlie

 

[Sean O'Halpin]

Tom Enebo actually suggested just naming them the library name, as in
libncurses, libtommath, whatever.

That's a little *nix-centric, no? If we had everything under ffi/
you'd be able to find everything in one place (e.g. easier to find
examples to copy from, easier to set up conventions for where to put
shared structs, constants, etc.). As for OS-specific libs, there could
be ffi/win32/ ffi/unix, where appropriate.

It would be useful to have conventions on how to implement thin layer
ffis (e.g. how to handle return values in *ptrs, how to implement
functons defined as macros, namespacing constants, etc.). This would
help POLS. Also, guidelines on how to portably implement layers for
32-bit/64-bit and endian-ness would be handy (I know I'd like some
tips).

While there is no general solution to converting header files,
projects such as SWIG have shown that you can get a lot done
automatically. It would be worthwhile to work on tools to help with
this - even an 80% solution removes a lot of the donkey work (as I'm
finding with my ncurses port).

My general opinion is that the more boring, mechanical, choice-free
and thoughtless we can make this work, the easier it will be to do and
so the more likely to get done.

Your cross-implementation FFI initiative is very welcome - it's a
liberating move. Now we can get on with the mindless (but worthy)
drudgery

Regards,
Sean

 

[Nit Khair]

automatically. It would be worthwhile to work on tools to help with
this - even an 80% solution removes a lot of the donkey work (as I'm
finding with my ncurses port).


Regards,
Sean

Eagerly awaiting your ncurses port...

Is it possible for you to document the procedure, how you go about doing
it, so one can follow that for another port, rather than be totally
lost.