wrapping a C struct[] constant

[Martin DeMello]

I have some C code with a struct definition

struct flag_str {
unsigned int val;
const char *str;
};

and an inline array

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_LAST, "last" },
{ FIEMAP_EXTENT_UNKNOWN, "unkown" },
{ FIEMAP_EXTENT_DELALLOC, "delalloc" },
{ FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
{ FIEMAP_EXTENT_SECONDARY, "secondary" },
{ FIEMAP_EXTENT_NET, "net" },
{ FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
{ FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
{ FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
{ FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
{ FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
{ FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
{ FIEMAP_EXTENT_MERGED, "merged" },
{ 0, NULL },
};

What's the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course )?

martin

 

[Eero Saynatkari]

I have some C code with a struct definition

struct flag_str {
unsigned int val;
const char *str;
};

and an inline array

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_LAST, "last" },
{ FIEMAP_EXTENT_UNKNOWN, "unkown" },
{ FIEMAP_EXTENT_DELALLOC, "delalloc" },
{ FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
{ FIEMAP_EXTENT_SECONDARY, "secondary" },
{ FIEMAP_EXTENT_NET, "net" },
{ FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
{ FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
{ FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
{ FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
{ FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
{ FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
{ FIEMAP_EXTENT_MERGED, "merged" },
{ 0, NULL },
};

What's the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course )?

How/where are you *using* it?

FFI is generally a good solution, although it
would probably be best to actually define the
Array on the Ruby side and pass it in as needed.


Eero

 

[Jason Roelofs]

[Note: parts of this message were removed to make it a legal post.]

I have some C code with a struct definition

struct flag_str {
unsigned int val;
const char *str;
};

and an inline array

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_LAST, "last" },
{ FIEMAP_EXTENT_UNKNOWN, "unkown" },
{ FIEMAP_EXTENT_DELALLOC, "delalloc" },
{ FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
{ FIEMAP_EXTENT_SECONDARY, "secondary" },
{ FIEMAP_EXTENT_NET, "net" },
{ FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
{ FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
{ FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
{ FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
{ FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
{ FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
{ FIEMAP_EXTENT_MERGED, "merged" },
{ 0, NULL },
};

What's the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course )?

How/where are you *using* it?

FFI is generally a good solution, although it
would probably be best to actually define the
Array on the Ruby side and pass it in as needed.


Eero

Even using FFI, you'll need to define the wrapping between C constant to
Ruby constant, and using Ruby's API, you'll be define_const-ing every one of
those, then putting them into an array. Just define them in the Ruby
yourself, there isn't a faster way to do it.

Jason

 

[Martin DeMello]

I have some C code with a struct definition

struct flag_str {
=A0 unsigned int val;
=A0 const char *str;
};

and an inline array

struct flag_str extent_flags[] =3D {
=A0 { FIEMAP_EXTENT_LAST, =A0 =A0"last" },
=A0 { FIEMAP_EXTENT_UNKNOWN, =A0"unkown" },
=A0 { FIEMAP_EXTENT_DELALLOC, =A0"delalloc" },
=A0 { FIEMAP_EXTENT_NO_BYPASS, =A0"no_bypass" },
=A0 { FIEMAP_EXTENT_SECONDARY, =A0"secondary" },
=A0 { FIEMAP_EXTENT_NET, =A0 =A0"net" },
=A0 { FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
=A0 { FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
=A0 { FIEMAP_EXTENT_NOT_ALIGNED, =A0"not_aligned" },
=A0 { FIEMAP_EXTENT_DATA_INLINE, =A0"data_inline" },
=A0 { FIEMAP_EXTENT_DATA_TAIL, =A0"data_tail" },
=A0 { FIEMAP_EXTENT_UNWRITTEN, =A0"unwritten" },
=A0 { FIEMAP_EXTENT_MERGED, =A0 =A0"merged" },
=A0 { 0, =A0 =A0 =A0 =A0NULL },
};

What's the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course )?

How/where are you *using* it?

Using it (or want to use it!) from the ruby side to generate a hash of
constant =3D> name and use that to unpack and display flag settings.

m.

 

[Martin DeMello]

Even using FFI, you'll need to define the wrapping between C constant to
Ruby constant, and using Ruby's API, you'll be define_const-ing every one of
those, then putting them into an array. Just define them in the Ruby
yourself, there isn't a faster way to do it.

I was afraid of that The docs are a bit on the sketchy side too. I
guess some sort of code generation would be the way to go then (swig
seems like overkill). Will give cgenerator a look.

martin

 

[Joel VanderWerf]

Using it (or want to use it!) from the ruby side to generate a hash of
constant => name and use that to unpack and display flag settings.

If all you want is this data structure in ruby, you could parse the
source (gccxml?), and then you have a pure ruby library that defines
this hash. Define the parse->xml>rb part as a rake task if you need to
keep up to date with the source files.

 

[Joel VanderWerf]

I was afraid of that The docs are a bit on the sketchy side too. I
guess some sort of code generation would be the way to go then (swig
seems like overkill). Will give cgenerator a look.

martin

Cgenerator will work, and might be a good idea if the source (the list
of values and strings) changes from time to time, and you would rather
have your program regenerate the ruby extension (and data structure)
automatically.

Here's how it works, assuming your source files are flag.c and flag.h:

$ ls
flag.c flag.h flag.rb
$ cat flag.h
struct flag_str {
unsigned int val;
const char *str;
};

extern struct flag_str extent_flags[];

typedef enum {
FIEMAP_EXTENT_LAST = 1,
FIEMAP_EXTENT_UNKNOWN,
FIEMAP_EXTENT_DELALLOC,
FIEMAP_EXTENT_NO_BYPASS,
FIEMAP_EXTENT_SECONDARY,
FIEMAP_EXTENT_NET,
FIEMAP_EXTENT_DATA_COMPRESSED,
FIEMAP_EXTENT_DATA_ENCRYPTED,
FIEMAP_EXTENT_NOT_ALIGNED,
FIEMAP_EXTENT_DATA_INLINE,
FIEMAP_EXTENT_DATA_TAIL,
FIEMAP_EXTENT_UNWRITTEN,
FIEMAP_EXTENT_MERGED
} FIEMAP_EXTENT;

$ cat flag.c
#include "flag.h"

#ifndef NULL
#define NULL 0
#endif

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_LAST, "last" },
{ FIEMAP_EXTENT_UNKNOWN, "unkown" },
{ FIEMAP_EXTENT_DELALLOC, "delalloc" },
{ FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
{ FIEMAP_EXTENT_SECONDARY, "secondary" },
{ FIEMAP_EXTENT_NET, "net" },
{ FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
{ FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
{ FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
{ FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
{ FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
{ FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
{ FIEMAP_EXTENT_MERGED, "merged" },
{ 0, NULL },
};

$ cat flag.rb
#!/usr/bin/env ruby

require 'cgen/cgen'
require 'fileutils'

module Flag; end

# Generate the extension source code.
lib = CGenerator::Library.new "flag_lib"
lib.include "flag.h"

lib.define_c_singleton_method(Flag, :extent_flags).instance_eval {
# no args
body %{\
VALUE a;
struct flag_str *pfs;

a = rb_ary_new();
for (pfs = extent_flags; pfs->val; pfs++) {
rb_ary_push(a,
rb_ary_new3(2,
INT2NUM(pfs->val),
rb_str_new2(pfs->str)
));
}
}
returns "a"
}

# Normally, this isn't needed, but we need to set up the external files
# as symlinks in this dir before calling commit.
FileUtils.mkdir_p("flag_lib")

# Put links to sources where they will be found.
Dir.chdir "flag_lib" do
%w{ flag.h flag.c }.each do |file|
FileUtils.ln_s("../#{file}", file) rescue nil
end
end

# Write and build
lib.commit

# Use the library
Flag::EXTENT_FLAGS = Hash[*Flag.extent_flags.flatten]
p Flag::EXTENT_FLAGS

$ ruby flag.rb
{5=>"secondary", 11=>"data_tail", 6=>"net", 12=>"unwritten", 1=>"last",
7=>"data_compressed", 13=>"merged", 2=>"unkown", 8=>"data_encrypted",
3=>"delalloc", 9=>"not_aligned", 4=>"no_bypass", 10=>"data_inline"}

 

[Martin DeMello]

Cgenerator will work, and might be a good idea if the source (the list of
values and strings) changes from time to time, and you would rather have
your program regenerate the ruby extension (and data structure)
automatically.

Here's how it works, assuming your source files are flag.c and flag.h:

Thanks, that looks great. I have to say, I'm a bit disappointed in the
state of ruby/C integration - another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator's
CShadow is all about?). Are these not common use cases?

martin

 

[Jason Roelofs]

[Note: parts of this message were removed to make it a legal post.]

Thanks, that looks great. I have to say, I'm a bit disappointed in the
state of ruby/C integration - another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator's
CShadow is all about?). Are these not common use cases?

martin

Could you give an example of how this would work? I'm having a hard time
picturing how such a thing would be possible to have done automagically.
There has to be something somewhere that defines how a Ruby class would map
to said C struct.

Jason

 

[Joel VanderWerf]

another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator's
CShadow is all about?). Are these not common use cases?

Would this be like swig in reverse? Input a ruby class with type
annotations, output C header, accessors, etc? I'd like to do that for
the special case of BitStruct classes (fixed-length, packed fields,
usually numeric or character, stored in a ruby String using
pack/unpack), so that once you define a network protocol or other binary
format in ruby, you can share your code with C programmers (and via swig
with other languages). That seems feasible, and I did actually do this
with a predecessor of BitStruct, using cgen. But how would you handle a
more general case, like a class with a hash or array attribute?

CShadow is similar, but instead of inheriting from String like
BitStruct, its instances are T_DATA (ruby objects with a blob that can
only be accessed from C). Unlike a BitStruct, this blob is not intended
for sending outside of the ruby+C world. For example, a "self" pointer
is automatically added at the beginning of the blob.

Including CShadow in your class lets you define the structure of the
T_DATA object in ruby, in terms of accessors of various types. When your
code runs, it uses cgenerator to build an extension that takes care of
all the boiler plate functions: mark, free, accessors with type
checking/conversion, alloc, init, inspect, marshal, and yaml. Plus it
handles inheritance and sets up introspection methods. [See example below.]

CShadow also gives you tools to distribute source code in separate .c
and .h files, so that when you make a minor change in the definitions
embedded in ruby, you minimize how much needs to be recompiled. This
works well enough that even if the generated code is a few Mb, you can
let cgenerator do its full generate/make cycle every time you run your
program, without noticing much delay.

You can add C functions to a CShadow class inline in ruby source just as
with cgenerator. But CShadow only supports a few kinds of attributes
"out of the box": ruby object references, native scalar data of various
sizes, and native pointers to char, double, array of double, etc. You
can add new attr types with a little work, but it's far from automatic.

These are just two special case solutions. I'm not seeing what a general
solution would look like. What kinds of classes/attributes are there in
your use cases?

--------------------------------
require 'cgen/cshadow'

class Parent
include CShadow

shadow_attr_accessor :ruby_str => String # type-checked VALUE type
shadow_attr_accessor :c_int => "int c_int"
end

class Child < Parent
shadow_attr_accessor :c_str => "char *c_str"
shadow_attr_accessor bj => Object # VALUE type
end

Parent.commit
# we're done adding attrs and methods, so make.

x = Child.new
x.ruby_str = "foo"
x.c_str = "bar"
x.obj = [1,2,3]
x.c_int = 3

p x

CShadow.allow_yaml
y x

__END__

Output:

#<Child:0xb79716cc ruby_str="foo", c_int=3, c_str="bar", obj=[1, 2, 3]>
--- !path.berkeley.edu,2006/cshadow:Child
ruby_str: foo
c_int: 3
c_str: bar
obj:
- 1
- 2
- 3

 

[Martin DeMello]

Would this be like swig in reverse? Input a ruby class with type
annotations, output C header, accessors, etc?

I actually meant like a simplified swig - you have a C struct
(arbitrary nesting of structs and arrays, actually), and you have a
ruby class that you define a typemap for and then have it
serialise/deserialise. Basically you know that you can mirror C
structures in ruby with a bit of gruntwork, all you need is a tool to
do the gruntwork for you

I'd like to do that for the
special case of BitStruct classes (fixed-length, packed fields, usually
numeric or character, stored in a ruby String using pack/unpack), so that
once you define a network protocol or other binary format in ruby, you can
share your code with C programmers (and via swig with other languages). That
seems feasible, and I did actually do this with a predecessor of BitStruct,
using cgen. But how would you handle a more general case, like a class with
a hash or array attribute?

Network protocols were one of the use cases I was thinking of, except
that you usually see them defined in C. I wanted the ability to use
the structs from ruby, without going through C the way
data_wrap_struct makes you do. Not asking for magic, just the ability
to make the common case trivial.

Including CShadow in your class lets you define the structure of the T_DATA
object in ruby, in terms of accessors of various types. When your code runs,
it uses cgenerator to build an extension that takes care of all the boiler
plate functions: mark, free, accessors with type checking/conversion, alloc,
init, inspect, marshal, and yaml. Plus it handles inheritance and sets up
introspection methods. [See example below.]

Okay, that's most of what I was looking for If it can convert an
array of structs into a ruby array, that's *all* of what I'm looking
for. Need to look through the CGenerator docs some more - great piece
of work!

martin

 

[Martin DeMello]

Could you give an example of how this would work? I'm having a hard time
picturing how such a thing would be possible to have done automagically.
There has to be something somewhere that defines how a Ruby class would map
to said C struct.

Yes, I was simply looking for something like

struct foo {
int bar;
char *baz;
}

cFoo = rb_define_class('cFoo')
rb_map_struct(cFoo, foo)
rb_accessor(cFoo, 'int', 'bar', bar)
rb_accessor(cFoo, 'str', 'bar', bar)

VALUE rb_function() {
foo* a = some_function_call()
return rb_deserialize_data(foo)
}

and from ruby

foo = rb_function
foo.bar
foo.baz

martin

 

[John W Higgins]

[Note: parts of this message were removed to make it a legal post.]

Yes, I was simply looking for something like

struct foo {
int bar;
char *baz;
}

cFoo = rb_define_class('cFoo')
rb_map_struct(cFoo, foo)
rb_accessor(cFoo, 'int', 'bar', bar)
rb_accessor(cFoo, 'str', 'bar', bar)

VALUE rb_function() {
foo* a = some_function_call()
return rb_deserialize_data(foo)
}

and from ruby

foo = rb_function
foo.bar
foo.baz

martin

You are attacking your problem from the wrong end. The problem is that C
doesn't have reflection so there is no way the rb_map_struct concept would
ever work. C won't tell you what the members are so you can't map it to
anything.

However, working in the other direction - you could use the C preprocessor
to build both your structs and the appropriate C code to define your ruby
class you want for each struct. It's not pretty but it does work - have a
look here for a nice defintion of what you are looking for
http://en.wikipedia.org/wiki/C_preprocessor#X-Macros

John

 

[Joel VanderWerf]

If it can convert an array of structs into a ruby array, that's *all* of what I'm looking for.

How is the length stored? A separate field? Or is it fixed length?

 

[Martin DeMello]

How is the length stored? A separate field? Or is it fixed length?

In true C style, you need to pass in a pointer in which the length is
returned I'm not thinking of anything magical; I'd expect to pass
in the length as a parameter.

martin