Ruby-based data language

[Intransition]

Has anyone ever endeavored to create a data/configuration file format
based on Ruby's syntax? In other words, a format like YAML but with a
Ruby-based syntax.

I tried to do this the easy way using $SAFE=4, but #method_missing
doesn't appear to work at that safe level.

 

[Brian Candler]

Has anyone ever endeavored to create a data/configuration file format
based on Ruby's syntax? In other words, a format like YAML but with a
Ruby-based syntax.

Which bit of Ruby syntax are you thinking of?

If you wanted key/value pairs in a Hash, then you might as well use
JSON. It's not pretty for config files, but it's OK.

If you wanted

foo = 123

then that's hard to make work, although maybe you could frig it with
binding and local_variables. You could instead use the Rails way:

config.foo = 123

or use constants:

module Config
Foo = 123
end

or globals:

$foo = 123

The Sinatra way would be:

set :foo, 123
setbar) { delayed_expr }

Maybe what you're trying to do is

foo 123

I don't see a particular problem with that, even with $SAFE=4 and
method_missing:

$h = {} => {}
def method_missing(k,v); $h[k] = v; end => nil
t = Thread.new { $SAFE=4; eval "foo 123" }

t.join

SecurityError: (irb):2:in `[]=': Insecure: can't modify hash
from (irb):3
from (irb):4:in `join'
from (irb):4
from :0=> {:foo=>123}

This is using ruby 1.8.7 (2010-01-10 patchlevel 249) [x86_64-linux]

 

[Intransition]

Which bit of Ruby syntax are you thinking of?

If you wanted key/value pairs in a Hash, then you might as well use
JSON. It's not pretty for config files, but it's OK.

If you wanted

=A0 =A0 foo =3D 123

then that's hard to make work, although maybe you could frig it with
binding and local_variables. You could instead use the Rails way:

=A0 =A0 config.foo =3D 123

or use constants:

=A0 =A0 module Config
=A0 =A0 =A0 Foo =3D 123
=A0 =A0 end

or globals:

=A0 =A0 $foo =3D 123

The Sinatra way would be:

=A0 =A0 set :foo, 123
=A0 =A0 setbar) { delayed_expr }

Maybe what you're trying to do is

=A0 =A0 foo 123

Yes, I should have been more specific. This is what I mean. A longer
example:

name "Joe Foo"
age 33
contact do
email "(e-mail address removed)"
phione "555-555-1234"
end

I don't see a particular problem with that, even with $SAFE=3D4 and
method_missing:

$h =3D {} =3D> {}
def method_missing(k,v); $h[k] =3D v; end =3D> nil
t =3D Thread.new { $SAFE=3D4; eval "foo 123" }

=3D> #<Thread:0x7f4eb7749a48 dead>>> t.join

SecurityError: (irb):2:in `[]=3D': Insecure: can't modify hash
=A0 from (irb):3
=A0 from (irb):4:in `join'
=A0 from (irb):4
=A0 from :0>> $h.taint
=3D> {}
=3D> #<Thread:0x7f4eb7726930 dead>>> t.join

=3D> #<Thread:0x7f4eb7726930 dead>>> $h

=3D> {:foo=3D>123}

This is using ruby 1.8.7 (2010-01-10 patchlevel 249) [x86_64-linux]

Ah. You are right. I made a mistake in my experiments. And thank you
for this example, #taint made all the difference.

So now the question: is $SAFE=3D4 safe enough? My gut says no. Even
though the above works, I think ultimately a true data format is
desirable for most needs, such as configuration files. I think Ruby
has a very nice syntax for it. Such a format would be to Ruby as JSON
is to Javascript. And Ruby has a nice advantage with blocks.

 

[Kirk Haines]

So now the question: is $SAFE=4 safe enough? My gut says no. Even
though the above works, I think ultimately a true data format is
desirable for most needs, such as configuration files. I think Ruby
has a very nice syntax for it. Such a format would be to Ruby as JSON
is to Javascript. And Ruby has a nice advantage with blocks.

Enough for what? To trust arbitrary configurations that are executable Ruby?

Trans, if you go way way back in the archives, you'll find a longish
running series of posts between Guy Decoux and a few other people
about $SAFE=4 and various clever hacks to try to, with Ruby, create an
environment that was truly safe. Guy showed in that typically terse,
let-my-code-speak-for-me way that he had, that every very clever Ruby
approach that was conceived of could also be circumvented.

The only way to have a "Ruby" data format that can contain executable
Ruby is to do something like what _Why did with his sandbox (which,
IIRC, came about in part because of this thread involving Guy) -- it
has to be supported at a very low level in the language.

You could create a Ruby-like format that is itself simply interpreted,
and if that's what you are talking about, I'd say go for it. But the
moment you allow anything in that format to be executed _as_Ruby_ you
can no longer trust arbitrary configurations.


Kirk Haines

 

[Brian Candler]

name "Joe Foo"
age 33
contact do
email "(e-mail address removed)"
phione "555-555-1234"
end

You could make a parser for that fairly easily - or transform it to JSON
or YAML and parse that (assuming that contact do .. end constructs a
nested Hash)

Such a format would be to Ruby as JSON is to Javascript. And Ruby has a nice advantage with blocks.

That's where I disagree with you. You are using blocks in a Builder or
Markaby way, but that's not the fundamental purpose or interpretation of
blocks in the language. And how would you handle Arrays?

The Ruby equivalent of JSON would be a literal Hash:

{
"name"=>"Joe Foo",
"age"=>33,
"contact"=>{
"email"=>"(e-mail address removed)",
"phone"=>"555-555-1234",
},
}

Or if you buy into 1.9 syntax, then you could have

{
name: "Joe Foo",
age: 33,
contact: {
email: "(e-mail address removed)",
phone: "555-555-1234",
},
}

Both are similar enough to JSON that I'd use that instead, and gain the
portability benefit. It's a shame that JSON doesn't allow trailing
commas.

 

[Intransition]

You could make a parser for that fairly easily - or transform it to JSON
or YAML and parse that (assuming that contact do .. end constructs a
nested Hash)
nice advantage with blocks.

That's where I disagree with you. You are using blocks in a Builder or
Markaby way, but that's not the fundamental purpose or interpretation of
blocks in the language. And how would you handle Arrays?

The Ruby equivalent of JSON would be a literal Hash:

{
=A0 "name"=3D>"Joe Foo",
=A0 "age"=3D>33,
=A0 "contact"=3D>{
=A0 =A0 "email"=3D>"(e-mail address removed)",
=A0 =A0 "phone"=3D>"555-555-1234",
=A0 },

}

Or if you buy into 1.9 syntax, then you could have

{
=A0 name: "Joe Foo",
=A0 age: 33,
=A0 contact: {
=A0 =A0 email: "(e-mail address removed)",
=A0 =A0 phone: "555-555-1234",
=A0 },

}

Both are similar enough to JSON that I'd use that instead, and gain the
portability benefit. It's a shame that JSON doesn't allow trailing
commas.

That's a good point. But if you look at Ruby-based configuration files
they always use block-based DSL notations, not hashes. So the analogy
isn't over the data structure, but rather the use of the underlying
language as a syntax model.

 

[Brian Candler]

But if you look at Ruby-based configuration files
they always use block-based DSL notations, not hashes.

I'd have said most packages use YAML. If you can provide examples which
use method_missing and block-based structure just for conveying static
configuration information, I'd be interested to see them. But then
that's what you asked for in the first place

So the analogy
isn't over the data structure, but rather the use of the underlying
language as a syntax model.

The difference is that JSON *is* a direct subset of Javascript, and can
be eval'd directly to give a value.

Let me put it another way. If you were parsing your example:

name "Joe Foo"
age 33
contact do
email "(e-mail address removed)"
phione "555-555-1234"
end

would you expect as Hash oas the result?

If yes: then it looks like you're proposing an alternative syntax for
Hash literals - one which isn't widely used. It's Ruby-inspired but not
really Ruby, since a simple eval of the above will fail without
additional supporting code. It would be similar to Builder, but (a)
creating a Hash as its output instead of XML, and (b) intended for use
with untrusted inputs, so parsed rather than eval'd.

If no: then what do you expect to get instead? I suppose you could have
a stream parser API, and trigger start/end actions as you go, but that's
not a very convenient API for reading a config file.

 

[Brian Candler]

If yes: then it looks like you're proposing an alternative syntax for

Hash literals - one which isn't widely used. It's Ruby-inspired but not
really Ruby, since a simple eval of the above will fail without
additional supporting code.

That is, to parse that record using eval I think you need something
along these lines:

class Parser # < BasicObject ??
def self.parse(*args, &blk)
o = new
o.instance_eval(*args, &blk)
o.instance_variable_get@value)
end

def initialize
@value = {}
end

def method_missing(label, value=:__MISSING__,&blk)
if value != :__MISSING__
raise "Cannot provide both value and block for #{label}" if
block_given?
@value[label] = value
else
raise "Must provide value or block for #{label}" unless
block_given?
@value[label] = self.class.parse(&blk)
end
end
end

person = Parser.parse <<'EOS'
name "Joe Foo"
age 33
contact do
email "(e-mail address removed)"
phione "555-555-1234"
end
EOS
p person

Now, to get the same capabilities as JSON, you also need syntax for
Arrays. You could do something like the following, although note that
the parser above doesn't handle this properly:

people([
person do
name "Joe"
age 33
end,
person do
name "Fred"
age 64
end,
])

You do need both parentheses and square brackets, unless you replace
do/end with braces:

people [
person {
name "Joe"
age 33
},
person {
name "Fred"
age 64
},
]

Add a few colons and commas and you're back to JSON. Or drop the closing
braces and brackets and keep the indentation, and you're back to YAML.

You could treat multiple arguments as an array:

people \
person {
name "Joe"
age 33
},
person {
name "Fred"
age 64
}

but then if you wanted a one-element array it would have to be a special
case. Or you could perhaps have a flag to indicate that you want an
Array:

people [],
person {
name "Joe"
age 33
},
person {
name "Fred"
age 64
}

JSON and YAML may be ugly, but IMO so is this.

 

[Intransition]

I'd have said most packages use YAML. If you can provide examples which
use method_missing and block-based structure just for conveying static
configuration information, I'd be interested to see them. But then
that's what you asked for in the first place

I think a Gemfile is a pretty good example --yes you can use
conditional code in these files, but IMO it's bad design.

Also, https://rubygems.org/gems/configuration looks fairly popular and
it is pretty close to what I'm talking about.

The difference is that JSON *is* a direct subset of Javascript, and can
be eval'd directly to give a value.

Let me put it another way. If you were parsing your example:

=A0 name "Joe Foo"
=A0 age 33
=A0 contact do
=A0 =A0 email "(e-mail address removed)"
=A0 =A0 phione "555-555-1234"
=A0 end

would you expect as Hash oas the result?

If yes: then it looks like you're proposing an alternative syntax for
Hash literals - one which isn't widely used. It's Ruby-inspired but not
really Ruby, since a simple eval of the above will fail without
additional supporting code. It would be similar to Builder, but (a)
creating a Hash as its output instead of XML, and (b) intended for use
with untrusted inputs, so parsed rather than eval'd.

If no: then what do you expect to get instead? I suppose you could have
a stream parser API, and trigger start/end actions as you go, but that's
not a very convenient API for reading a config file.

I would expect to get an object that gave me access to the data. What
kind of object depends on the parser. If only #eval were the parser
and nothing more, I'd expect a method missing error.

I get what you are saying. But even JSON goes through a parser in
Javascript and is not simply evaled, for the same reasons I would like
to see a Ruby-syntax data/config format. And when someone thinks "Ruby-
syntax data/config format" they are thinking builder-style, not hash
literals.

 

[Brian Candler]

I think a Gemfile is a pretty good example

Do you mean a Gemfile from Bundler? e.g.

source "http://rubygems.org"

gem "rails", "3.0.0.rc"
gem "rack-cache"
gem "nokogiri", "~> 1.4.2"

Some attributes have one value, some attributes have more than one
value, some attributes can be repeated. I guess a generic parser API for
that would be a call to callback(key, *args), just like method_missing,
or it could build a linear data structure like

[[:source,"http://rubygems.org"],
[:gem,"rack-cache"],
[:gem,"nokogiri","~> 1.4.2"]]

I'm not sure what output you'd want from nested blocks.

I would expect to get an object that gave me access to the data. What
kind of object depends on the parser.

If the parser is generic (not application-specific) then I guess you'd
just get the structure I've shown above. If you want to build that into
application-specific objects, then you can do so. For example, for each
"gem" line you might want to build a Gem object and append it to an
Array of gems.

I get what you are saying. But even JSON goes through a parser in
Javascript

It *can* go through a parser for security reasons, but the result from
parsing it is exactly the same as if you read it in as a Javascript
literal. And there are cases where you intentionally interpret it as
Javascript code, e.g. JSONP.

when someone thinks "Ruby-
syntax data/config format" they are thinking builder-style, not hash
literals.

Perhaps. I think more common would be a Rails-style configuration or a
gemspec:

Gem::Specification.new do |s|
s.name = %q{snailgun}
s.version = "1.0.6"
... etc
end

Of course, the advantage of having it as real executable code is that
you can use Ruby to assemble data constructs, and conditional inclusion.

s.files = [ ...lots of items... ]
s.files.concat [ ...more items... ]

if s.respond_to? :specification_version then
s.specification_version = 2
end

 

[Sean O'Halpin]

Hi Tom,

maybe the Doodle gem does what you're after:

require 'doodle'

class Contact < Doodle
has :email, :kind => String
has hone, :kind => String do
must "be of form xxx-xxx-xxxx" do |s|
s =~ /\d{3}-\d{3}-\d{4}/
end
end
end

class Person < Doodle
has :name, :kind => String
has :age, :kind => Integer
has Contact
end

class People < Doodle
has eople, :collect => Person
end

people = People do
person do
name "Joe Foo"
age 33
contact do
email "(e-mail address removed)"
phone "555-555-1234"
end
end
end

p people
__END__
#<People:0x8c22ec0 @people=[#<Person:0x8c31100 @name="Joe Foo",
@age=33, @contact=#<Contact:0x8c3ccbc @email="(e-mail address removed)",
@phone="555-555-1234">>]>

Regards,
Sean

 

[Intransition]

I think a Gemfile is a pretty good example

Do you mean a Gemfile from Bundler? e.g.

=A0 source "http://rubygems.org"

=A0 gem "rails", "3.0.0.rc"
=A0 gem "rack-cache"
=A0 gem "nokogiri", "~> 1.4.2"

Some attributes have one value, some attributes have more than one
value, some attributes can be repeated. I guess a generic parser API for
that would be a call to callback(key, *args), just like method_missing,
or it could build a linear data structure like

=A0 [[:source,"http://rubygems.org"],
=A0 =A0[:gem,"rack-cache"],
=A0 =A0[:gem,"nokogiri","~> 1.4.2"]]

I'm not sure what output you'd want from nested blocks.

I would expect to get an object that gave me access to the data. What
kind of object depends on the parser.

If the parser is generic (not application-specific) then I guess you'd
just get the structure I've shown above.

That's one way. The parser could just offer a couple of options to
vary how the generic structure is formed.

It *can* go through a parser for security reasons, but the result from
parsing it is exactly the same as if you read it in as a Javascript
literal. And there are cases where you intentionally interpret it as
Javascript code, e.g. JSONP.


Perhaps. I think more common would be a Rails-style configuration or a
gemspec:

Gem::Specification.new do |s|
=A0 s.name =3D %q{snailgun}
=A0 s.version =3D "1.0.6"
=A0 ... etc
end

That's another way to do it, yes. But can it be made a pure data
format?

Of course, the advantage of having it as real executable code is that
you can use Ruby to assemble data constructs, and conditional inclusion.

=A0 s.files =3D [ ...lots of items... ]
=A0 s.files.concat [ ...more items... ]

=A0 if s.respond_to? :specification_version then
=A0 =A0 s.specification_version =3D 2
=A0 end

But there's the disadvantage here too --it's no longer just data.
Depends on what you're trying to achieve.

 

[Intransition]

Hi Tom,

maybe the Doodle gem does what you're after:

require 'doodle'

class Contact < Doodle
=A0 has :email, :kind =3D> String
=A0 has hone, :kind =3D> String do
=A0 =A0 must "be of form xxx-xxx-xxxx" do |s|
=A0 =A0 =A0 s =3D~ /\d{3}-\d{3}-\d{4}/
=A0 =A0 end
=A0 end
end

class Person < Doodle
=A0 has :name, :kind =3D> String
=A0 has :age, :kind =3D> Integer
=A0 has Contact
end

class People < Doodle
=A0 has eople, :collect =3D> Person
end

people =3D People do
=A0 person do
=A0 =A0 name "Joe Foo"
=A0 =A0 age 33
=A0 =A0 contact do
=A0 =A0 =A0 email "(e-mail address removed)"
=A0 =A0 =A0 phone "555-555-1234"
=A0 =A0 end
=A0 end
end

p people
__END__
#<People:0x8c22ec0 @people=3D[#<Person:0x8c31100 @name=3D"Joe Foo",
@age=3D33, @contact=3D#<Contact:0x8c3ccbc @email=3D"(e-mail address removed)",
@phone=3D"555-555-1234">>]>

Regards,
Sean

Close. The only thing with Doodle is that you have to pre-define the
data structure. Hmm... Doodle would be something akin to a Schema
language for the format I am proposing.

 

[Intransition]

If yes: then it looks like you're proposing an alternative syntax for
Hash literals - one which isn't widely used. It's Ruby-inspired but not
really Ruby, since a simple eval of the above will fail without
additional supporting code.

That is, to parse that record using eval I think you need something
along these lines:

class Parser # < BasicObject ??
=A0 def self.parse(*args, &blk)
=A0 =A0 o =3D new
=A0 =A0 o.instance_eval(*args, &blk)
=A0 =A0 o.instance_variable_get@value)
=A0 end

=A0 def initialize
=A0 =A0 @value =3D {}
=A0 end

=A0 def method_missing(label, value=3D:__MISSING__,&blk)
=A0 =A0 if value !=3D :__MISSING__
=A0 =A0 =A0 raise "Cannot provide both value and block for #{label}" if
block_given?
=A0 =A0 =A0 @value[label] =3D value
=A0 =A0 else
=A0 =A0 =A0 raise "Must provide value or block for #{label}" unless
block_given?
=A0 =A0 =A0 @value[label] =3D self.class.parse(&blk)
=A0 =A0 end
=A0 end
end

person =3D Parser.parse <<'EOS'
=A0 name "Joe Foo"
=A0 age 33
=A0 contact do
=A0 =A0 email "(e-mail address removed)"
=A0 =A0 phione "555-555-1234"
=A0 end
EOS
p person

Now, to get the same capabilities as JSON, you also need syntax for
Arrays. You could do something like the following, although note that
the parser above doesn't handle this properly:

=A0 people([
=A0 =A0 person do
=A0 =A0 =A0 name "Joe"
=A0 =A0 =A0 age 33
=A0 =A0 end,
=A0 =A0 person do
=A0 =A0 =A0 name "Fred"
=A0 =A0 =A0 age 64
=A0 =A0 end,
=A0 ])

You do need both parentheses and square brackets, unless you replace
do/end with braces:

=A0 people [
=A0 =A0 person {
=A0 =A0 =A0 name "Joe"
=A0 =A0 =A0 age 33
=A0 =A0 },
=A0 =A0 person {
=A0 =A0 =A0 name "Fred"
=A0 =A0 =A0 age 64
=A0 =A0 },
=A0 ]

Add a few colons and commas and you're back to JSON. Or drop the closing
braces and brackets and keep the indentation, and you're back to YAML.

You could treat multiple arguments as an array:

=A0 people \
=A0 =A0 person {
=A0 =A0 =A0 name "Joe"
=A0 =A0 =A0 age 33
=A0 =A0 },
=A0 =A0 person {
=A0 =A0 =A0 name "Fred"
=A0 =A0 =A0 age 64
=A0 =A0 }

but then if you wanted a one-element array it would have to be a special
case. Or you could perhaps have a flag to indicate that you want an
Array:

=A0 people [],
=A0 =A0 person {
=A0 =A0 =A0 name "Joe"
=A0 =A0 =A0 age 33
=A0 =A0 },
=A0 =A0 person {
=A0 =A0 =A0 name "Fred"
=A0 =A0 =A0 age 64
=A0 =A0 }

You make a good point about arrays. I'm thinking about configuration
files as the common use case, so complex arrays aren't as common. But,
maybe there are better solutions in anycase:

people do
person :name=3D>"Joe", :age=3D>33,
preson :name=3D>"Fred", :age=3D> 64
end

The format is Ruby syntax so it can handle hash arguments. Or,
alternately:

people ["Joe", 33], ["Fred", 64]

The underlying app would know how to turn this into persons. We might
even do:

people ['name', 'age'],
['Joe', 33],
['Fred', 64]