[ANN] Mongoose 0.2.0

[Jamey Cribbs]

You can download it from: http://rubyforge.org/projects/mongoose/

*What's New*

Thanks, to Ezra Zygmuntowicz, this release features a much cleaner query
language, so I decided to get this out before anyone gets too used to
the old version. Ezra pointed out how I could eliminate having to pass
the table's class as a block parameter to the query, and also eliminate
having to qualify every column name in the query with the table's
class. So, a query that looked like this:

Plane.find do |plane|
plane.speed > 350
plane.any do
plane.country == 'USA'
plane.country == 'Great Britain'
end
end

Now, looks like this:

Plane.find do
speed > 350
any do
country == 'USA'
country == 'Great Britain'
end
end

Much cleaner. Thanks Ezra!

Also, Ezra contributed a Table.create method and Logan Capaldo
contributed a fix to eliminate a string eval. There is a new example
script from Daniel Sheppard showing how to integrate ActiveRecord
validations into Mongoose. Plus the usual code cleanup and bug fixes.

*What is Mongoose?*

Mongoose is a database management system written in Ruby. It has an
ActiveRecord-like interface, uses Skiplists for its indexing, and
Marshal for its data serialization. I named it Mongoose, because, like
Rudyard Kipling's Rikki-Tikki-Tavi, my aim is for it to be small, quick,
and friendly.

You can find rudimentary documentation in the README file and some
sample scripts in the example directory.

Jamey Cribbs
(e-mail address removed)

 

[Logan Capaldo]

You can download it from: http://rubyforge.org/projects/mongoose/

*What's New*

Thanks, to Ezra Zygmuntowicz, this release features a much cleaner
query language, so I decided to get this out before anyone gets too
used to the old version. Ezra pointed out how I could eliminate
having to pass the table's class as a block parameter to the query,
and also eliminate having to qualify every column name in the query
with the table's class. So, a query that looked like this:

Plane.find do |plane|
plane.speed > 350
plane.any do
plane.country == 'USA' plane.country == 'Great Britain'
end
end

Now, looks like this:

Plane.find do
speed > 350
any do
country == 'USA'
country == 'Great Britain'
end
end

Much cleaner. Thanks Ezra!

Also, Ezra contributed a Table.create method and Logan Capaldo
contributed a fix to eliminate a string eval. There is a new
example script from Daniel Sheppard showing how to integrate
ActiveRecord validations into Mongoose. Plus the usual code
cleanup and bug fixes.

*What is Mongoose?*

Mongoose is a database management system written in Ruby. It has
an ActiveRecord-like interface, uses Skiplists for its indexing,
and Marshal for its data serialization. I named it Mongoose,
because, like Rudyard Kipling's Rikki-Tikki-Tavi, my aim is for it
to be small, quick, and friendly.

You can find rudimentary documentation in the README file and some
sample scripts in the example directory.

Jamey Cribbs
(e-mail address removed)

I'm just too slow with sending in my patches.

 

[simonh]

Jamey, just out of curiosity, how long did it take you to write
mongoose 0.1.0? Also Do you think mongoose will be more maintainable
now its split into multiple files?

 

[Jamey Cribbs]

Jamey, just out of curiosity, how long did it take you to write
mongoose 0.1.0? Also Do you think mongoose will be more maintainable
now its split into multiple files?

I probably started working on it the middle of last week and have been
working on it in most of my spare time (probably a 2-3 hours a day; my
wife would probably say more like 5-6 hours a day ). However, over
the last few months, I had already been experimenting with Skiplists and
with using Marshal for storage. Also, I was able to borrow some great
code from Logan Capaldo's KirbyRecord.

Yes, I do think it will be more maintainable. KirbyBase is one BIG file
and, so far, I definitely like the feel of having Mongoose split up into
different files for different components.

Jamey

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient(s), you are hereby notified that any dissemination, unauthorized review, use, disclosure or distribution of this email and any materials contained in any attachments is prohibited. If you receive this message in error, or are not the intended recipient(s), please immediately notify the sender by email and destroy all copies of the original message, including attachments.

 

[Logan Capaldo]

I probably started working on it the middle of last week and have
been working on it in most of my spare time (probably a 2-3 hours a
day; my wife would probably say more like 5-6 hours a day ).
However, over the last few months, I had already been experimenting
with Skiplists and with using Marshal for storage. Also, I was
able to borrow some great code from Logan Capaldo's KirbyRecord.

Yes, I do think it will be more maintainable. KirbyBase is one BIG
file and, so far, I definitely like the feel of having Mongoose
split up into different files for different components.

Jamey

Confidentiality Notice: This email message, including any
attachments, is for the sole use of the intended recipient(s) and
may contain confidential and/or privileged information. If you are
not the intended recipient(s), you are hereby notified that any
dissemination, unauthorized review, use, disclosure or distribution
of this email and any materials contained in any attachments is
prohibited. If you receive this message in error, or are not the
intended recipient(s), please immediately notify the sender by
email and destroy all copies of the original message, including
attachments.

Jamey, while we are on the topic of Mongoose, since the query
language isn't straight up "loop thru all the records of the table,
pass the row into this proc and see if it returns true", any chance
of query optimization and or arbitrary joins? If not, would you
object if I tried my hand at adding joins and query optimization?
(This is not a promise, it's more like if I have time it will slowly
start to show up).

I have a couple of ideas, One involving how to do Hash-based equi-
joins and cheat a little bit, if what i believe about Marshal is true
and reliable.

 

[John W. Long]

Plane.find do |plane|
plane.speed > 350
plane.any do
plane.country == 'USA' plane.country == 'Great Britain'
end
end

Now, looks like this:

Plane.find do
speed > 350
any do
country == 'USA'
country == 'Great Britain'
end
end

James, could you elaborate on how this is done? I would assume that the
first method is better because you don't lose the the surrounding scope.
What's going on in that do block?

 

[Jamey Cribbs]

Jamey, while we are on the topic of Mongoose, since the query language
isn't straight up "loop thru all the records of the table, pass the
row into this proc and see if it returns true", any chance of query
optimization and or arbitrary joins? If not, would you object if I
tried my hand at adding joins and query optimization? (This is not a
promise, it's more like if I have time it will slowly start to show up).

That would be great. That's one of the things I am really excited about
Mongoose is that, now that I can look at the query *before* it is
executed (unlike KirbyBase), it can be optimized. I've already got a
few ideas about query optimization, but I would definitely welcome both
ideas and code.

I have a couple of ideas, One involving how to do Hash-based
equi-joins and cheat a little bit, if what i believe about Marshal is
true and reliable.

Sounds good!

Jamey

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient(s), you are hereby notified that any dissemination, unauthorized review, use, disclosure or distribution of this email and any materials contained in any attachments is prohibited. If you receive this message in error, or are not the intended recipient(s), please immediately notify the sender by email and destroy all copies of the original message, including attachments.

 

[Jamey Cribbs]

country in ['USA','Great Britian']
would be nice!

I will add it to the feature requests list.

By the way, you can currently do:

country.one_of 'USA', 'Great Britain'

which is pretty close.

Jamey

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient(s), you are hereby notified that any dissemination, unauthorized review, use, disclosure or distribution of this email and any materials contained in any attachments is prohibited. If you receive this message in error, or are not the intended recipient(s), please immediately notify the sender by email and destroy all copies of the original message, including attachments.

 

[Jamey Cribbs]

James, could you elaborate on how this is done? I would assume that
the first method is better because you don't lose the the surrounding
scope. What's going on in that do block?

Actually, I like the second way better, because you don't have to
clutter up your query with lots of references to your table class.

The way it works is that in the Plane.find method, I grab the block and
pass it to #instance_eval. This executes the block in the Plane classes
environment, so it knows that when it sees the attribute "speed" or
"country" in the block, it knows to call Plane.speed and Plane.country.
Planes.speed happens to be a reference to a SkiplistColumn instance and
it knows how to handle the "> 350" passed to it.

I have to credit Ezra for pointing me to this.

The reference to "any" just calls Plane.any which is a method in the
Plane class that knows how to take the block inside of it and format it
as a sub-query of the main query.

Jamey

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient(s), you are hereby notified that any dissemination, unauthorized review, use, disclosure or distribution of this email and any materials contained in any attachments is prohibited. If you receive this message in error, or are not the intended recipient(s), please immediately notify the sender by email and destroy all copies of the original message, including attachments.

 

[John W. Long]

The way it works is that in the Plane.find method, I grab the block and
pass it to #instance_eval. This executes the block in the Plane classes
environment, so it knows that when it sees the attribute "speed" or
"country" in the block, it knows to call Plane.speed and Plane.country.
Planes.speed happens to be a reference to a SkiplistColumn instance and
it knows how to handle the "> 350" passed to it.

The problem with this approach is that you loose access to instance
variables. For instance:
=> nil

In the above example @test when used within the context of the with
block is thought to be an instance variable of the object, which is why
it returns nil instead of 1. I consider this behavior undesirable.

 

[Jamey Cribbs]

The problem with this approach is that you loose access to instance
variables. For instance:

=> nil

In the above example @test when used within the context of the with
block is thought to be an instance variable of the object, which is
why it returns nil instead of 1. I consider this behavior undesirable.

Hmm, I see your point. I've been playing around with this all morning,
but I can't see a way out of this except to go back to passing the Table
class to the query block and also qualifying the table column names in
the block.

Even though it does clutter up the query block more, I think John is
right that it is definitely more important to have access to the
instance variables in the calling object.

Thanks for pointing this out to me, John.

Jamey

 

[Logan Capaldo]

The problem with this approach is that you loose access to instance
variables. For instance:

=> nil

In the above example @test when used within the context of the with
block is thought to be an instance variable of the object, which is
why it returns nil instead of 1. I consider this behavior undesirable.

I can think of how to do and operations with a parameter passing
style w/o instance_eval, and how to do _ors_ but not both at the
same time. Unless we want to get rid of the any method and use pipe
as logical or. The problem with that is now you can't do bitfield
comparisons. Say you store a Fixnum with some flags in the the db:

find do |plane|
( plane.bitfield | 0b100 ) == 0b100
end

How do we do ors now w/o instance_eval?

any could be toplevel, but that's very polluting of the global
namespace.

*shrug*

 

[Bob Hutchison]

Hi,

The problem with this approach is that you loose access to instance
variables. For instance:

=> nil

In the above example @test when used within the context of the with
block is thought to be an instance variable of the object, which is
why it returns nil instead of 1. I consider this behavior undesirable.

Is this really a problem? If we write it this way:

class Object
def with(&block)
instance_eval(&block)
end
end

class Toy
def play
@test = 'hello there'
with { puts "!!!! [#{@test}] !!!!" }
end
end

toy = Toy.new
toy.play

you get the output:

!!!! [hello there] !!!!

which is using the @test as I think you want.

Cheers,
Bob


----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>

 

[Jamey Cribbs]

Hi,

The problem with this approach is that you loose access to instance
variables. For instance:

=> nil

In the above example @test when used within the context of the with
block is thought to be an instance variable of the object, which is
why it returns nil instead of 1. I consider this behavior undesirable.

Is this really a problem? If we write it this way:

class Object
def with(&block)
instance_eval(&block)
end
end

class Toy
def play
@test = 'hello there'
with { puts "!!!! [#{@test}] !!!!" }
end
end

toy = Toy.new
toy.play

you get the output:

!!!! [hello there] !!!!

which is using the @test as I think you want.

I think the following code is an example of what John is pointing out:

class Employee
def self.find(&block)
instance_eval(&block)
end
end

class Department
def initialize
@dept = 'Accounting'
end

def employees
Employee.find { puts "employee.dept == #{@dept}" }
end
end

Department.new.employees

Run this and you get:

employee.dept ==

Replace that instance_eval in Employee.find with a yield statement, run
it again, and you get:

employee.dept == Accounting


If I could be sure that everyone would only do a Table.find in Mongoose
from the top level environment, so that variables that they declared
would be seen inside the Table class, I would be ok. But, what happens
when someone, like the example above, is doing a Table.find from within
another object? If I use #instance_eval inside of Table.find, I lose
access to the instance variables from the calling object.

No, it looks like I will have to go back to doing a yield in Table.find,
which means the user is going to have to specify a block parameter in
the #find block. In other words:

Employee.find { |emp| emp.dept == @dept }

I guess it doesn't look that bad.

Jamey

 

[Caleb Clausen]

No, it looks like I will have to go back to doing a yield in Table.find,
which means the user is going to have to specify a block parameter in
the #find block. In other words:

Employee.find { |emp| emp.dept == @dept }

I guess it doesn't look that bad.

Ok, this is a totally different sort of query language, but have you
considered something like Reg? (http://rubyforge.org/projects/reg/ or
gem install reg) I took a more declarative approach (at least, I think
so) to a query languange, which (IMNSHO) is a better fit with the
problem. Reg is entirely based off #===, so all you'd need to do is
allow your find to take a parameter which responds to #===. That would
make the above example look like:

Employee.find(item_that.dept==@dept)

or alternately:

Employee.find(-{:dept=>@dept})

This should solve your problem with instance variable visibility in
blocks, since there are no blocks, but does it create other problems?
I don't know.

Reg is entirely designed from the point of view of queries on
in-memory Ruby object trees, so that's a little different than a
database model. But maybe you can get some ideas if nothing else...

I don't know anything about skiplists or database query optimization,
but Reg expressions look like nicely parsed trees of objects, so that
ought to be something you can work with...

 

[Bob Hutchison]

I think the following code is an example of what John is pointing out:

class Employee
def self.find(&block)
instance_eval(&block)
end
end

class Department
def initialize
@dept = 'Accounting'
end

def employees
Employee.find { puts "employee.dept == #{@dept}" }
end
end

Department.new.employees

Run this and you get:

employee.dept ==

Replace that instance_eval in Employee.find with a yield statement,
run it again, and you get:

employee.dept == Accounting

Hmm. Right.

Okay, though this is getting perverse...

module Mongoose
def find(klass, &block)
klass.mongoose_find(self, block)
end

module ClassMethods
def mongoose_find(obj, block)
obj.instance_eval(&block)
end
def check_find(block)
instance_eval(&block)
end
end

def self.included(klass);
klass.extend(ClassMethods);
end
end

class Employee
include Mongoose
end

class Department
include Mongoose

def initialize
@dept = 'Accounting'
end

def employees
find(Employee) { puts "FIND: employee.dept == #{@dept}" }
end
end

Department.new.employees


I don't know why I'm arguing this because I actually like the
Employee.find { |emp| emp.dept == @dept } syntax better anyway.

Cheers,
Bob

----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>

 

[Jamey Cribbs]

module Mongoose
def find(klass, &block)
klass.mongoose_find(self, block)
end

module ClassMethods
def mongoose_find(obj, block)
obj.instance_eval(&block)
end
def check_find(block)
instance_eval(&block)
end
end

def self.included(klass);
klass.extend(ClassMethods);
end
end

class Employee
include Mongoose
end

class Department
include Mongoose

def initialize
@dept = 'Accounting'
end

def employees
find(Employee) { puts "FIND: employee.dept == #{@dept}" }
end
end

Department.new.employees

Thanks! I will study this and try to figure out what you are doing.

I don't know why I'm arguing this because I actually like the
Employee.find { |emp| emp.dept == @dept } syntax better anyway.

What's the old saying about climbing Mt. Everest? "Because it's
there!"

Jamey

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient(s), you are hereby notified that any dissemination, unauthorized review, use, disclosure or distribution of this email and any materials contained in any attachments is prohibited. If you receive this message in error, or are not the intended recipient(s), please immediately notify the sender by email and destroy all copies of the original message, including attachments.

 

[Bob Hutchison]

What's the old saying about climbing Mt. Everest? "Because it's
there!"

I think that must be it

BTW, the check_find method isn't used so I should have deleted it
before posting.

Cheers,
Bob

Jamey

----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>