file-backed array of classes?

[Rasputin]

I was looking at prototyping a write once file service, a bit like
Plan9s Venti system. I delete files a lot
As usual, my grand designs are thwarted by the basics...

I'm at the 'get the unit tests of the simplest possible implementation to
pass' stage, and need something like an append-only file of fixed
size records (data blocks). Have we got classes or methods to do
record-based file IO? I dont mind reinventing a wheel otherwise.

Next problem (I learned to never look more than 1 problem ahead or I
never start anything). I need an index that gets dynamically updated as
the record file is written to.

The ideal solution would be a hash that was backed by a file somehow-
then I realised I could implement the data blocks themselves
the same way.

I looked through pickAxe and RAA for these two and found a lot of SQL
persistence APIs, but that feels like overkill....

Does anyone know of a more lightweight solution?

 

[Robert Klemme]

I was looking at prototyping a write once file service, a bit like
Plan9s Venti system. I delete files a lot
As usual, my grand designs are thwarted by the basics...

I'm at the 'get the unit tests of the simplest possible implementation to
pass' stage, and need something like an append-only file of fixed
size records (data blocks). Have we got classes or methods to do
record-based file IO? I dont mind reinventing a wheel otherwise.

A typical solution might involve IO with binary mode with Array#pack and
String.unpack. Then you'll have to do the byte layout and record size
fixing yourself.

Or you can use Marshalling for this: define a class and marshal instances
one at a time into the file:

st = Struct.new( "FooRecord", :name, :age )
records = (1..10).map{|i| st.new("hello", i)}

File.open("foo.bin", "wb") do |io|
records.each do |rec|
Marshal.dump(rec, io)
end
end

File.open("foo.bin", "rb") do |io|
until io.eof?
obj = Marshal.load(io)
puts obj
end
end

Of courese you can marshal a complete array, too. But then, all instances
are loaded at once and with the approach above you can read one record at
a time, thus navigating (not very efficient though).

Next problem (I learned to never look more than 1 problem ahead or I
never start anything). I need an index that gets dynamically updated as
the record file is written to.

If the dataset will grow not too big, you could use a Hash and Marshal
that in a file...

The ideal solution would be a hash that was backed by a file somehow-
then I realised I could implement the data blocks themselves
the same way.

I looked through pickAxe and RAA for these two and found a lot of SQL
persistence APIs, but that feels like overkill....

Does anyone know of a more lightweight solution?

Not perfectly what you are looking for but maybe some ideas to play
with...

robert

 

[Neil Mc Laughlin]

Have you considered PStore in the standard library?


----- Original Message -----
From: "Rasputin" <[email protected]>
To: "ruby-talk ML" <[email protected]>
Sent: Friday, November 14, 2003 11:59 AM
Subject: file-backed array of classes?

 

[Simon Strandgaard]

I'm at the 'get the unit tests of the simplest possible implementation to
pass' stage, and need something like an append-only file of fixed
size records (data blocks). Have we got classes or methods to do
record-based file IO? I dont mind reinventing a wheel otherwise.

record-based file IO, can you explain further?

Next problem (I learned to never look more than 1 problem ahead or I
never start anything). I need an index that gets dynamically updated as
the record file is written to.

I have written a robust-iterator primitive, where iterators gets notified
when somebody insert/delete data, so they iterator keeps pointing at the
same position.

http://rubyforge.org/cgi-bin/viewcv...jects/experimental/iterator2/?cvsroot=aeditor