key/value store optimized for disk storage

[Jon Clements]

Thanks. I think I'm ruling out bsddb, since it's recently deprecated:

http://www.gossamer-threads.com/lists/python/python/106494

I'll give sqlite3 a spin. Has anybody out there wrapped sqlite3
behind a hash interface already? I know it's simple to do
conceptually, but there are some minor details to work out for large
amounts of data (like creating the index after all the inserts), so if
somebody's already tackled this, it would be useful to see their
code.


Thanks.

Could also look at Tokyo cabinet or Kyoto cabinet (but I believe that has slightly different licensing conditions for commercial use).

 

[John Nagle]

That's awful. There's no point in compressing six characters
with zlib. Zlib has a minimum overhead of 11 bytes. You just
made the data bigger.

John Nagle

 

[Paul Rubin]

That's awful. There's no point in compressing six characters
with zlib. Zlib has a minimum overhead of 11 bytes. You just
made the data bigger.

This hack is about avoiding the initialization overhead--do you really
get 11 bytes after every SYNC_FLUSH? I do remember with that SYNC_FLUSH
trick, that I got good compression with a few hundred bytes of input
(the records I was dealing with at the time).

 

[Steve Howell]

    That's awful. There's no point in compressing six characters
with zlib.  Zlib has a minimum overhead of 11 bytes.  You just
made the data bigger.

The actual strings that I'm compressing are much longer than six
characters. Obviously, "foobar" was just for example purposes.