shelf-like list?

[kj]

I'm looking for a module that implements "persistent lists": objects
that behave like lists except that all their elements are stored
on disk. IOW, the equivalent of "shelves", but for lists rather
than a dictionaries.

Does anyone know of such a module?

(I suppose that I could slap together a crude implementation of
such a thing by wrapping a shelf with suitable methods to simulate
the list interface. But I'd rather not roll my own if a tested
implementation already exist.)

TIA!

~K

 

[Thomas Jollans]

I'm looking for a module that implements "persistent lists": objects
that behave like lists except that all their elements are stored
on disk. IOW, the equivalent of "shelves", but for lists rather
than a dictionaries.

Does anyone know of such a module?

(I suppose that I could slap together a crude implementation of
such a thing by wrapping a shelf with suitable methods to simulate
the list interface. But I'd rather not roll my own if a tested
implementation already exist.)

You could simply use pickle to save the data every once in a while.

 

[Raymond Hettinger]

. . .
You could simply use pickle to save the data every once in a while.

That is a very reasonable solution.

 

[kj]

That is a very reasonable solution.

Sorry I don't follow. Some sample code would be helpful.
TIA,
~K

 

[Chris Rebert]

Sorry I don't follow.  Some sample code would be helpful.

I would assume something along the lines of (untested):

from pickle import dump

MOD_THRESHOLD = 42

class PersistentList(list):
def __init__(self, filepath):
self.filepath = filepath
self._mod_count = 0

def save(self):
with open(self.filepath, 'w') as f:
dump(self, f)
self._mod_count = 0

def append(self, *args, **kwds):
super(PersistentList, self).append(*args, **kwds)
self._mod_count += 1
if self._mod_count >= MOD_THRESHOLD:
# obviously time-since-last-dump or other
# more sophisticated metrics might be used instead
self.save()
# define similar wrappers for list's other mutator methods:
__delitem__, __iadd__, __imul__, __setitem__, extend, insert, pop,
remove, etc.
# using decorators should help decrease code duplication

Cheers,
Chris

 

[Martin v. Loewis]

Does anyone know of such a module?

ZODB supports persistent lists.

Regards,
Martin

 

[kj]

I would assume something along the lines of (untested):

from pickle import dump

MOD_THRESHOLD =3D 42

class PersistentList(list):
def __init__(self, filepath):
self.filepath =3D filepath
self._mod_count =3D 0

def save(self):
with open(self.filepath, 'w') as f:
dump(self, f)
self._mod_count =3D 0

def append(self, *args, **kwds):
super(PersistentList, self).append(*args, **kwds)
self._mod_count +=3D 1
if self._mod_count >=3D MOD_THRESHOLD:
# obviously time-since-last-dump or other
# more sophisticated metrics might be used instead
self.save()

Even though it is saved periodically to disk, it looks like the
whole list remains in memory all the time? (If so, it's not what
I'm looking for; the whole point of saving stuff to disk is to keep
the list's memory footprint low.)

~K

 

[kj]

ZODB supports persistent lists.

Thanks; I'll check it out.

~K

 

[Dave Angel]

Even though it is saved periodically to disk, it looks like the
whole list remains in memory all the time? (If so, it's not what
I'm looking for; the whole point of saving stuff to disk is to keep
the list's memory footprint low.)

~K

It sounds like we all made the wrong assumption about your
requirements. It's not persistence you want, but low memory footprint
? Do you or don't you also want the data to survive multiple runs of
the program?

If you want the data to be in a persistent disk file, you probably
should use a database. And if you don't, then you could either increase
the size of your swapfile, or use a more complex scheme to do a memory
mapped file while the program is running, and delete it on exit. The
swapfile is likely to be execute more quickly than anything else you
could cook up in pure python.

DaveA

 

[Emile van Sebille]

ZODB supports persistent lists.

Thanks; I'll check it out.
[/QUOTE]

I wouldn't expect a low memory footprint however.

Emile