stack-like file object

[Rodrigo Benenson]

Hi,
I need to implement a stack on a file.
The idea is to have a big list, and put part of his head on the disk. The
model of access to the file is like a stack (read in order only the tail,
write only at the tail).
How could I create this, knowing that I need to pickle arbritary objects ?

(to my eyes the problem is: how to pickle various objects in file and access
them as separate)

RodrigoB.

 

[Serge Orlov]

Hi,
I need to implement a stack on a file.
The idea is to have a big list, and put part of his head on the disk. The
model of access to the file is like a stack (read in order only the tail,
write only at the tail).

Why don't you use a collection of file organized as stack in one directory
or in many directories if your underlying file system cannot effectively
support many files in one directory. If your pickles are small (much less
than block size of the file system) you can keep several of them in one
file about the size of the block size. Since the files will be small you can
simply rewrite them competely each time you need change them.

-- Serge.

 

[Aahz]

I need to implement a stack on a file.

Why not use a real database?

 

[Josiah Carlson]

Rodrigo,

I need to implement a stack on a file.
The idea is to have a big list, and put part of his head on the disk. The
model of access to the file is like a stack (read in order only the tail,
write only at the tail).
How could I create this, knowing that I need to pickle arbritary objects ?

While I don't condone the use of files as stacks, the below should work
for you. It doesn't reduce in size when an object is removed, but as
long as you don't try to share the stack between processes or threads,
it should work fine.

- Josiah

import cPickle
import struct
import os

class filestack:
def __init__(self, filename):
try: os.remove(filename)
except: pass
self.fn = filename
self.f = open(filename, 'wb+')
self.s = len(struct.pack('!i', 0))

def __del__(self):
self.f.close()
del self.f
os.remove(self.fn)

def append(self, data):
st = cPickle.dumps(data)
self.write(st)
self.write(struct.pack('!i', len(st)))

def pop(self):
posn = self.f.tell()
if posn <= 0:
raise IndexError
self.f.seek(posn-self.s)
s = struct.unpack('!i', self.f.read(self.s))
self.f.seek(posn-self.s-s)
ret = cPickle.loads(self.f.read(s))
self.f.seek(posn-self.s-s)
return ret

 

[Jp Calderone]

Rodrigo,


While I don't condone the use of files as stacks, the below should work
for you. It doesn't reduce in size when an object is removed, but as
long as you don't try to share the stack between processes or threads,
it should work fine.

condone
v : excuse, overlook, or make allowances for; be lenient with;

[snip - file-based stack implementation]

Jp

 

[Josiah Carlson]

While I don't condone the use of files as stacks, the below should work

condone
v : excuse, overlook, or make allowances for; be lenient with;

Jp

I was using it in a similar tone as: "I don't condone the robbing of
banks, but below is a plan to do so, if one were to want to."

- Josiah