seeking thru a file

[Mag Gam]

I have a compressed CSV gziped file. I was wondering if it is possible
to seek thru a file

For example:

I want to load the first 100 lines into an array. Process the data

Seek from 101 line to 200 lines. Process the data (remove lines 0 -
100) from memory

Seek 201 to 300 line. Process the data (remove 200-300)) from memory

etc..etc..

 

[Bruno Desthuilliers]

Mag Gam a écrit :

I have a compressed CSV gziped file.

Then gunzip it first...

I was wondering if it is possible
to seek thru a file

For example:

I want to load the first 100 lines into an array. Process the data

Seek from 101 line to 200 lines. Process the data (remove lines 0 -
100) from memory

Seek 201 to 300 line. Process the data (remove 200-300)) from memory

etc..etc..

Yes, you can... Now is there any reason you want to micro-manage stuff
already managed by Python and the file system ?-)

To iterate on a text file's lines, just open the file - the file object
is it's own iterator, and will (with help from the filesystem AFAIK)
properly handle cache, buffers and whatnot. The canonical code snippet is:

# iterating over a file line by line without
# loading the while file in memory
f = open("/path/to/some/file.txt")
for line in f:
do_something_with(line)

f.close()


Unless you have a good reason (micro-managing IO buffers not being one)
to want to work by 100-lines batch, then this should be just enough.

Now note that Python also have a csv module in the standard lib, which
will probably (except perhaps in a very few corner cases) be more robust
and efficient that whatever you'd write.


HTH