Reading a file that is changing and getting the new lines

[Ouray Viney]

Hi:

Problem:
=========
I want to read a ASCII text file that can have data appended to it. I
have hacked some code together that handles the basics, but it falls
short. My code doesn't read in the new lines that could have been
added to the end of the file. Not python's fault, more that I don't
know how to do it in python.

Example scenario: As the python script is running a user/application
adds new entries to the end of the test case file, example, adds the
following to the file.

test4
test5
test6

The python code below, won't pick these changes up.

Analysis:
========
I understand why my code doesn't handle the addition of new data.
This is simply because the os.open reads all the lines in the
beginning and since the data is added after that, the for loop doesn't
iterate through it since it doesn't know about it. Please note, my
code is not pretty and isn't at all advanced.

Code:
======
import os,string,sys,time

# define locals
tcListFile="testcase_list.txt"
fh=open(tcListFile,"r")
tcList=fh.readlines()
fh.close()

for tc in tcList:
print tc.strip()
print "sleeping for 2 seconds"
time.sleep(2)

# close the file handle
fh.close()

text file:
======
test1
test2
test3

Question:
=========
What is the best way to handle this type of scenario using python?

 

[Sean DiZazzo]

Problem:
=========
I want to read a ASCII text file that can have data appended to it.

Example scenario:  As the python script is running a user/application
adds new entries to the end of the test case file, example, adds the
following to the file.

Question:
=========
What is the best way to handle this type of scenario using python?

There was a thread regarding 'imitating "tail -f"' recently that you
might find useful.

The best approach IMO is this one: http://www.dabeaz.com/generators/follow.py

~Sean

 

[Dave Angel]

Hi:

Problem:
=========
I want to read a ASCII text file that can have data appended to it. I
have hacked some code together that handles the basics, but it falls
short. My code doesn't read in the new lines that could have been
added to the end of the file. Not python's fault, more that I don't
know how to do it in python.

Example scenario: As the python script is running a user/application
adds new entries to the end of the test case file, example, adds the
following to the file.

test4
test5
test6

I need clarification. Is this an independent process, that appends data
onto the same file, after the python script has opened it, but while
it's still running? Can you give me an example point in your source
code that your script might be running when the other application does
its append?

The python code below, won't pick these changes up.

Analysis:
========
I understand why my code doesn't handle the addition of new data.
This is simply because the os.open reads all the lines in the

os.open() doesn't read the lines, the readlines() call does. Once
you've got the data in a list, of course changing the file has no impact
on your code. And by the way, you didn't call os.open(), you called
open(). They are not interchangeable.

beginning and since the data is added after that, the for loop doesn't
iterate through it since it doesn't know about it. Please note, my
code is not pretty and isn't at all advanced.

Code:
======
import os,string,sys,time

# define locals
tcListFile="testcase_list.txt"
fh=open(tcListFile,"r")
tcList=fh.readlines()
fh.close()

for tc in tcList:
print tc.strip()
print "sleeping for 2 seconds"
time.sleep(2)

# close the file handle
fh.close()

text file:
======
test1
test2
test3

Question:
=========
What is the best way to handle this type of scenario using python?

You can solve your first level of problem by simply skipping the
readlines() call, and eliminating the first close() call. You close it
again later, let that be the only close. You want your loop to actually
get a line from the file, not from a list which is frozen in time.
Naturally, you'd want a better name than tcList.

Next problem is likely to be one of sharing the file. You open it
read-only, the other application simultaneously tries to append. Does
the OS permit that? You find out. You might actually have to close the
file during those sleep() calls, and then do an open/seek to get to the
earlier position. You most likely will have to switch from open() to
os.open(). The calls are very different.

Final problem that I note is how to decide when to quit your loop. Just
because you hit eof (end of file), doesn't mean the other application
isn't about to append another 50 lines. So you need a different
termination condition.

I suspect the sharing problems aren't going to be portable between
operating systems, so you might want to answer the usual "what version
of Python, on what OS" question that we seem to have for most new questions.

DaveA

 

[r0g]

Hi:

Problem:
=========
I want to read a ASCII text file that can have data appended to it. I
have hacked some code together that handles the basics, but it falls
short. My code doesn't read in the new lines that could have been
added to the end of the file. Not python's fault, more that I don't
know how to do it in python.

Example scenario: As the python script is running a user/application
adds new entries to the end of the test case file, example, adds the
following to the file.

test4
test5
test6

The python code below, won't pick these changes up.

Analysis:
========
I understand why my code doesn't handle the addition of new data.
This is simply because the os.open reads all the lines in the
beginning and since the data is added after that, the for loop doesn't
iterate through it since it doesn't know about it. Please note, my
code is not pretty and isn't at all advanced.

Code:
======
import os,string,sys,time

# define locals
tcListFile="testcase_list.txt"
fh=open(tcListFile,"r")
tcList=fh.readlines()
fh.close()

for tc in tcList:
print tc.strip()
print "sleeping for 2 seconds"
time.sleep(2)

# close the file handle
fh.close()

text file:
======
test1
test2
test3

Question:
=========
What is the best way to handle this type of scenario using python?

Read your file once.
Make a note of the length (X)
Close the file
Setup a loop with a sleep at the end.
Each time round check the file size
If it has increased, make a note of it (Y) and
open the file, seek to X and read Y - X bytes
This gives you the new data, you can then append it to the inital read
or whatever.

Do bar in mind though that all OS's buffer IO so the program that's
generating the file might have a different view of it than your python
script i.e. program A might output 300 chars to file but they may not be
written to disk until the buffer receives 4096 chars. If this might be a
problem to you make sure the program generating the data flushes the
cache after every write (sys.stdout.flush() in python).

Roger.