waiting for file lock?

[s99999999s2003]

hi
i need to go into a directory to grab some files and do some
processing.
The thing is, i need to wait till the process that generates the files
in that directory to finish
before i can grab the files. eg if file A is being generated and has
not finished, my python script will not go into the directory.
how can i check that file A has actually finished? thanks

 

[Lawrence D'Oliveiro]

i need to go into a directory to grab some files and do some
processing.
The thing is, i need to wait till the process that generates the files
in that directory to finish
before i can grab the files. eg if file A is being generated and has
not finished, my python script will not go into the directory.
how can i check that file A has actually finished?

I wrote a similar system that watches for new files arriving in an
"uploads" directory, whether copied there via FTP or using a GUI desktop
script. My heuristic was to only process files whose last-modified
date/time was at least 5 minutes in the past. My assumption was that it
was unlikely that 5 minutes would go by between more information being
added to a file.

 

[Christos Georgiou]

I wrote a similar system that watches for new files arriving in an
"uploads" directory, whether copied there via FTP or using a GUI desktop
script. My heuristic was to only process files whose last-modified
date/time was at least 5 minutes in the past. My assumption was that it
was unlikely that 5 minutes would go by between more information being
added to a file.

This works (unless there are long network timeouts, when downloading...),
but another idea is to wait for the existence of a zero-byte sentinel file
that is created last, after the transfer (or process, in general) has ended.

This method is the one that has worked best for me.

 

[Larry Bates]

hi
i need to go into a directory to grab some files and do some
processing.
The thing is, i need to wait till the process that generates the files
in that directory to finish
before i can grab the files. eg if file A is being generated and has
not finished, my python script will not go into the directory.
how can i check that file A has actually finished? thanks

I don't know if you have control of the program that is writing the
file. If you do, change it so that it writes the files with some
temporary name and renames them when it is done. Since rename is
basically atomic, the file won't show up as a candidate to be
processed until it is actually completed.

Something like:

Process A -> writes files into directory with .tmp extensions.
Process B -> scans directory but never processes anything with
.tmp extensions
Process A -> renames file.tmp to file when it is finished so
that process B picks it up on its next scan.


-Larry Bates