timing issue: shutil.rmtree and os.makedirs

[Tim]

I have the following function (Python2.7 on FreeBSD) that results in an OSError.

My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:

def make_clean_dir(directory):
if os.path.exists(directory):
if os.path.isdir(directory):
shutil.rmtree(directory)
else:
os.unlink(directory)
os.makedirs(directory)

The last bit of the traceback is:
File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
os.makedirs(directory)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'

The directory 'testing_html' existed when I executed the function;
So I suppose the directory wasn't finished being removed by the time os.makedirs was invoked. How can avoid this? (A context manager maybe?).

thanks,
--Tim

 

[Chris Angelico]

My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:

def make_clean_dir(directory):
if os.path.exists(directory):
if os.path.isdir(directory):
shutil.rmtree(directory)
else:
os.unlink(directory)
os.makedirs(directory)

The last bit of the traceback is:
File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
os.makedirs(directory)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'

The directory 'testing_html' existed when I executed the function;

First thing I'd check is: Did rmtree succeed? Try removing the
makedirs and test it again; then, when your process has completely
finished, see if the directory is there. If it is, the problem is in
rmtree - for instance:

* You might not have permission to remove everything
* There might be a messed-up object in the file system
* If the directory is a remote share mount point, the other end might
have lied about the removal
* Something might have been created inside the directory during the removal
* Myriad other possibilities

As I understand rmtree's docs, any errors *that it detects* will be
raised as exceptions (since you haven't told it to suppress or handle
them), but possibly there's an error that it isn't able to detect.
Worth a test, anyhow.

ChrisA

 

[Tim]

My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:
def make_clean_dir(directory):
if os.path.exists(directory):
if os.path.isdir(directory):
shutil.rmtree(directory)
else:
os.unlink(directory)
os.makedirs(directory)

The last bit of the traceback is:
File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
os.makedirs(directory)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'

The directory 'testing_html' existed when I executed the function;

First thing I'd check is: Did rmtree succeed? Try removing the
makedirs and test it again; then, when your process has completely
finished, see if the directory is there. If it is, the problem is in
rmtree - for instance:

* You might not have permission to remove everything
* There might be a messed-up object in the file system
* If the directory is a remote share mount point, the other end might
have lied about the removal
* Something might have been created inside the directory during the removal
* Myriad other possibilities
As I understand rmtree's docs, any errors *that it detects* will be
raised as exceptions (since you haven't told it to suppress or handle
them), but possibly there's an error that it isn't able to detect.
Worth a test, anyhow.

ChrisA

Thanks Chris, but the directory was actually removed on the first run in spite of the traceback; when I run it a second time (immediately after the first time), it runs fine. That's why I thought it was a timing issue. I thought about just putting a sleep in there, but that made me feel dirty.

hmm, now that you mention it, this is executing on a remote box with accessto the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.

But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.

thanks,
--Tim

 

[Chris Angelico]

hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.

But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.

Hmm. What system is used for the file system sharing? I know quite a
few of them lie about whether something's been completely done or not.

Can you use inotify to tell you when the directory's been deleted?
Seems stupid though.

Worst case, all you need is a quick loop at the bottom, eg:

for delay in 100,300,600,1000,3000,5000,10000:
if not os.path.exists(directory): break
sleep(delay)

That'll sleep a maximum of 20 seconds, tune as required. Of course, if
there's a way to tune the FS to guarantee that the removal blocks
correctly, that would be way better than sleep()!

ChrisA

 

[Steven D'Aprano]

for delay in 100,300,600,1000,3000,5000,10000:
if not os.path.exists(directory): break
sleep(delay)

That'll sleep a maximum of 20 seconds, tune as required.

Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7
minutes (assuming the directory doesn't get deleted instantaneously).

time.sleep() takes an argument in seconds.

 

[Chris Angelico]

Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7
minutes (assuming the directory doesn't get deleted instantaneously).

time.sleep() takes an argument in seconds.

LOL! Whoops. That's what I get for not checking my docs. This is why
we have public responses, my errors can be caught by someone else.

ChrisA

 

[Tim]

Hmm. What system is used for the file system sharing? I know quite a
few of them lie about whether something's been completely done or not.
Can you use inotify to tell you when the directory's been deleted?
Seems stupid though.
Worst case, all you need is a quick loop at the bottom, eg:

ChrisA

Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction).

thanks much!
--Tim

 

[Chris Angelico]

Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction).

I've used several different networked file systems, including
NetBIOS/NetBEUI/SMB/Samba/etc, sshfs/cifs, and nfs. Not one of them
"feels" as clean as I'd like, though sshfs comes closest. There always
seem to be hacks around.

ChrisA

 

[Göktuğ Kayaalp]

access to the same file system my local calling program is on. That is,
there is a local call to an intermediate script that connects to a socket
on the remote where the above program actually runs, but the file system is
the same place for both local and remote.on the same machine (even though it's remote), I would think the mkdir
couldn't execute until the rmtree was completely finished.

Hmm. What system is used for the file system sharing? I know quite a
few of them lie about whether something's been completely done or not.

Can you use inotify to tell you when the directory's been deleted?
Seems stupid though.

Inotify is a linux thing, but there is kqueue for Free?BSD. OP can run the
deletion procedure, wait for a NOTE_DELETE event, which would block, and
create the fresh directory afterwards. It may require some C hacking
though, in case Python lacks a kqueue wrapper. I think that this kind of
approach would be more sound than a check-wait-loop approach.

(I would elaborate more with pointers to appropriate documentation, but I'm
on a silly tablet, please excuse me for that.)

-gk