Windows process ownership trouble

[geoffbache]

Am currently being very confused over the following code on Windows

import subprocess, os

file = open("filename", "w")
try:
proc = subprocess.Popen("nosuchprogram", stdout=file)
except OSError:
file.close()
os.remove("filename")

This produces the following exception:

Traceback (most recent call last):
File "C:\processown.py", line 10, in <module>
os.remove("filename")
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process: 'filename'

How can it be in use by another process? The process didn't even
start, right?

Would appreciate some help: is this a Python bug, or a Windows bug, or
just me being confused...?

 

[Tim Golden]

Am currently being very confused over the following code on Windows

import subprocess, os

file = open("filename", "w")
try:
proc = subprocess.Popen("nosuchprogram", stdout=file)
except OSError:
file.close()
os.remove("filename")

This produces the following exception:

Traceback (most recent call last):
File "C:\processown.py", line 10, in <module>
os.remove("filename")
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process: 'filename'

How can it be in use by another process? The process didn't even
start, right?

[slight aside: best not to use "file" as an identifier; it shadows the
builtin "file" factory function. Not fatal, but might trip you up
somewhere later.]

This is awkward. What's happening behind the scenes is that,
as it creates the Popen object, the subprocess module gets
hold of the Windows handle behind your file. Then it calls
the CreateProcess API to create the process. If that succeeds,
it does whatever it does, closes the file handles and returns.
If it fails -- and here it fails, obviously -- it reraises the error
as a WindowsError. WindowsError is a subclass of OSError so
is trapped by your exception handler.

But -- and this is the point -- this exception is raised before the
the internal handles are closed, and altho' I don't know exactly
what effect this will have it clearly prevents the file from being
removed. And, as far as I can see, will continue to prevent its
being removed until the whole Python process exits and its
handles released.

I'm happy to be corrected on this, but it looks to me like a
bug in the subprocess error handling. If I'm right, I'm a little
surprised this hasn't bitten someone before but a quick search
of the bugs database doesn't seem to turn anything up.

TJG

 

[Tim Golden]

Am currently being very confused over the following code on Windows

import subprocess, os

file = open("filename", "w")
try:
proc = subprocess.Popen("nosuchprogram", stdout=file)
except OSError:
file.close()
os.remove("filename")

Forgot to say: slightly awkward, but you can work around
it like this:

<code>
import os
import subprocess

f = open ("filename", "w")
try:
proc = subprocess.Popen ("blah", stdout=f)
except OSError:
os.close (f.fileno ())

os.remove ("filename")

</code>

TJG

 

[geoffbache]

Thanks Tim, very helpful again.

I've now reported this as http://bugs.python.org/issue3210
and implemented your suggested workaround.

Regards,
Geoff

 

[geoffbache]

Tim,

Unfortunately my previous message was premature, it seems your
workaround doesn't work
either on my system (Windows XP, Python 2.5.1) I get the following
printed out

Traceback (most recent call last):
File "C:\TextTest\processown.py", line 12, in <module>
os.remove ("filename")
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process: 'filename'
close failed: [Errno 9] Bad file descriptor

Any ideas?

Geoff

 

[Tim Golden]

Tim,

Unfortunately my previous message was premature, it seems your
workaround doesn't work
either on my system (Windows XP, Python 2.5.1) I get the following
printed out

Traceback (most recent call last):
File "C:\TextTest\processown.py", line 12, in <module>
os.remove ("filename")
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process: 'filename'
close failed: [Errno 9] Bad file descriptor

Strange. I've just double-checked and it works perfectly
well for me, although I haven't traced all the way through
the code paths of the subprocess module: it was a bit of
an educated guess. Just to confirm, I'm talking about this
code running on Python 2.5.2 on Win XP SP2.

<code>
import os
import subprocess

f = open ("filename", "w")
try:
proc = subprocess.Popen ("blah", stdout=f)
except OSError:
os.close (f.fileno ())

os.remove ("filename")
</code>

If I get a chance I'll try to look more closely at the subprocess
code. Could you dump out the code (or the interpreter session
in its entirety) just to make sure there's not a typo or a thinko
getting in the way?

TJG

 

[Tim Golden]

Tim,

Unfortunately my previous message was premature, it seems your
workaround doesn't work
either on my system (Windows XP, Python 2.5.1) I get the following
printed out

Traceback (most recent call last):
File "C:\TextTest\processown.py", line 12, in <module>
os.remove ("filename")
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process: 'filename'
close failed: [Errno 9] Bad file descriptor

(Assuming you have the pywin32 extensions installed...)

If you tweak your copy of subprocess.py by switching on
the use of the pywin32 extensions in preference to the
_subprocess module, the problem goes away, I think
because the PyHANDLE object auto-closes. I'm not saying
it won't ever fail -- the interactions of objects, scope,
frames, exceptions and garbage collection are things I've
never looked at too closely. But if you change line 378 from
if 0: to if 1:, then the following works fine:

<code>
import os
import subprocess

f = open ("filename", "w")
try:
p = subprocess.Popen ("blah", stdout=f)
except WindowsError:
f.close ()

os.remove ("filename")

</code>

Note that the os.remove is *outside* the except handler.
This is, I think, because the underlying handle object
is still active until the try/except block closes. At which
point it is, probably, gc-ed and closes. And you can
remove the file.

TJG

 

[Tim Golden]

Tim,

Unfortunately my previous message was premature, it seems your
workaround doesn't work
either on my system (Windows XP, Python 2.5.1) I get the following
printed out

Traceback (most recent call last):
File "C:\TextTest\processown.py", line 12, in <module>
os.remove ("filename")
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process: 'filename'
close failed: [Errno 9] Bad file descriptor

[... snip purported solution using pywin32 ...]

..... or maybe I should just stop talking rubbish and go home
since that solution works even *without* the tweak to
subprocess.py.

TJG

 

[geoffbache]

Tim,

I copied your code exactly from my browser and ran it, so I don't
think there was a typo.
I could upgrade to Python 2.5.2 I suppose to compare and contrast, but
I need to support older
Python versions anyway so it's a bit academic...

Your speculation about garbage collection did set me going, however,
and I discovered
that the following code does work on my system, so now I have a
functional workaround:

import os
import subprocess

def runProcess():
f = open ("filename", "w")
try:
proc = subprocess.Popen ("blah", stdout=f)
except OSError:
f.close ()

runProcess()
os.remove ("filename")

So it seems that some things are only being garbage collected when the
function exits, but not when the
except clause exits or when the exception is thrown.

Geoff