File access denied after subprocess completion on Windows platform

[Claudiu Nicolaie CISMARU]

Hello,

I have a program that uses pyside for an QT interface and a thread that
downloads a lot of files. The thread is created with QThread object. But
my problem I don't think it's QT related.

The thread retrieves with pycurl a file that contains a list of files
and start to downloads them. The downloading is done as following:
- instantiate a Curl object
- open the file on local filesystem for write in binary mode (in a try
block), with the name suffixed with .part.
- pass the description to the curl object for save.
- curl retrieve and save it. It has also a callback function that
updates the interface, sending a QT signal to the interface.
(1) - use os.rename to rename the file with .part sufix to the final
file.

On my interface I have 3 buttons. One of the buttons runs an .exe file.
One button closes the interface and one is deactivated.

On the button that runs the exe I have a callback function that uses
subprocess.Popen (for not waiting) for running a program (.exe) and
returns. For now I configured to run calc.exe. The callback is not
defined inside the downloader thread. It's defined globally (nor in
QMainWindow object).

The problem appears when I close the called program (in our case
calc.exe). The (1) part (the call of os.rename) raise an exception:

<type 'exceptions.WindowsError'>
(32, 'The process cannot access the file because it is being used by
another process')
[Error 32] The process cannot access the file because it is being used
by another process

Question is why? And how to avoid this issue? The same program on Linux
works very fine (that's because Linux doesn't has this violation
access)! If I remove (1) part the program works fine. Somehow after
closing the spawned process (calc.exe - you see, it has nothing to do
with a open file somewhere else) the thread losses the acces to the
current opened file by itself.

--
Claudiu Nicolaie CISMARU
GNU GPG Key: http://claudiu.targujiu.net/key.gpg
T: +40 755 135455
E: (e-mail address removed), (e-mail address removed)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)

iEYEABECAAYFAk3bgfgACgkQhU2qCxfP7CrPPgCfWDXbBKZkS6M2WGcGyMG8Vkfi
SiYAn2Rmjty2ydRfp8qBe4F/EP/5TsyB
=Fc3a
-----END PGP SIGNATURE-----

 

[Claudiu Nicolaie CISMARU]

I'm quoting a message that I received on personal address and wasn't
sent to list:

try adding argument close_fds=True to subprocess.Popen

harish

And Tim's message:

It's not quite clear from your description above whether you
can be sure that the called subprocess has closed all its handles
by the time the os.rename runs.

Seems that close_fds did the trick. Anyway, I read that description on
the documentation last night but I think I was so tired that I
understood that in Windows has no effect...

Thank you, all.
--
Claudiu Nicolaie CISMARU
GNU GPG Key: http://claudiu.targujiu.net/key.gpg
T: +40 755 135455
E: (e-mail address removed), (e-mail address removed)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)

iEYEABECAAYFAk3b6CcACgkQhU2qCxfP7Cpt0ACgv3TJ06uM32/Y60G8ZP8Cl9xY
wSQAn0K2+tP6IECpkdeHUDa01AapUYjt
=ZYPP
-----END PGP SIGNATURE-----

 

[Claudiu Nicolaie CISMARU]

Seems that close_fds did the trick. Anyway, I read that description on

the documentation last night but I think I was so tired that I
understood that in Windows has no effect...

Now. There is one more issue. Seems that on faster computers and/or
Windows 7 (the Win32 thing I have tested on a HVM Xen machine with
Windows XP) the os.rename is too fast after fp.close() and generates the
same Exception. The code follows:

curl.close()
fp.close()
os.rename(tfile, actualfile)

Where, tfile is the .part file, actual file is the real destination, fp
was opened with open(..., "wb") and the descriptor passed to curl.

I have solved the issue with self.msleep(10) - msleep is a method of
QThread. But I don't think it's an elegant and normal solution. Did
fp.close() is delayed, or? I mean, I don't want to rely on a "sleep" in
order to workaround the access issue.

On this issue there is no more process spawn, nothing, just the
downloader thread and the main window. And the access denied appears at
random time.

--
Claudiu Nicolaie CISMARU
GNU GPG Key: http://claudiu.targujiu.net/key.gpg
T: +40 755 135455
E: (e-mail address removed), (e-mail address removed)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)

iEYEABECAAYFAk3cEpgACgkQhU2qCxfP7CqWVACgqp8EQ1VtJJHSvt+5xcOT53DW
IzQAnR5PpRhRXhzVPz68XImVUM5B4VeH
=tPKH
-----END PGP SIGNATURE-----

 

[Terry Reedy]

Now. There is one more issue. Seems that on faster computers and/or
Windows 7 (the Win32 thing I have tested on a HVM Xen machine with
Windows XP) the os.rename is too fast after fp.close() and generates the
same Exception. The code follows:

curl.close()
fp.close()
os.rename(tfile, actualfile)

Where, tfile is the .part file, actual file is the real destination, fp
was opened with open(..., "wb") and the descriptor passed to curl.

I have solved the issue with self.msleep(10) - msleep is a method of
QThread. But I don't think it's an elegant and normal solution. Did
fp.close() is delayed, or? I mean, I don't want to rely on a "sleep" in
order to workaround the access issue.

On this issue there is no more process spawn, nothing, just the
downloader thread and the main window. And the access denied appears at
random time.

I would go with what works. In my experience, mysterious and seemingly
buggy error messages, including Access Denied are not unusual on Windows.

 

[Tim Golden]

Now. There is one more issue. Seems that on faster computers and/or
Windows 7 (the Win32 thing I have tested on a HVM Xen machine with
Windows XP) the os.rename is too fast after fp.close() and generates the
same Exception. The code follows:

curl.close()
fp.close()
os.rename(tfile, actualfile)

Where, tfile is the .part file, actual file is the real destination, fp
was opened with open(..., "wb") and the descriptor passed to curl.

I have solved the issue with self.msleep(10) - msleep is a method of
QThread. But I don't think it's an elegant and normal solution. Did
fp.close() is delayed, or? I mean, I don't want to rely on a "sleep" in
order to workaround the access issue.

There used to be a problem with subprocess fds being held by
a traceback. IIRC, the problem could be triggered by having
an except clause around a subprocess call within which something
attempted to, eg,
remove one of the affected files. I'm sorry if that's a bit
of a woolly description but if you think this might be
biting you I'll dive in and look at the code. What version
of Python are you using?

(That said, the fact that the behaviour varies between faster
and slower computers makes that cause unlikely. Maybe we're
back to looking at virus checkers and the like...)

TJG

 

[Claudiu Nicolaie CISMARU]

There used to be a problem with subprocess fds being held by
a traceback. IIRC, the problem could be triggered by having
an except clause around a subprocess call within which something
attempted to, eg,
remove one of the affected files.

I have no subprocess call.. in this last issue. And I didn't triggered
one (disabled the Button that runs the subprocess).

try:
for line in t.splitlines():

[...]

ret = self.downloadFileToDisk(filename, do_rename)

if not ret:
print "DEBUG: Problema la download"
raise Exception()

(1) except Exception as inst:
print type(inst)
print inst.args

self.updateText.emit("EROARE: Eroare la descarcare")
self.updateStatusBar.emit("EROARE: Eroare la descaracare
fisiere")
return

Where downloadFileToDisk():

def downloadFileToDisk(self, filename, final_rename=True):
dfilename = os.path.join(saveBasePATH, filename)
sfilename = dfilename + ".part"

dfolder = os.path.dirname(sfilename)
if dfolder != "":
if not os.path.isdir(dfolder):
os.makedirs(dfolder)

try:
fp = open(sfilename, "wb")
except:
return False

curl = pycurl.Curl()

curl.setopt(pycurl.URL, baseUpdateURL + "/client/" + filename)
curl.setopt(pycurl.CONNECTTIMEOUT, 30)
curl.setopt(pycurl.NOPROGRESS, 0)
curl.setopt(pycurl.FOLLOWLOCATION, 1)
curl.setopt(pycurl.MAXREDIRS, 5)
curl.setopt(pycurl.PROGRESSFUNCTION, self.updateFileProgress)
curl.setopt(pycurl.WRITEDATA, fp)
curl.setopt(pycurl.BUFFERSIZE, 4194304)
curl.setopt(pycurl.NOSIGNAL, 1)
curl.perform()

retcode = curl.getinfo(pycurl.HTTP_CODE)

curl.close()

fp.close()

if retcode != 200:
(2) os.unlink(sfilename)
return False

self.msleep(10)

if final_rename:
os.rename(sfilename, dfilename)

return True

Without self.msleep(10), (1) catches WindowsError: file access ... blah
blah. Maybe at (2) can be an access violation, but it wasn't triggered
yet in tests. I will move the sleep after fp.close(). Yes, I know that
what it've done with raise Exception() is UGLY, since this program it's
more a quick hack solution to a problem . Anyway the Exception that is
catched is not rised by my code (it's a WindowsError).

I'm sorry if that's a bit
of a woolly description but if you think this might be
biting you I'll dive in and look at the code. What version
of Python are you using?

Last 2.7.x. Well, if os.rename is instead a subprocess call, then it's
subprocess based. I'm new to Python but 99% I think it's a system call

(That said, the fact that the behaviour varies between faster
and slower computers makes that cause unlikely. Maybe we're
back to looking at virus checkers and the like...)

On that virtual machine there is no virus checker. On the faster machine
I disabled and closed the antivirus.

--
Claudiu Nicolaie CISMARU
GNU GPG Key: http://claudiu.targujiu.net/key.gpg
T: +40 755 135455
E: (e-mail address removed), (e-mail address removed)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)

iEYEABECAAYFAk3cu98ACgkQhU2qCxfP7CorqACg8BXUtXsad6PfvulC6WX9w7tQ
6s4Ani3pwYRR4nGrjTRPatywPV60imyX
=yjTq
-----END PGP SIGNATURE-----