Multi-threaded printing and UIErrors in Windows

[Roger Binns]

My program has a random smattering of print statements
scattered throughout that are mainly a debugging aid.
They are typically things like this:

print "entering key mode"

The code is multithreaded, and the printing typically
happens in both the main thread and a worker thread
doing serial port IO. The problem I am experiencing
when the program is packaged up with py2exe is that
exceptions happen:

IOError: [Errno 9] Bad file descriptor

This can happen in either thread. Why is this
happening? It doesn't happen on Linux or Mac,
and hasn't happened when not using py2exe.

Python 2.3, wxPython 2.4, py2exe 0.5 (and the
previous version).

Roger

 

[David Fraser]

My program has a random smattering of print statements
scattered throughout that are mainly a debugging aid.
They are typically things like this:

print "entering key mode"

The code is multithreaded, and the printing typically
happens in both the main thread and a worker thread
doing serial port IO. The problem I am experiencing
when the program is packaged up with py2exe is that
exceptions happen:

IOError: [Errno 9] Bad file descriptor

This can happen in either thread. Why is this
happening? It doesn't happen on Linux or Mac,
and hasn't happened when not using py2exe.

Python 2.3, wxPython 2.4, py2exe 0.5 (and the
previous version).

Roger

sys.stdout is not threadsafe - try wrapping each call with a mutext and
see if that fixes it...

 

[Roger Binns]

sys.stdout is not threadsafe - try wrapping each call with a mutext and
see if that fixes it...

Having to wrap hundreds of random print statements just using for
debugging seems really silly. Note that I don't care if the
output gets garbled. (It doesn't on Mac or Linux ever).

And I am only hitting this problem on Windows when wrapped
with py2exe. If running outside of py2exe there is never
any problem.

When using py2exe, the stdout doesn't go anywhere anyway
(it is gui app not a console app).

The bizarre thing is that the IOError is very infrequent,
but very annoying when it happens.

Roger

 

[Thomas Heller]

Having to wrap hundreds of random print statements just using for
debugging seems really silly. Note that I don't care if the
output gets garbled. (It doesn't on Mac or Linux ever).

And I am only hitting this problem on Windows when wrapped
with py2exe. If running outside of py2exe there is never
any problem.

When using py2exe, the stdout doesn't go anywhere anyway
(it is gui app not a console app).

The bizarre thing is that the IOError is very infrequent,
but very annoying when it happens.

If you have a small program which allows to reproduce this, I'll look
into it.

Thomas

 

[David Bolen]

When using py2exe, the stdout doesn't go anywhere anyway
(it is gui app not a console app).

The bizarre thing is that the IOError is very infrequent,
but very annoying when it happens.

Have you tried replacing sys.stdout with your own object that just
throws away anything it is given? If you're not getting the output
anywhere anyway, it would avoid any attempt to actually use the
underlying I/O facilities. You could conditionalize this code based
on whether or not you were running under py2exe.

I know there can be an issue under Windows under some startup
scenarios where you don't actually have opened O/S filesystem handles
for the handles underlying stdin/stdout (which can cause this sort of
problem), such as running as a service, but haven't normally had this
problem with GUI applications. Although it may matter on how you are
starting the application (from a command line, the start menu, as a
service, etc...), since it will inherit its parent environment.

If that's the problem, I'd expect it to happen any time actual I/O was
attempted. However, since there is probably some buffering going on
beneath the level of print within the underlying C library, maybe the
randomness of the error you are seeing is due to how long it takes to
print enough data to cause the C library to attempt to flush the data
to the underlying O/S filesystem handle. If that were the case, doing
some test prints in your code of very long strings (the C buffer is
probably anywhere from 2-8K) should be able to reproduce the problem
more reliably.

-- David

 

[David Bolen]

Having to wrap hundreds of random print statements just using for
debugging seems really silly. Note that I don't care if the
output gets garbled. (It doesn't on Mac or Linux ever).

To follow up on this point to my previous reply - I've never found
sys.stdout not to be threadsafe, at least in terms of how I mean
"threadsafe" - that is generating exceptions or crashes.

If instead, threadsafe is meant to imply atomic I/O (guaranteed output
of entire I/O as a unit) which is a broader definition, then yes, it
is definitely true that multiple threads all using sys.stdout may have
their output interleaved, particularly when you are using the 'print'
statement, since a single print statement generates multiple
individual writes to sys.stdout, generally for each argument, as well
as the trailing newline.

-- David

 

[Roger Binns]

If you have a small program which allows to reproduce this, I'll look
into it.

I cannot get it to repeat with test programs I have written (and boy have
I tried). Not very many users reported the problem. It is probably
some sort of race condition somewhere, and possibly only occurs
on slower machines. My real program has many wxPython OnIdle handlers
which is predominantly where the error shows up combined with a background
thread doing heavy serial port activity to a cell phone, also doing
prints.

Is it possible for py2exe to replace stdout with something that
just ignores all writes, as opposed to None (or is that what
is already happening).

Roger