Overriding Thread.join in derived class.

[Graham Dumpleton]

If one creates a thread using threading.Thread and makes it a daemon
by calling setDaemon(), then when Python is exiting it will not
attempt to call join() on that thread and wait for it to complete
first.

The problem with this is that the daemonised thread will continue to
run while atexit register callbacks are being called and while Python
is being subsequently destroyed

The result is that the daemonised thread may access application
components that have been cleaned up and in the worst case as Python
progresses with destroying interpreters it could try accessing things
like sys.modules which may no longer exist.

End result is that it could result in a Python exception at some
point, evidenced by messages starting with:

Exception in thread Thread-1 (most likely raised during interpreter
shutdown)

or:

Unhandled exception in thread started by <bound method
Thread.__bootstrap of <Thread(Thread-1, started daemon)>>

In order to avoid this, do people think that as an alternative to
creating daemonised thread that it would be reasonable to create a
derived Thread class for the particular task which overrides
Thread.join() method to set some flag or otherwise take some action
that the thread would notice to tell it to stop, and then call base
class Thread.join().

This way you can flag the thread to shutdown automatically when Python
is going around calling join() on non daemonised threads.

Or am I missing the obvious as far as ways of flagging threads to tell
them to stop?

Note that this is where Python is being used embedded in a C program.
There is no Python main function where code can be put to signal
threads to otherwise stop.

Graham

 

[Gabriel Genellina]

If one creates a thread using threading.Thread and makes it a daemon
by calling setDaemon(), then when Python is exiting it will not
attempt to call join() on that thread and wait for it to complete
first. [...]
End result is that it could result in a Python exception at some
point, evidenced by messages starting with:
Exception in thread Thread-1 (most likely raised during interpreter
shutdown) [...]

In order to avoid this, do people think that as an alternative to
creating daemonised thread that it would be reasonable to create a
derived Thread class for the particular task which overrides
Thread.join() method to set some flag or otherwise take some action
that the thread would notice to tell it to stop, and then call base
class Thread.join().

This way you can flag the thread to shutdown automatically when Python
is going around calling join() on non daemonised threads.

Or am I missing the obvious as far as ways of flagging threads to tell
them to stop?

Note that this is where Python is being used embedded in a C program.
There is no Python main function where code can be put to signal
threads to otherwise stop.

This last statement is important. You need "some" way to tell the
threads to stop working. Usually the thread's run() method is a big
loop; you need to tell it somehow to exit the loop. By example, if you
are using a Queue, put a sentinel object in it; or use an Event
object; or at least a global variable.
Overriding thread.join() as you suggest above may be a good place to
do that, if you don't have another chance.

 

[Graham Dumpleton]

If one creates a thread using threading.Thread and makes it a daemon
by calling setDaemon(), then when Python is exiting it will not
attempt to call join() on that thread and wait for it to complete
first. [...]
End result is that it could result in a Python exception at some
point, evidenced by messages starting with:
Exception in thread Thread-1 (most likely raised during interpreter
shutdown) [...]

In order to avoid this, do people think that as an alternative to
creating daemonised thread that it would be reasonable to create a
derived Thread class for the particular task which overrides
Thread.join() method to set some flag or otherwise take some action
that the thread would notice to tell it to stop, and then call base
class Thread.join().

This way you can flag the thread to shutdown automatically when Python
is going around calling join() on non daemonised threads.

Or am I missing the obvious as far as ways of flagging threads to tell
them to stop?

Note that this is where Python is being used embedded in a C program.
There is no Python main function where code can be put to signal
threads to otherwise stop.

This last statement is important. You need "some" way to tell the
threads to stop working. Usually the thread's run() method is a big
loop; you need to tell it somehow to exit the loop. By example, if you
are using a Queue, put a sentinel object in it; or use an Event
object; or at least a global variable.
Overriding thread.join() as you suggest above may be a good place to
do that, if you don't have another chance.

I already use the Queue method to flag the thread to stop. In practice
the question is more theoretical at the moment as I am able to stop
the thread by registering an atexit callback which stuffs something on
the queue and everything works okay.

The question was me more thinking out loud and wandering what other
peoples opinions were in it.

What makes this stuff even more tricky is that prior to Python 2.5,
the joining with non daemonised threads was done in a callback
registered with the atexit module. This meant it could occur in the
middle of other atexit callbacks being called and thus no absolute
certainty as to ordering.

In Python 2.5, the joining with non daemonised threads was moved to a
distinct phase before atexit callbacks were called. This was good for
non daemonised threads as it gave you more certainty. Still less for
daemonised thread though as although can use atexit callbacks to
trigger a thread to shutdown and then wait on it if necessary, still
doing stuff when some atexit callbacks may have already been called.

Thus why I was hunting for a slightly more predictable method for
daemonised threads, although would be dependent on Python 2.5 or later
being used if you absolutely need it to occur before atexit callbacks
called.

Also, just to make it even more complicated, when using Python in an
embedded manner and using secondary sub interpreters, destroying a
secondary sub interpreter does not result in the atexit callbacks
being called, nor joining with non daemonised threads (at least for <
2.5, can't remember for certain about 2.5 but don't think it does it
either). Thus, if you need these things to occur for secondary sub
interpreters, you must write code yourself to trigger both the thread
joining and atexit callbacks. :-(

Graham