Killing threads

[ericwoodworth]

Hi,
I'm new to python and even newer to threading and it seems as
though I'm missing something fundamental about threads. Basically I
have a program that looks like this:

class ThreadOne(threading.Thread):
while 1:
do stuff

class ThreadTwo(threading.Thread):
while 1:
do other stuff


first = ThreadOne()
second = ThreadTwo()

while 1:
do stuff


The issue that I'm having is...I don't know how to kill this app in
window. I hit ctrl-c but that only seems to kill one of the threads.
The rest of the app just lingers. There's got to be a more graceful
way but so far I haven't googled anything up.

I'm using queues to talk between these threads so I could certainly
put some kind of message on the queue that causes the threads to
commit suicide but I'm thinking there's a more built in way to do what
I want. I'm just not sure what it is.

I'd appreciate it if somebody could point me in the right direction.
Thanks.

 

[Aahz]

I'm using queues to talk between these threads so I could certainly
put some kind of message on the queue that causes the threads to
commit suicide but I'm thinking there's a more built in way to do what
I want. I'm just not sure what it is.

There isn't, you have the right idea about using queues.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan

 

[ericwoodworth]

There isn't, you have the right idea about using queues.
--
Aahz ([email protected])           <*>        http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it."  --Brian W. Kernighan

Ok good to know. I was letting the search for a really cool solution
stop me from rolling out what I think I already know how to do.
Thanks.

 

[imageguy]

Hi,
     I'm new to python and even newer to threading and it seems as
though I'm missing something fundamental about threads.  Basically I
have a program that looks like this:

class ThreadOne(threading.Thread):
     while 1:
          do stuff

class ThreadTwo(threading.Thread):
     while 1:
          do other stuff

first = ThreadOne()
second = ThreadTwo()

while 1:
    do stuff

The issue that I'm having is...I don't know how to kill this app in
window.  I hit ctrl-c but that only seems to kill one of the threads.
The rest of the app just lingers.  There's got to be a more graceful
way but so far I haven't googled anything up.

I'm using queues to talk between these threads so I could certainly
put some kind of message on the queue that causes the threads to
commit suicide but I'm thinking there's a more built in way to do what
I want.  I'm just not sure what it is.

I'd appreciate it if somebody could point me in the right direction.
Thanks.

I am not an expert either, however, I think the standard practice is
to use and threading.Event to communicate with the threads.

So the structure goes something like this;
In the main thread set a threading.Event called 'keepgoing'.
Pass this object into the thread at instantiation or as a parameter in
the .start method
Over-ride the .run method to handle the process
Change the while loop's condition to check the .keepgoing.isSet()
In the primary thread, when you want the threads to stop, simply .clear
() the keepgoing event
The while loop terminates and at the end of the .run method, the
thread will terminate.

In the primary program, you should then .join() the worker threads.
- doing this will block the program until all of the thread have
stopped.

So your example would become;

class ThreadOne(threading.Thread):

def __init__(self, keepgoing):
threading.Thread.__init__(self)
self.keepgoing = keepgoing

def run(self)
while self.keepgoing.isSet():
do stuff thread one

class ThreadTwo(threading.Thread):

def __init__(self, keepgoing):
threading.Thread.__init__(self)
self.keepgoing = keepgoing

def run(self)
while self.keepgoing.isSet():
do stuff in thread two
#
#now start the main process
#

keepgoing = threading.Event()
keepgoing.set()

first = ThreadOne(keepgoing)
second = ThreadTwo(keepgoing)

# you could also pass in the variable with a thread.start() method,
which will then call the thread.run()

while 1:
do stuff
if stop_condition: #keyboard interrupt ?
keepgoing.clear()

for worker in threading.enumerate():
if worker.isAlive():
worker.join()

FULL DISCLAIMER: I am NOT a threading expert and have limited
experience, so if this doesn't work, you are on your own !

 

[Aahz]

I am not an expert either, however, I think the standard practice is
to use and threading.Event to communicate with the threads.

Nope, Eric had the right idea the first time: use Queue.Queue. And the
reason is precisely because you're not a thread expert; using Queue is
the easiest way to avoid deadlocks and other threading problems.

For more info, see the slides from my thread tutorial:
http://pythoncraft.com/OSCON2001/
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan

 

[Dennis Lee Bieber]

If yor threads are not set as 'deamons' using Thread.setDaemon method,
then your main program at its termination should call Thread.join for
each of the thread spawned, otherwise the whole process will not quit.

.join() alone won't do anything but wait for the thread itself to
quit -- which means one still has to signal the threads to commit
suicide.

1) Use of a global scope "SHUTDOWN = true" which each thread tests at
some point...

2) Use of an instance SHUTDOWN, if subclassing from threading, to
permit targeted suicide requests to specific threads, again the thread
must test at some point...

3) Send some special marker via a QUEUE (which may be shared by
multiple worker threads -- so they must PUT the marker back on the queue
before committing hari kari so the next worker discovers it; or thread
specific queues for targeted shutdowns). Threads must be in a state to
read from the queue.

Main program, after sending the signal (1, 2, or 3, above) may then
wait on .join() operations for the particular threads.

If the thread has the capability to become blocked on some operation
(say a socket read without timeout), none of these solutions will work.
That just leaves setting the threads daemonic at the start -- which
indicates the runtime may brutally kill them when the main program
exits.


--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[imageguy]

For more info, see the slides from my thread tutorial:http://pythoncraft.com/OSCON2001/

Aahz, thanks for this reference and link to your presentation. At the
risk of highjacking the OP's question, I am bit confused as to how
using an Event would cause a deadlock. You presentation outlines
Events, but doesn't elaborate on the specifics or usage.

In threading.Event python 2.5 docs say;
"This is one of the simplest mechanisms for communication between
threads: one thread signals an event and other threads wait for it. "

Again, I have limited experience, however, in my reading of the
threading manual and review examples, Events were specifically design
to be a thread safe way to communicate a 'state' to running threads ?
In the OP's example 'do stuff' was open to wide interpretation,
however, if within the thread's main 'while' loop the tread checks to
see if the 'keepgoing' Event.isSet(), in what scenario would this
create deadlock ?

Thanks for your patience. I thought I had grasped the basics of using
threads/threading, so hope I can learn more.

g.

 

[Dennis Lee Bieber]

In threading.Event python 2.5 docs say;
"This is one of the simplest mechanisms for communication between
threads: one thread signals an event and other threads wait for it. "

Again, I have limited experience, however, in my reading of the
threading manual and review examples, Events were specifically design
to be a thread safe way to communicate a 'state' to running threads ?
In the OP's example 'do stuff' was open to wide interpretation,
however, if within the thread's main 'while' loop the tread checks to
see if the 'keepgoing' Event.isSet(), in what scenario would this
create deadlock ?

If you are going to perform a CPU intensive polling loop, there is
no sense in using the Event system in the first place... Just create a
globally accessible flag and set it to true when you want to signal the
threads (or false if you don't want to use the negation "while not
flagged: do next processing step")

Event is optimized for the case wherein threads can WAIT (block) on
the Event object.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[ericwoodworth]

        If you are going to perform a CPU intensive polling loop, there is
no sense in using the Event system in the first place... Just create a
globally accessible flag and set it to true when you want to signal the
threads (or false if you don't want to use the negation "while not
flagged: do next processing step")

        Event is optimized for the case wherein threads can WAIT (block) on
the Event object.
--
        Wulfraed        Dennis Lee Bieber               KD6MOG
        (e-mail address removed)             (e-mail address removed)
                HTTP://wlfraed.home.netcom.com/
        (Bestiaria Support Staff:               (e-mail address removed))
                HTTP://www.bestiaria.com/

Well it turns out my problem was with queues not with threads. I had
a self.die prop in my thread object that defaults to FALSE and that I
set to true when i wanted the thread to die. then my loop would be
while not die: It seemed pretty simple so I didn't know why it was
failing. What I didn't know, because I'm quite new to python, is that
queue.get was blocking. So my producer thread why dying immediately
but my worker threads were all blocking on their queue.gets. So they
were never falling off the loop. I changed it to queue.get_nowait()
and added a queue.empty exception and everything worked as expected.

So I thought I knew what was going on and that I was having a really
esoteric problem when i was actually having a pretty boring problem I
didn't recognize.

Thanks everybody for the help!

 

[bieffe62]

        .join() alone won't do anything but wait for the thread itself to
quit -- which means one still has to signal the threads to commit
suicide.

Yes. Mine was an 'additional suggestion' to the ones that the OP
already received.
I guests that was not clear enough ...

... follows a nice explanation on methods to stop threads that I was
too lazy to write ...

        If the thread has the capability to become blocked on some operation
(say a socket read without timeout), none of these solutions will work.
That just leaves setting the threads daemonic at the start -- which
indicates the runtime may brutally kill them when the main program
exits.

You know, this bugger me a little. I know that killing threads is hard
in any language
(I'm facing now the issue in a C++ program I'm writing at work),
expecially doing in
a platform-independent way, but Java managed to do it. Now python is
in many ways an
higher level language than Java, but when it comes to threading I feel
it lacks something.
I know that often it is not too hard to avoid blocking reads, and you
can always use subprocesses that
with the new multiprocessing module are almost as easy as threads, but
still ...

Ciao

 

[bieffe62]

Well it turns out my problem was with queues not with threads.  I had
a self.die prop in my thread object that defaults to FALSE and that I
set to true when i wanted the thread to die.  then my loop would be
while not die:  It seemed pretty simple so I didn't know why it was
failing.  What I didn't know, because I'm quite new to python, is that
queue.get was blocking.  So my producer thread why dying immediately
but my worker threads were all blocking on their queue.gets.  So they
were never falling off the loop.  I changed it to queue.get_nowait()
and added a queue.empty exception and everything worked as expected.

So I thought I knew what was going on and that I was having a really
esoteric problem when i was actually having a pretty boring problem I
didn't recognize.

Thanks everybody for the help!>

I've gone through that also, when I started with python threads
Be aware that using get_nowait may lead to your thread using too much
CPU in checking a queue often empty. I tend to use Queue.get with a
timeout, smaller enough to keep the thread responsive but large enough
not
to waste CPU in too-frequent checks.

Ciao

 

[ericwoodworth]

I've gone through that also, when I started with python threads
Be aware that using get_nowait may lead to your thread using too much
CPU in checking a queue often empty. I tend to use  Queue.get with a
timeout, smaller enough to keep the thread responsive but large enough
not
to waste CPU in too-frequent checks.

Ciao

Ok thanks - good to now. I'm trying to throttle it with a 1/2 sec
sleep statement in the loop but I might just have the "main" loop toss
some stuff on that queue as another solution. I'm still kicking it
around

 

[Aahz]

I know that killing threads is hard in any language (I'm facing now
the issue in a C++ program I'm writing at work), expecially doing in a
platform-independent way, but Java managed to do it.

That's not my understanding:

http://www.roseindia.net/javatutorials/shutting_down_threads_cleanly.shtml
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"...string iteration isn't about treating strings as sequences of strings,
it's about treating strings as sequences of characters. The fact that
characters are also strings is the reason we have problems, but characters
are strings for other good reasons." --Aahz

 

[Aahz]

Aahz:

Aahz, thanks for this reference and link to your presentation. At the
risk of highjacking the OP's question, I am bit confused as to how
using an Event would cause a deadlock. You presentation outlines
Events, but doesn't elaborate on the specifics or usage.

In threading.Event python 2.5 docs say;
"This is one of the simplest mechanisms for communication between
threads: one thread signals an event and other threads wait for it. "

Again, I have limited experience, however, in my reading of the
threading manual and review examples, Events were specifically design
to be a thread safe way to communicate a 'state' to running threads ?
In the OP's example 'do stuff' was open to wide interpretation,
however, if within the thread's main 'while' loop the tread checks to
see if the 'keepgoing' Event.isSet(), in what scenario would this
create deadlock ?

Consider two threads trying to pass information to each other. You need
two Event objects; unless you are very careful, they can both block on
waiting for each other. It's easier if Queue is the only construct you
use, it can do everything that the lower-level constructs can do, without
needing to figure out the little gotchas -- and you'll need a Queue
eventually, so just learn that one. It's all about simplifying.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"...string iteration isn't about treating strings as sequences of strings,
it's about treating strings as sequences of characters. The fact that
characters are also strings is the reason we have problems, but characters
are strings for other good reasons." --Aahz