Using a signal to terminate a programm running with an asyncore loop

[david]

I'm developing a program that runs using an asyncore loop. Right now
I can adequately terminate it using Control-C, but as things get
refined I need a better way to stop it. I've developed another
program that executes it as a child process using popen2.Popen4(). I
was attempting to use signals to stop it (using os.kill()) but I keep
running into a problem where sending the signal causes an infinite
loop of printing the message "warning: unhandled exception". This
message appears to be coming from asyncore.py.

Searching online I've found this old post that describes setting up a
signal handler using signal.signal():

http://mail.python.org/pipermail/medusa-dev/2000/000571.html

I've tried it but I don't see any indication that my handler method is
being called. My best guess is that something somewhere else is
overriding my signal handling callbacks and sending the signals
elsewhere until they make their way into asyncore where they cause the
error message.

Any other ideas on how to get this to work? If there's a different
signal I can use that other code won't override, that's fine (I don't
care what the signal is, as long as I can catch it). Or perhaps there
is something different I can do? The program, at present, doesn't
have much in the way of an internal shutdown mechanism.

I'm using Python 2.5.1 on Fedora 8; the program does not need to be
portable to Windows.

- David

(p.s. please post replies on the list; this email address doesn't work
so I won't see any replies sent directly to me)

 

[Larry Bates]

I'm developing a program that runs using an asyncore loop. Right now
I can adequately terminate it using Control-C, but as things get
refined I need a better way to stop it. I've developed another
program that executes it as a child process using popen2.Popen4(). I
was attempting to use signals to stop it (using os.kill()) but I keep
running into a problem where sending the signal causes an infinite
loop of printing the message "warning: unhandled exception". This
message appears to be coming from asyncore.py.

Searching online I've found this old post that describes setting up a
signal handler using signal.signal():

http://mail.python.org/pipermail/medusa-dev/2000/000571.html

I've tried it but I don't see any indication that my handler method is
being called. My best guess is that something somewhere else is
overriding my signal handling callbacks and sending the signals
elsewhere until they make their way into asyncore where they cause the
error message.

Any other ideas on how to get this to work? If there's a different
signal I can use that other code won't override, that's fine (I don't
care what the signal is, as long as I can catch it). Or perhaps there
is something different I can do? The program, at present, doesn't
have much in the way of an internal shutdown mechanism.

I'm using Python 2.5.1 on Fedora 8; the program does not need to be
portable to Windows.

- David

(p.s. please post replies on the list; this email address doesn't work
so I won't see any replies sent directly to me)

Please share a little of the code you used to "catch" the signal. What
signal(s) are you catching? What kill level are you sending (these have to
match up). I've used this quite successfully in some of my programs.

-Larry

 

[david]

Please share a little of the code you used to "catch" the signal. What
signal(s) are you catching? What kill level are you sending (these have to
match up). I've used this quite successfully in some of my programs.

-Larry

Sure. First, here is the signal I'm sending, from the parent process:

os.kill(self.child.pid, signal.SIGTERM)

self.child is the popen2.Popen4 object returned from creating the
child process.

In the child process, one of the first things in the "main" section
(if __name__ == '__main__' is to set up the signal handler:

signal.signal(signal.SIGTERM, handle_shutdown_signal)

handle_shutdown_signal looks a lot like Sam Rushing's example code
from his 2000 post, though I didn't see much of a reason to have the
callback function turn around and call another function, so I combined
them:

SHUTDOWN_PERFORMED = False

def handle_shutdown_signal(signum, frame):
global SHUTDOWN_PERFORMED
print 'in shutdown handler, caught signal', signum #diagnostic msg
if not SHUTDOWN_PERFORMED:
#do some stuff
asyncore.socket_map.clear()
SHUTDOWN_PERFORMED = True
raise asyncore.ExitNow

I also tried replacing the raise asyncore.ExitNow command with a
simple exit() call but that didn't help.

I've also tried using SIGHUP and SIGKILL and they didn't seem to make
a difference.

- David

 

[Larry Bates]

Sure. First, here is the signal I'm sending, from the parent process:

os.kill(self.child.pid, signal.SIGTERM)

self.child is the popen2.Popen4 object returned from creating the
child process.

In the child process, one of the first things in the "main" section
(if __name__ == '__main__' is to set up the signal handler:

signal.signal(signal.SIGTERM, handle_shutdown_signal)

handle_shutdown_signal looks a lot like Sam Rushing's example code
from his 2000 post, though I didn't see much of a reason to have the
callback function turn around and call another function, so I combined
them:

SHUTDOWN_PERFORMED = False

def handle_shutdown_signal(signum, frame):
global SHUTDOWN_PERFORMED
print 'in shutdown handler, caught signal', signum #diagnostic msg
if not SHUTDOWN_PERFORMED:
#do some stuff
asyncore.socket_map.clear()
SHUTDOWN_PERFORMED = True
raise asyncore.ExitNow

I also tried replacing the raise asyncore.ExitNow command with a
simple exit() call but that didn't help.

I've also tried using SIGHUP and SIGKILL and they didn't seem to make
a difference.

- David

When signal is caught handle_shutdown_signal is called. At that point
SHUTDOWN_PERFORMED will ALWAYS be False. Normally all you do in this function
is to set SHUTDOWN_PERFORMED to True and have a test somewhere in your main
program loop that tests if SHUTDOWN_PERFORMED:... I know that when you reach

if not SHUTDOWN_PERFORMED:

SHUTDOWN_PERFORMED will always be False so #do some stuff will get executed.

Setting SHUTDOWN_PERFORMED = True doesn't do any good unless you catch this back
in the main program.

If I were you I would try to issue kill from console first to see if it works.
Then try os.kill().

Hope this helps some.

-Larry

 

[david]

When signal is caught handle_shutdown_signal is called.  At that point
SHUTDOWN_PERFORMED will ALWAYS be False.  Normally all you do in this function
is to set SHUTDOWN_PERFORMED to True and have a test somewhere in your main
program loop that tests if SHUTDOWN_PERFORMED:...  I know that when you reach

if not SHUTDOWN_PERFORMED:

SHUTDOWN_PERFORMED will always be False so #do some stuff will get executed.

Setting SHUTDOWN_PERFORMED = True doesn't do any good unless you catch this back
in the main program.

If I were you I would try to issue kill from console first to see if it works.
Then try os.kill().

Hope this helps some.

-Larry

I won't have a chance to try your suggestions until next week. Though
I'm not convinced that handle_shutdown_signal is even getting called,
since I'm not seeing the diagnostic print statement, neither on the
console nor where stdout/stderr is being redirected. I did try using
the kill command from the console to issue SIGTERM to the process
(e.g. kill -s SIGTERM <pid>) and I still got the infinite series of
"warning: unhandled exception" messages.

- David

 

[david]

When signal is caught handle_shutdown_signal is called. At that point
SHUTDOWN_PERFORMED will ALWAYS be False. Normally all you do in this function
is to set SHUTDOWN_PERFORMED to True and have a test somewhere in your main
program loop that tests if SHUTDOWN_PERFORMED:... I know that when you reach

if not SHUTDOWN_PERFORMED:

SHUTDOWN_PERFORMED will always be False so #do some stuff will get executed.

Setting SHUTDOWN_PERFORMED = True doesn't do any good unless you catch this back
in the main program.

If I were you I would try to issue kill from console first to see if it works.
Then try os.kill().

Hope this helps some.

-Larry

Sorry for the slow response. I had other tasks come up last week that
took priority. Got back to this today and tried it. I simplified my
code so right now the signal handler looks like this:

def handle_shutdown_signal(signum, frame):
print 'in shutdown handler, caught signal', signum
asyncore.socket_map.clear()
exit()

I'm still registering the handler the same way:

signal.signal(signal.SIGTERM, handle_shutdown_signal)

I'm still not convinced the handler function is getting called. I
tried killing the process from the command line (# kill -s SIGTERM
<pid>) and my print message doesn't get shown; and running ps ax shows
the program is still running.

I'm wondering if there is something else going on. I've checked my
code and nothing else registers any signal handlers, but there are
places that use asyncore.dispatcher and asynchat to handle some
network communication; perhaps something in there re-registers the
signal handlers?

I tried writing a simple test program; running it from one shell and
killing it from another works fine:

#!/usr/bin/env python
import signal
def handler(signum, frame):
print 'caught signal', signum
exit()
signal.signal(signal.SIGTERM, handler)
while True:
a = 1
print 'out'

After I issue the fill command, I get the following output:

caught signal 15

Which is what I expected; since the handler calls exit() I didn't
expect to see the 'out' message ever get printed.

Any thoughts?

- David