killing a script

[Russ P.]

I have a Python (2.6.x) script on Linux that loops through many
directories and does processing for each. That processing includes
several "os.system" calls for each directory (some to other Python
scripts, others to bash scripts).

Occasionally something goes wrong, and the top-level script just keeps
running with a stack dump for each case. When I see that, I want to
just kill the whole thing and fix the bug. However, when I hit Control-
C, it apparently just just kills whichever script happens to be
running at that instant, and the top level script just moves to the
next line and keeps running. If I hit Control-C repeatedly, I
eventually get "lucky" and kill the top-level script. Is there a
simple way to ensure that the first Control-C will kill the whole darn
thing, i.e, the top-level script? Thanks.

--Russ P.

 

[MRAB]

I have a Python (2.6.x) script on Linux that loops through many
directories and does processing for each. That processing includes
several "os.system" calls for each directory (some to other Python
scripts, others to bash scripts).

Occasionally something goes wrong, and the top-level script just keeps
running with a stack dump for each case. When I see that, I want to
just kill the whole thing and fix the bug. However, when I hit Control-
C, it apparently just just kills whichever script happens to be
running at that instant, and the top level script just moves to the
next line and keeps running. If I hit Control-C repeatedly, I
eventually get "lucky" and kill the top-level script. Is there a
simple way to ensure that the first Control-C will kill the whole darn
thing, i.e, the top-level script? Thanks.

You could look at the return value of os.system, which may tell you the
exit status of the process.

 

[Russ P.]

On 29/08/2011 02:15, Russ P. wrote:> I have a Python (2.6.x) script on Linux that loops through many


You could look at the return value of os.system, which may tell you the
exit status of the process.

Thanks for the suggestion. Yeah, I guess I could do that, but it seems
that there should be a simpler way to just kill the "whole enchilada."
Hitting Control-C over and over is a bit like whacking moles.

--Russ P.

 

[Chris Angelico]

Thanks for the suggestion. Yeah, I guess I could do that, but it seems
that there should be a simpler way to just kill the "whole enchilada."
Hitting Control-C over and over is a bit like whacking moles.

I believe the idea of this suggestion is for the outer script to
notice that the inner script terminated via Ctrl-C, and would then
immediately choose to terminate itself - thus avoiding the
whack-a-mole effect.

ChrisA

 

[Paul Rubin]

Thanks for the suggestion. Yeah, I guess I could do that, but it seems
that there should be a simpler way to just kill the "whole enchilada."
Hitting Control-C over and over is a bit like whacking moles.

Hit Ctrl-Z, which stops execution of the subprogram but doesn't kill it.
Then kill both the subprogram and the control program from a terminal
window.

 

[Russ P.]

I believe the idea of this suggestion is for the outer script to
notice that the inner script terminated via Ctrl-C, and would then
immediately choose to terminate itself - thus avoiding the
whack-a-mole effect.

ChrisA

Yes, but if I am not mistaken, that will require me to put a line or
two after each os.system call. That's almost like whack-a-mole at the
code level rather than the Control-C level. OK, not a huge deal for
one script, but I was hoping for something simpler. I was hoping I
could put one line at the top of the script and be done with it.

--Russ P.

 

[Chris Rebert]

Yes, but if I am not mistaken, that will require me to put a line or
two after each os.system call.

Er, just write a wrapper for os.system(), e.g.:

def mysystem(cmd):
if os.system(cmd):
sys.exit()

Also, you may want to switch to using the `subprocess` module instead.

Cheers,
Chris

 

[Russ P.]

Er, just write a wrapper for os.system(), e.g.:

def mysystem(cmd):
    if os.system(cmd):
        sys.exit()

Also, you may want to switch to using the `subprocess` module instead.

Cheers,
Chris

Sounds like a good idea. I'll give it a try. Thanks.

--Russ P.

 

[Nobody]

Is there a
simple way to ensure that the first Control-C will kill the whole darn
thing, i.e, the top-level script? Thanks.

You might try using subprocess.Popen() or subprocess.call() rather than
os.system().

os.system() calls the platform's system() function. On Unix, this
specifically ignores SIGINT and SIGQUIT for the duration of the call,
ensuring that Ctrl-C and Ctrl-\ only affect the child process and not the
parent.

subprocess.Popen() doesn't perform any implicit signal handling; it's
implemented in Python in terms of os.fork() and os.execvp[e](). It also
has a better interface (i.e. you get to directly control the argument list
passed to the child process, rather than having to construct a shell
command and hope that you got the quoting/escaping correct).

This may not suffice if any of the descendent processes are moved into
their own process group, as signals generated by the tty driver are sent
only to the foreground process group. However, this is unlikely to be an
issue for simple non-interactive programs (e.g. standard Unix "commands").

 

[Thomas Jollans]

Yes, but if I am not mistaken, that will require me to put a line or
two after each os.system call. That's almost like whack-a-mole at the
code level rather than the Control-C level. OK, not a huge deal for
one script, but I was hoping for something simpler. I was hoping I
could put one line at the top of the script and be done with it.

It's perfectly normal error-handling procedure. In Python, errors are
usually handled by exceptions, but if you embed a system that doesn't
support exceptions, e.g. external processes or a C library via ctypes,
you will of course have to write a little more code in order to handle
errors correctly.

T

 

[Russ P.]

Er, just write a wrapper for os.system(), e.g.:

def mysystem(cmd):
    if os.system(cmd):
        sys.exit()

Also, you may want to switch to using the `subprocess` module instead.

Cheers,
Chris

I ended up with this:

def systemx(cmd):

if system(cmd): exit("\nERROR: " + cmd + " failed\n")

This is good enough for my purposes in this case. Thanks for all the
suggestions.

--Russ P.

 

[Arnaud Delobelle]

Yes, but if I am not mistaken, that will require me to put a line or
two after each os.system call. That's almost like whack-a-mole at the
code level rather than the Control-C level. OK, not a huge deal for
one script, but I was hoping for something simpler. I was hoping I
could put one line at the top of the script and be done with it.

Write a function! That's what they're for after all

 

[Steven D'Aprano]

On Tue, 30 Aug 2011 08:53 am Arnaud Delobelle wrote:

[...]

Write a function! That's what they're for after all

I'm not sure that this is actually as simple as that, especially using
os.system.

As I understand it, the scenario is this:

The main script looks something like this:

for x in whatever:
os.system('something.py x')

Each time through the loop, a new Python process is started. Each process
runs in the foreground, capturing standard input, and so hitting Ctrl-C
kills *that* process, not the main script. Unless, by chance, the Ctrl-C
happens after the system call returns, but before the next one starts, it
is completely invisible to the parent process (the main script). Wrapping
os.system in a function does nothing to fix that.

Possibly using the subprocess module may help.

Otherwise, the only way around this I can think of is to ensure that
the 'something.py' script (or scripts!) each return an error code for "User
Cancelled":

for x in whatever:
result = os.system('something.py x')
if result == 3: # User Cancelled
break

But if the 'something.py' scripts are arbitrary scripts, I don't think you
have any easy way around it. Perhaps use threads, and have the main script
ask the thread to kill the child process?

Regardless, this is a hard problem, and it isn't possible to just have some
magic switch at the top of your script to make it work. You actually have
to do the work yourself.

(But of course you can do the work inside a function, and re-use it
elsewhere.)

 

[Hans Mulder]

On Tue, 30 Aug 2011 08:53 am Arnaud Delobelle wrote:

[...]

Write a function! That's what they're for after all

I'm not sure that this is actually as simple as that, especially using
os.system.

As I understand it, the scenario is this:

The main script looks something like this:

for x in whatever:
os.system('something.py x')

Each time through the loop, a new Python process is started. Each process
runs in the foreground, capturing standard input, and so hitting Ctrl-C
kills *that* process, not the main script. Unless, by chance, the Ctrl-C
happens after the system call returns, but before the next one starts, it
is completely invisible to the parent process (the main script). Wrapping
os.system in a function does nothing to fix that.

Possibly using the subprocess module may help.

Using the subprocess module is likely to help, because unlike os.system,
subprocess does not set control-C to 'ignore' in the parent process

Otherwise, the only way around this I can think of is to ensure that
the 'something.py' script (or scripts!) each return an error code for "User
Cancelled":

for x in whatever:
result = os.system('something.py x')
if result == 3: # User Cancelled
break

On my system, os.system returns 2 if the subprocess was killed by a
control-C. The portable way is to use the constant SIGINT from the
signal module.

Not that os.system does not return 2 if the child ended by doing
sys.exit(2); in that case, os.system in the parent returns 2<<8.

But if the 'something.py' scripts are arbitrary scripts, I don't think you
have any easy way around it.

If the arbitrary script does not trap SIGINT, then os.system
will return the number of the signal that kill the script, i.e.
signal.SIGINT.

If the script traps SIGINT and does some cleanup and then
exits, the best you can hope for, is that in the error case,
the script exits with a special value. And you have to
remember the <<8 issue.

Perhaps use threads, and have the main script
ask the thread to kill the child process?

How would that help?

The problem is that the parent process ignores control-C while
os.system is running (by design). If the parent uses threads,
it still won't receive the signal, because signal state is
global.

Regardless, this is a hard problem, and it isn't possible to just have some
magic switch at the top of your script to make it work. You actually have
to do the work yourself.

You could monkey-patch the os module and replace os.system by
(untested):

def sub_system(command):
pr = subprocess.Popen(command, shell=True)
status = os.waitpid(pr.pid, 0)[1]
return status

That should have the same effect as os.system, except it does
not ignore control-C.

(But of course you can do the work inside a function, and re-use it
elsewhere.)

Even if you don't re-use it, you should still do it in a
function, if only for readability.

Hope this helps,

-- HansM

 

[Cameron Simpson]

| On Tue, 30 Aug 2011 08:53 am Arnaud Delobelle wrote:
| >> Yes, but if I am not mistaken, that will require me to put a line or
| >> two after each os.system call. That's almost like whack-a-mole at the
| >> code level rather than the Control-C level. OK, not a huge deal for
| >> one script, but I was hoping for something simpler. I was hoping I
| >> could put one line at the top of the script and be done with it.
| >
| > Write a function! That's what they're for after all
|
| I'm not sure that this is actually as simple as that, especially using
| os.system.
|
| As I understand it, the scenario is this:
|
| The main script looks something like this:
|
| for x in whatever:
| os.system('something.py x')
|
| Each time through the loop, a new Python process is started. Each process
| runs in the foreground, capturing standard input, and so hitting Ctrl-C
| kills *that* process, not the main script. Unless, by chance, the Ctrl-C
| happens after the system call returns, but before the next one starts, it
| is completely invisible to the parent process (the main script). Wrapping
| os.system in a function does nothing to fix that.

Presuming you're talking about UNIX, this is not correct.

Ctrl-C at the terminal delivers SIGINT to _every_ process in the controlling
process group for the terminal. It also has _nothing_ to do with the standard
input.

When you run a script, yea even a Python script, thus:

myscript ...

then job control capable shells (all of them, these days) put the python
process running "myscript" in its own process group as the leader
(being, initially, the only process in the group). If myscript forks
other processes, as happens in os.system(), they are _also_ in that
process group. _ALL_ of them receive the SIGINT from your Ctrl-C.

Cheers,

 

[Steven D'Aprano]

On 30Aug2011 14:13, Steven D'Aprano <[email protected]>
wrote:
| On Tue, 30 Aug 2011 08:53 am Arnaud Delobelle wrote:
| >> Yes, but if I am not mistaken, that will require me to put a line or
| >> two after each os.system call. That's almost like whack-a-mole at the
| >> code level rather than the Control-C level. OK, not a huge deal for
| >> one script, but I was hoping for something simpler. I was hoping I
| >> could put one line at the top of the script and be done with it.
| >
| > Write a function! That's what they're for after all
|
| I'm not sure that this is actually as simple as that, especially using
| os.system.
|
| As I understand it, the scenario is this:
|
| The main script looks something like this:
|
| for x in whatever:
| os.system('something.py x')
|
| Each time through the loop, a new Python process is started. Each
| process runs in the foreground, capturing standard input, and so hitting
| Ctrl-C kills *that* process, not the main script. Unless, by chance, the
| Ctrl-C happens after the system call returns, but before the next one
| starts, it is completely invisible to the parent process (the main
| script). Wrapping os.system in a function does nothing to fix that.

Presuming you're talking about UNIX, this is not correct.

Ctrl-C at the terminal delivers SIGINT to _every_ process in the
controlling process group for the terminal. It also has _nothing_ to do
with the standard input.

There may be something to what you say, but the behaviour experienced by the
Original Poster still needs explaining. See below.

When you run a script, yea even a Python script, thus:

myscript ...

then job control capable shells (all of them, these days) put the python
process running "myscript" in its own process group as the leader
(being, initially, the only process in the group). If myscript forks
other processes, as happens in os.system(), they are _also_ in that
process group. _ALL_ of them receive the SIGINT from your Ctrl-C.

I can replicate to OP's problem with these two simple Python scripts:

[steve@sylar ~]$ cat script.py
#!/usr/bin/python
print "inside script.py"
print "type Ctrl-C to exit"
while True:
pass

[steve@sylar ~]$ cat test.py
import os
print "calling script.py with os.system"
for i in range(3):
os.system('./script.py')


And now run them:

[steve@sylar ~]$ python test.py
calling script.py with os.system
inside script.py
type Ctrl-C to exit
Traceback (most recent call last):
File "./script.py", line 4, in <module>
while True:
KeyboardInterrupt
inside script.py
type Ctrl-C to exit
Traceback (most recent call last):
File "./script.py", line 5, in <module>
pass
KeyboardInterrupt
inside script.py
type Ctrl-C to exit
Traceback (most recent call last):
File "./script.py", line 4, in <module>
while True:
KeyboardInterrupt


Sure enough, I now have to hit Ctrl-C repeatedly, once per invocation of
script.py. While script.py is running, it receives the Ctrl-C, the calling
process does not.

 

[Hans Mulder]

Sure enough, I now have to hit Ctrl-C repeatedly, once per invocation of
script.py. While script.py is running, it receives the Ctrl-C, the calling
process does not.

You misinterpret what you are seeing: the calling process *does* receive
the ctrl-C, it just chooses to ignore it.

This is documented behaviour of os.system. It you don't want this, then
use the subprocess module, which does not behave this way.

-- HansM

 

[Steven D'Aprano]

You misinterpret what you are seeing: the calling process *does* receive
the ctrl-C, it just chooses to ignore it.

This is documented behaviour of os.system.

Documented where? Neither the on-line documentation nor the function
docstring mentions anything about it that I can see:

http://docs.python.org/py3k/library/os.html#os.system
Help on built-in function system in module posix:

system(...)
system(command) -> exit_status

Execute the command (a string) in a subshell.

 

[Cameron Simpson]

| Hans Mulder wrote:
| > On 9/09/11 11:07:24, Steven D'Aprano wrote:
| >> Sure enough, I now have to hit Ctrl-C repeatedly, once per invocation of
| >> script.py. While script.py is running, it receives the Ctrl-C, the
| >> calling process does not.
| >
| > You misinterpret what you are seeing: the calling process *does* receive
| > the ctrl-C, it just chooses to ignore it.
| >
| > This is documented behaviour of os.system.
|
| Documented where? Neither the on-line documentation nor the function
| docstring mentions anything about it that I can see:
|
| http://docs.python.org/py3k/library/os.html#os.system

My copy of the 2.7 docs says:

This is implemented by calling the Standard C function system(), and
has the same limitations.

and sure enough, "man 3 system" says:

The system() function hands the argument command to the command
interpreter sh(1). The calling process waits for the shell to finish
executing the command, ignoring SIGINT and SIGQUIT, and blocking
SIGCHLD.

os.system() is very convenient for simple stuff, but one size does not
fit all. Continuing with the Python docs for os.system:

On Unix, the return value is the exit status of the process encoded in
the format specified for wait().

and it is easy to inspect that value for "the subprocess died from a
signal". Not inspecting the exit status correctly will always be an
opportunity for incorrect app behaviour.

Cheers,
--
Cameron Simpson <[email protected]> DoD#743
http://www.cskk.ezoshosting.com/cs/

Mac OS X. Because making Unix user-friendly is easier than debugging Windows.
- Mike Dawson, Macintosh Systems Administrator and Consultation.
(e-mail address removed) http://herowars.onestop.net

 

[Steven D'Aprano]

On 09Sep2011 22:16, Steven D'Aprano <[email protected]>
wrote:
| Hans Mulder wrote:
| > On 9/09/11 11:07:24, Steven D'Aprano wrote:
| >> Sure enough, I now have to hit Ctrl-C repeatedly, once per invocation
| >> of script.py. While script.py is running, it receives the Ctrl-C, the
| >> calling process does not.
| >
| > You misinterpret what you are seeing: the calling process *does*
| > receive the ctrl-C, it just chooses to ignore it.
| >
| > This is documented behaviour of os.system.
|
| Documented where? Neither the on-line documentation nor the function
| docstring mentions anything about it that I can see:
|
| http://docs.python.org/py3k/library/os.html#os.system

My copy of the 2.7 docs says:

This is implemented by calling the Standard C function system(), and
has the same limitations.

and sure enough, "man 3 system" says:

I don't consider having to look up documentation for a function in a
completely different language (in this case, C) as "documented behaviour of
os.system".

Does the C standard define the behaviour of system(), or is that
implementation dependent? It sounds to me that the Python developers are
implicitly refusing responsibility for the detailed behaviour of os.system
by noting that it depends on the C function. What do Jython, PyPy and
IronPython do?

Perhaps the docs for os.system should explicitly note that the behaviour is
implementation dependent, rather than just hint at it. Either that or
explicitly state what os.system does.

os.system() is very convenient for simple stuff, but one size does not
fit all.

I never said it does. Back in my first comment on this thread, I said

"Possibly using the subprocess module may help."

Continuing with the Python docs for os.system:

On Unix, the return value is the exit status of the process encoded in
the format specified for wait().

and it is easy to inspect that value for "the subprocess died from a
signal". Not inspecting the exit status correctly will always be an
opportunity for incorrect app behaviour.

Except that the subprocess can catch the KeyboardInterrupt before exiting,
and there's no guarantee that it will return an appropriate error code.

You're right though, os.system is good for simple stuff and not much more.