KeyboardInterrupt vs extension written in C

[Tamas Nepusz]

Hi everyone,

I have tried to do some googling before asking my question here, but I
haven't found any suitable answer. I am developing a Python API for a
graph library written in pure C. The library is doing an awful lot of
math computations, and some of them can take a pretty long time
(depending on the size of the input). If I invoke such calculation from
Python using my own extension, the interpreter locks until the
calculation is complete (I mean, it is not responding to Ctrl-C or
SIGINT).

My question is: is there any way to allow the user to use Ctrl-C to
generate a KeyboardInterrupt and cancel such large computations? I'm
assuming that although pressing Ctrl-C may generate a KeyboardInterrupt
in Python, it is not propagated to the user level until the call into
the graph library returns. I tried to add Py_BEGIN_ALLOW_THREADS before
the call to the underlying C library and Py_END_ALLOW_THREADS after
returning from it, but I had no luck.

Thanks in advance for your answers,

 

[Diez B. Roggisch]

Hi everyone,

I have tried to do some googling before asking my question here, but I
haven't found any suitable answer. I am developing a Python API for a
graph library written in pure C. The library is doing an awful lot of
math computations, and some of them can take a pretty long time
(depending on the size of the input). If I invoke such calculation from
Python using my own extension, the interpreter locks until the
calculation is complete (I mean, it is not responding to Ctrl-C or
SIGINT).

My question is: is there any way to allow the user to use Ctrl-C to
generate a KeyboardInterrupt and cancel such large computations? I'm
assuming that although pressing Ctrl-C may generate a KeyboardInterrupt
in Python, it is not propagated to the user level until the call into
the graph library returns. I tried to add Py_BEGIN_ALLOW_THREADS before
the call to the underlying C library and Py_END_ALLOW_THREADS after
returning from it, but I had no luck.

I'm not especially an expert with C-extensions. So I'm not sure if I can
be of any help here. But if what you really are after is "only"
terminating the whole program (instead of just the computation, and then
continue), you might consider threads. Do the calls in a separate
thread, and make that a daemon-thread. Then things _should_ work as
expected, as the main thread will receive the signal. Hopefully... if
your call releases the GIL. Which Py_*_ALLOW_THREADS seems to do. So -
give it a try

Diez

 

[Tamas Nepusz]

No, that's actually a bit more complicated. The library I'm working on
is designed for performing calculations on large-scale graphs (~10000
nodes and edges). I want to create a Python interface for that library,
so what I want to accomplish is that I could just type "from igraph
import *" in a Python command line and then access all of the
functionalities of the igraph library. Now it works, except the fact
that if, for example, I start computing the diameter of a random graph
of ~100000 nodes and ~200000 edges, I can't cancel it, because the
KeyboardInterrupt is not propagated to the Python toplevel (or it isn't
even generated until the igraph library routine returns). I would like
to allow the user to cancel the computation with the usual Ctrl-C and
return to the Python interactive interface.
This whole feature is not vital, but it would mean a big step towards
user-friendliness.
I have access to the source code of igraph as well (since one of my
colleagues is developing it), so another possible solution would be to
inject some calls into the igraph source code which temporarily cancels
the computation and checks whether there's something waiting in the
Python interpreter, but I haven't found any function in the API which
allows me to do this.

Tamas

 

[Donn Cave]

Quoth "Tamas Nepusz" <[email protected]>:
| No, that's actually a bit more complicated. The library I'm working on
| is designed for performing calculations on large-scale graphs (~10000
| nodes and edges). I want to create a Python interface for that library,
| so what I want to accomplish is that I could just type "from igraph
| import *" in a Python command line and then access all of the
| functionalities of the igraph library. Now it works, except the fact
| that if, for example, I start computing the diameter of a random graph
| of ~100000 nodes and ~200000 edges, I can't cancel it, because the
| KeyboardInterrupt is not propagated to the Python toplevel (or it isn't
| even generated until the igraph library routine returns). I would like
| to allow the user to cancel the computation with the usual Ctrl-C and
| return to the Python interactive interface.
| This whole feature is not vital, but it would mean a big step towards
| user-friendliness.
| I have access to the source code of igraph as well (since one of my
| colleagues is developing it), so another possible solution would be to
| inject some calls into the igraph source code which temporarily cancels
| the computation and checks whether there's something waiting in the
| Python interpreter, but I haven't found any function in the API which
| allows me to do this.

Hm, tough question. Even in C, this would be a little awkward.
If you only wanted the program to abort on ^C, then you could just
replace the default sigint handler with SIG_DFL. But then you'd be
back to the shell prompt, not the python prompt. If you want to
catch the signal, but also abort a computation, then as you say,
the computation needs to check periodically.

Rather than peek into Python for this, I'm wondering if your module
could set its own signal handler for SIGINT, which would set a library
flag. Then call PyErr_SetInterrupt(), to emulate the normal signal
handler.

Donn Cave, (e-mail address removed)

 

[Dieter Maurer]

The library I'm working on
is designed for performing calculations on large-scale graphs (~10000
nodes and edges). I want to create a Python interface for that library,
so what I want to accomplish is that I could just type "from igraph
import *" in a Python command line and then access all of the
functionalities of the igraph library. Now it works, except the fact
that if, for example, I start computing the diameter of a random graph
of ~100000 nodes and ~200000 edges, I can't cancel it, because the
KeyboardInterrupt is not propagated to the Python toplevel (or it isn't
even generated until the igraph library routine returns).

Python installs a "SIGINT" handler that just notes that
such a signal was received. The note is handled during
bytecode execution. This way, Python handles the (dangerous)
asynchronous signal synchronously (which is much safer).
But, it also means that a signal during execution of your C extension
is only handled when it finished.

What you can do in your wrapper code:

Temporarily install a new handler for "SIGINT" that
uses "longjmp" to quit the C extension execution when
the signal occurs.

Note that "longjmp" is dangerous. Great care is necessary.

It is likely that SIGINT occurrences will lead to big
resource leaks (because your C extension will have no
way to release resources when it gets quit with "longjmp").


Dieter