multiprocessing, pool and process crashes

[dragonk]

I want to use the multiprocessing.Pool object to run multiple tasks in
separate processes.

The problem is that I want to call an external C function (from a
shared library, with help from ctypes) and this function tends to
crash (SIGSEGV,SIGFPE,etc.) on certain data sets (the purpose of this
thing is to test the behavior of the function and see when it
crashes).

My question is if the Pool object is able to spawn a new process in
place of the crashed one and resume operation. I killed with SIGSEGV
the subprocesses and the script locked up

Any ideas/alternatives?

Thanks!

 

[Jesse Noller]

I want to use the multiprocessing.Pool object to run multiple tasks in
separate processes.

The problem is that I want to call an external C function (from a
shared library, with help from ctypes) and this function tends to
crash (SIGSEGV,SIGFPE,etc.) on certain data sets (the purpose of this
thing is to test the behavior of the function and see when it
crashes).

My question is if the Pool object is able to spawn a new process in
place of the crashed one and resume operation.  I killed with SIGSEGV
the subprocesses and the script locked up

Any ideas/alternatives?

You're going to want to use a custom pool, not the built in pool. In
your custom pool, you'll need to capture the signals/errors you want
and handle them accordingly. The built in pool was not designed for
your use case.

-jesse