extension module, thread safety?

[Torsten Mohr]

Hi,

i write an extension module in C at the moment. This
module does some work on some own data types that
consist of some values.
The functions that can change the data are written in C.

The question came up if this is by itself thread safe,
if some two or more threads try to change these data types,
are the C functions by themselves are "atomic" or can they
be interrupted be the perl interpreter and then (data types
are in some inconsistent half-changed state) another function
that works on these data is called?


Thanks for hints,
Torsten.

 

[David Bolen]

The question came up if this is by itself thread safe,
if some two or more threads try to change these data types,
are the C functions by themselves are "atomic" or can they
be interrupted be the perl interpreter and then (data types
are in some inconsistent half-changed state) another function
that works on these data is called?

I presume you mean "Python" and not "perl"...

If the threads under discussion are all Python threads, then by
default yes, the extension module C functions will appear to be atomic
from the perspective of the Python code. When the Python code calls
into the extension module, the GIL (global interpreter lock) is still
being held. Unless the extension module code explicitly releases the
GIL, no other Python threads can execute (even though those threads
are in fact implemented as native platform threads).

So in general, if you write an extension module where none of its
functions ever release the GIL, there's no way for two of its
functions to be run from different Python threads simultaneously.

Note that this restriction won't necessarily hold if there are other
ways (at the C level, or from other extension modules) to trigger code
in the extension module, since that's outside of the control of the
Python GIL. Nor will it necessarily hold true if your extension
module calls back out into Python (as a callback, or whatever) since
once the interpreter is back in Python code the interpreter itself
will periodically release the GIL, or some other extension code that
the callback code runs may release it.

To the extent possible, it's considered good practice to release the
GIL in an extension module whenever you are doing lengthy processing
so as to permit other Python threads (that may have nothing to do with
using your extension module) to execute. For short routines this
really isn't an issue, but if your extension module will be spending
some time managing its data, you may wish to add some internal thread
protection around that data, so that you can use your own locks rather
than depending on the GIL.

-- David

 

[Pierre Barbier de Reuille]

David Bolen a écrit :

If the threads under discussion are all Python threads, then by
default yes, the extension module C functions will appear to be atomic
from the perspective of the Python code. When the Python code calls
into the extension module, the GIL (global interpreter lock) is still
being held. Unless the extension module code explicitly releases the
GIL, no other Python threads can execute (even though those threads
are in fact implemented as native platform threads).

Indeed, there is this (so annoying) GIL ... is there any project about
extracting the (few ?) critical points of the python interpreter to put
locks around them instead of choosing the opposite strategy (ie. locking
anythime but when we know the section is not critical) ? Because it
would made embedding a python interpreter in another language much more
easy !

With the current CPython, it's very hard to mix Python and C in a
multithreading application (with C-threads, not Python-threads). In fact
I never really succeeded in that task because of that GIL ! I have a
multi-thread application but every bit of Python code must be run into a
Python thread. To be more precise, I wanted to be able to call Python
code in response to some GUI events, and I didn't want to instanciate a
new interpreter for I wanted to be able to access the environment of my
main Python interpreter.

By the way, if someone succeeded in a similar task, I'll be happy to
hear about it

Pierre

PS: if the main code is in C (in fact C++ ...) and not Python it's for
historical reasons of course

 

[Nick Coghlan]

With the current CPython, it's very hard to mix Python and C in a
multithreading application (with C-threads, not Python-threads). In fact
I never really succeeded in that task because of that GIL ! I have a
multi-thread application but every bit of Python code must be run into a
Python thread. To be more precise, I wanted to be able to call Python
code in response to some GUI events, and I didn't want to instanciate a
new interpreter for I wanted to be able to access the environment of my
main Python interpreter.

I don't understand. This is what PyGILState_Ensure and PyGILState_Release are
for - so C code can leave the GIL unlocked by default, and only grab it when
they want to call into the C/Python API.

Regards,
Nick.

 

[Pierre Barbier de Reuille]

Nick Coghlan a écrit :

I don't understand. This is what PyGILState_Ensure and
PyGILState_Release are for - so C code can leave the GIL unlocked by
default, and only grab it when they want to call into the C/Python API.

Regards,
Nick.

Ok, I wondered why I didn't know these functions, but they are new to
Python 2.4 ( and I didn't take the time to look closely at Python 2.4 as
some modules I'm working with are still not available for Python 2.4).
But if it really allows to call Python code outside a Python thread ...
then I'll surely use that as soon as I can use Python 2.4 Thanks for
the hint

Pierre

 

[Nick Coghlan]

Ok, I wondered why I didn't know these functions, but they are new to
Python 2.4 ( and I didn't take the time to look closely at Python 2.4 as
some modules I'm working with are still not available for Python 2.4).
But if it really allows to call Python code outside a Python thread ...
then I'll surely use that as soon as I can use Python 2.4 Thanks for
the hint

The Python 2.4 docs claim the functions were added in Python 2.3, even though
they aren't documented in the 2.3.4 docs.

The 2.3 release PEP (PEP 283) confirms that PEP 311 (which added these
functions) went in.

Cheers,
Nick.

 

[Pierre Barbier de Reuille]

Nick Coghlan a écrit :

The Python 2.4 docs claim the functions were added in Python 2.3, even
though they aren't documented in the 2.3.4 docs.

The 2.3 release PEP (PEP 283) confirms that PEP 311 (which added these
functions) went in.

Indeed, I just tested it and now it works fine Thanks a lot

 

[David Bolen]

The Python 2.4 docs claim the functions were added in Python 2.3, even
though they aren't documented in the 2.3.4 docs.

The 2.3 release PEP (PEP 283) confirms that PEP 311 (which added these
functions) went in.

And even before that it was certainly possible to call into the Python
interpreter from a native thread using existing functions, albeit the
newer functions are more convenient (and perhaps more robust, I don't
know).

My earliest interaction with Python (~1999, while writing a module
that extended and embedded Python 1.5.2) used PyEval_AcquireThread()
and PyEval_ReleaseThread() to get access to a thread state from a
native C application thread (not initiated by the Python interpreter)
to allow me to call safely into an executing Python script upon
asynchronous data reception by the C code.

-- David

 

[Pierre Barbier de Reuille]

David Bolen a écrit :

And even before that it was certainly possible to call into the Python
interpreter from a native thread using existing functions, albeit the
newer functions are more convenient (and perhaps more robust, I don't
know).

My earliest interaction with Python (~1999, while writing a module
that extended and embedded Python 1.5.2) used PyEval_AcquireThread()
and PyEval_ReleaseThread() to get access to a thread state from a
native C application thread (not initiated by the Python interpreter)
to allow me to call safely into an executing Python script upon
asynchronous data reception by the C code.

-- David

Yes, now you mention this I remember trying to use these functions ! But
there is two problems with them (and only one is solved by the new
functions) ! The first problem is the dead lock is you try to acquire
the same thread twice. This implies a very low level use of these
function, and it can be a real pain to find where to put them without
risking this dead lock. This is no more a problem with the new
functions. But the second (and not solved) problem is: these functions
block the current thread when you try to get the GIL ! This is an issue
if you don't want your GUI thread to be blocked when a buggy module does
not release the GIL during long computations. I didn't find any function
like "test the state of the GIL and if it's free, then acquire it" !
And, IMO, it's missing ... Then, if python goes to a model where
critical sections are identified and the GIL acquired only there, all
these problems will disappear ! That's why I really hope Python 3 will
include this kind of thread support ...

Pierre