problem to write a THREAD enabled python extension

[Andreas Otto]

Hi,

the following scenario:

1. using GIL
2. a pthread is created in a library and have to be "announced" to python
3. the thread is created in a "PyObject_CallObject" function call from
extension c code using a other extension c-code function called
from python code

python CRASH with invalid thread-state object
http://www.linuxjournal.com/article/3641
has information that GIL is NOT able to keep thread state information
for cross thread operation.

workaround ->
.......
PyGILState_STATE gilState = PyGILState_Ensure(); // PyEval_RestoreThread
// Call Python/C API...
PyGILState_Release(gilState); // PyEval_SaveThread
.......
// Create new thread...
.......
PyGILState_STATE gilState = PyGILState_Ensure(); // PyEval_RestoreThread
// Call Python/C API...
PyGILState_Release(gilState); // PyEval_SaveThread
.......

the problem is that PyGILState_Release have to be done in the call of
"PyObject_CallObject" and not in the extension c-code

-> for now i call it impossible to provide thread support if thread is
created in extension code

-> more info by valgrind
==29105==
==29105== Invalid read of size 4
==29105== at 0x4F099AB: PyEval_EvalFrameEx (ceval.c:3728)
==29105== by 0x4F0B476: PyEval_EvalCodeEx (ceval.c:3180)
==29105== by 0x4E9DE50: function_call (funcobject.c:630)
==29105== by 0x4E7610C: PyObject_Call (abstract.c:2160)
==29105== by 0x4E8BEFB: method_call (classobject.c:323)
==29105== by 0x4E7610C: PyObject_Call (abstract.c:2160)
==29105== by 0x4F04975: PyEval_CallObjectWithKeywords (ceval.c:3624)
==29105== by 0x72132DF: pymsgque_ProcCall (misc_python.c:48)
==29105== by 0x742A23E: pTokenInvoke (token.c:327)
==29105== by 0x742E7E2: sMqEventStart (msgque.c:908)
==29105== by 0x7425365: pEventStart (event.c:360)
==29105== by 0x742ECBF: MqProcessEvent (msgque.c:1081)
==29105== by 0x7215F24: pymsgque_ProcessEvent (context_python.c:176)
==29105== by 0x4F09BFA: PyEval_EvalFrameEx (ceval.c:3961)
==29105== by 0x4F0B476: PyEval_EvalCodeEx (ceval.c:3180)
==29105== by 0x4F0B58A: PyEval_EvalCode (ceval.c:650)
==29105== by 0x4F2EE29: PyRun_FileExFlags (pythonrun.c:1697)
==29105== by 0x4F2F134: PyRun_SimpleFileExFlags (pythonrun.c:1182)
==29105== by 0x4F43CFB: Py_Main (main.c:625)
==29105== by 0x400D11: main (python.c:152)
==29105== Address 0x24 is not stack'd, malloc'd or (recently) free'd


-> extension code ... (including some debugging output)

M0
PyGILState_STATE gstate = PyGILState_Ensure();
PyObject *result;
PyObject * const self = (PyObject*) CONTEXT->self;
PyObject * const callable = (PyObject*) dataP;
enum MqErrorE ret = MQ_OK;

// clean Python and libmsgque error
PyErr_Clear();
MqErrorReset(msgque->error);

// call the function
M1
printP(PyThreadState_Get())
if (PyMethod_Check(callable) && PyMethod_Self(callable) == self) {
//PyGILState_Release(gstate);
result = PyObject_CallObject(callable, NULL);
//gstate = PyGILState_Ensure();
} else {
//PyGILState_Release(gstate);
result = PyObject_CallFunctionObjArgs(callable, self, NULL);
//gstate = PyGILState_Ensure();
}
M2
printP(PyThreadState_Get())
Py_XDECREF(result);

// no error return OK
if (PyErr_Occurred() != NULL) {
NS(ErrorSet) (self);
ret = MqErrorGetCode(msgque->error);
}
PyGILState_Release(gstate);
return ret;
}

with GIL wrapper 'PyGILState_Release' and 'PyGILState_Ensure' enabled for
'PyObject_CallObject' the following SEG happen

if (PyMethod_Check(callable) && PyMethod_Self(callable) == self) {
PyGILState_Release(gstate);
result = PyObject_CallObject(callable, NULL);
gstate = PyGILState_Ensure();
} else {
PyGILState_Release(gstate);
result = PyObject_CallFunctionObjArgs(callable, self, NULL);
gstate = PyGILState_Ensure();
}
pymsgque_ProcCall(misc_python.c:32) -> 00000000000000000
pymsgque_ProcCall(misc_python.c:44) -> 11111111111111111
pymsgque_ProcCall(misc_python.c:45) -> PyThreadState_Get()<0x68c2ad8>
==29223==
==29223== Thread 2:
==29223== Invalid read of size 4
==29223== at 0x4E760EA: PyObject_Call (abstract.c:2158)
==29223== by 0x4F04975: PyEval_CallObjectWithKeywords (ceval.c:3624)
==29223== by 0x72132F7: pymsgque_ProcCall (misc_python.c:48)
==29223== by 0x7430092: MqLinkCreate (msgque.c:644)
==29223== by 0x7430AB0: MqDefaultLinkCreate (msgque.c:681)
==29223== by 0x742F73B: MqLinkCreate (msgque.c:346)
==29223== by 0x7432C3D: sSysServerThreadCreate (sys.c:502)
==29223== by 0x521606F: start_thread (in /lib64/libpthread-2.9.so)
==29223== by 0x5B5B10C: clone (in /lib64/libc-2.9.so)
==29223== Address 0x18 is not stack'd, malloc'd or (recently) free'd
==29223==
==29223== Process terminating with default action of signal 11 (SIGSEGV)


mfg

aotto1968

 

[Stefan Behnel]

Hi,

the following scenario:

1. using GIL
2. a pthread is created in a library and have to be "announced" to python
3. the thread is created in a "PyObject_CallObject" function call from
extension c code using a other extension c-code function called
from python code

python CRASH with invalid thread-state object

You forgot to create a thread state for the new thread. See the
PyThreadState_New() function.

Stefan

 

[Andreas Otto]

You forgot to create a thread state for the new thread. See the
PyThreadState_New() function.

this does not really solve the problem even if the original error does no
happen anymore ...

but a bucket of other error happen >>> sometimes <<< and this mean
that time and (I call it so) internal state stuff has influence on the
problem and create errors like:

1. http://mail.python.org/pipermail/python-list/2004-January/242740.html
2. various PyThreadState_Get -> PyThreadState_Get: no current thread
3. sometimes ThreadState has NULL sometimes not and NULL create an
CORE for "tstate" pointer access operations
4. ... a lot of other problems too

all problems together I come to the conclusion:

1. I spend ~ 1week for a job done in C# or java in a day - and have no
usable solution
2. python need a couple of 100+ C API function for the same job
java need 10+
3. the design on the C-API is amateurish -> significant parts are
implemented as GLOBAL variables
4. the thread support is not more than a joke -> go to 3
5. after pay a lot of effort to get something working the final solution
is significant slower than C# or JAVA
6. you can not truss any-thing in python because you never know
if under some circumstance you get a CORE

the resulting PYTHON code is usable but not significant more readable
as JAVA, C* or even C++ code


mfg

aotto1968