Embedded python with threads

[nik]

hi,

I have a C++ app that loads a python module and calls a function from
it. The call is getting the contents of a list that's global in that module.

However, I'd like the python script to be also running a thread that
fills that global list, but can't figure out how to do it. Essentially
the code looks like;

in the C++ app;

////
Py_Initialize();

// PyImport_ImportModule blocks until the myModule script has run
// through...
PyObject* module = PyImport_ImportModule("myModule");

PyObject* function = PyObject_GetAttrString(module, "GetList");
while(1) {
PyObject* pyList = PyObject_CallFunction(function, "");
// use the stuff in pyList
sleep(15);
}
////

and in the myModule.py;

####
import thread
import time
orderListLock = thread.allocate_lock()

pyList = []

def GetList():
global pyList
tmpList = []
orderListLock.acquire()
tmpList = pyList
orderListLock.release()
return tmpList

def keepFillingListThread():
global pyList
while 1:
orderListLock.acquire()
pyList.append(somestuff)
orderListLock.release()
time.sleep(5)

# the following statement happens when the C++ app imports the module,
# but the thread only lives as long the main python thread (which is
# over straight away on my linux PC)

thread.start_new_thread(keepFillingListThread, ())

####

Has anyone any ideas?

thanks,
nik

 

[nik]

hi,

I have a C++ app that loads a python module and calls a function from
it. The call is getting the contents of a list that's global in that
module.

However, I'd like the python script to be also running a thread that
fills that global list, but can't figure out how to do it. Essentially
the code looks like;

in the C++ app;

////
Py_Initialize();

// PyImport_ImportModule blocks until the myModule script has run
// through...
PyObject* module = PyImport_ImportModule("myModule");

PyObject* function = PyObject_GetAttrString(module, "GetList");
while(1) {
PyObject* pyList = PyObject_CallFunction(function, "");
// use the stuff in pyList
sleep(15);
}
////

and in the myModule.py;

####
import thread
import time
orderListLock = thread.allocate_lock()

pyList = []

def GetList():
global pyList
tmpList = []
orderListLock.acquire()
tmpList = pyList
orderListLock.release()
return tmpList

def keepFillingListThread():
global pyList
while 1:
orderListLock.acquire()
pyList.append(somestuff)
orderListLock.release()
time.sleep(5)

# the following statement happens when the C++ app imports the module,
# but the thread only lives as long the main python thread (which is
# over straight away on my linux PC)

thread.start_new_thread(keepFillingListThread, ())

####

Has anyone any ideas?

thanks,
nik

Hi again,

I think I'm one step further. I've put the
thread.start_new_thread(keepFillingListThread, ()) command into its own
function, like

def startThreads():
thread.start_new_thread(keepFillingListThread, ())
while 1:
pass

and then in the C++ part, I've used boost::thread to create a thread to
call startThreads like;

void myBoostThread:perator () ()
{
PyObject* function = PyObject_GetAttrString(m_cfg.module, "createThreads");
PyObject* result = PyObject_CallFunction(function, "");
}

// after the PyObject* module = PyImport_ImportModule("myModule"); line
from above:
myBoostThreadParams params;
params.module = module;
myBoostThread mythread(params);
boost::thread theThread(mythread);

this seems to work (at least, the python threads stay alive), but it
rapidly crashes out with

Fatal Python error: ceval: tstate mix-up
Aborted

as soon as I call the function GetList above.

Can anyone help me?

thanks

 

[Greg Chapman]

////
Py_Initialize();

// PyImport_ImportModule blocks until the myModule script has run
// through...
PyObject* module = PyImport_ImportModule("myModule");

PyObject* function = PyObject_GetAttrString(module, "GetList");
while(1) {
PyObject* pyList = PyObject_CallFunction(function, "");
// use the stuff in pyList
sleep(15);
}

I'm not sure I fully understand your problem, but in the loop above, you should
have:

Py_BEGIN_ALLOW_THREADS
sleep(15);
Py_END_ALLOW_THREADS

This releases the Python GIL during the sleep call, allowing your list-filling
thread to run.

# the following statement happens when the C++ app imports the module,
# but the thread only lives as long the main python thread (which is
# over straight away on my linux PC)

This is what I don't understand, since the Python main thread should be the
thread on which Py_Initialize was called; it should remain alive (from Python's
perspective) until Py_Finalize is called.

 

[nik]

I'm not sure I fully understand your problem, but in the loop above, you should
have:

Py_BEGIN_ALLOW_THREADS
sleep(15);
Py_END_ALLOW_THREADS

This releases the Python GIL during the sleep call, allowing your list-filling
thread to run.




This is what I don't understand, since the Python main thread should be the
thread on which Py_Initialize was called; it should remain alive (from Python's
perspective) until Py_Finalize is called.

I think I was making an assumption that since the list filling threads
didn't run that the import module was calling the create thread command,
then exiting (killing the children threads along with it) - I'm a novice
at threads with python, so I'm probably misunderstanding the general
picture. However, maybe your suggestion of the Py_BEGIN_ALLOW_THREADS
will make the difference. I'll try again as soon as I can (not until
next week now)...

many thanks for the reply,
nik

 

[Greg Chapman]

I think I was making an assumption that since the list filling threads
didn't run that the import module was calling the create thread command,
then exiting (killing the children threads along with it) - I'm a novice
at threads with python, so I'm probably misunderstanding the general
picture. However, maybe your suggestion of the Py_BEGIN_ALLOW_THREADS
will make the difference. I'll try again as soon as I can (not until
next week now)...

OK, this sounds like you were definitely not releasing the GIL (the Global
Interpreter Lock). To get an idea of how Python's threading works (from the C
side of things), I'd suggest reading this:

http://www.python.org/dev/doc/devel/api/threads.html

That's from the documentation for 2.4, but I believe everything there applies to
2.3 as well. I'm linking to it because it includes some discussion of the
PyGILState functions (this discussion is missing from the 2.3 docs, but the
functions are there).