Module missing when embedding?

[Garthy]

Hi all,

I am attempting to embed Python 3.3.3 into an application.

Following advice which suggests not using multiple interpreters, I am
experimenting using a *single* interpreter and multiple threads.

So far I have been loading directly into "__main__", ie. something like
this:

PyObject *main_module = PyImport_AddModule("__main__");

PyObject *dg = PyModule_GetDict(main_module);
PyObject *dl = PyModule_GetDict(main_module);
PyObject *rv = PyRun_String(str, Py_file_input, dg, dl);

A stripped down script looks like this:

import mymodule

class Foo:
def bar(self):
mymodule.mycall("a")

mymodule is set up once as:

PyImport_AppendInittab("mymodule", PyInit_mymodule);
Py_SetProgramName((wchar_t *)"foo");
Py_InitializeEx(0);
PyEval_InitThreads();
mtstate = PyThreadState_Get();
PyEval_ReleaseThread(mtstate);

And per thread as:

PyImportModule_ImportModule("mymodule");

This works, and when an instance of Foo is created, calling bar() on it
triggers the mymodule mycall call.

I want to load scripts into their own dedicated module- I don't want
each thread loading into "__main__" if there is only one interpreter!
Anyway, let's try:

PyObject *module = PyModule_New();

PyObject *dg = PyModule_GetDict(module);
PyObject *dl = PyModule_GetDict(module);
PyObject *rv = PyRun_String(str, Py_file_input, dg, dl);

No good. I get:

"__import__ not found"

on load. Trying again: Let's point dg at the "__main__" module instead:

PyObject *dg = PyModule_GetDict(main_module);
PyObject *dl = PyModule_GetDict(module);
PyObject *rv = PyRun_String(str, Py_file_input, dg, dl);

and it loads. Is this the right way to go about it or have I done
something foolish?

Anyway, later on, I create an object of type Foo and call bar() on it,
exactly as I did before. I get:

"global name 'mymodule' is not defined"

Darn. The offending line is:

mymodule.mycall("a")

Now, in the script I load, this line is okay:

import mymodule

and this works if I try it:

from mymodule import mycall

but this does not:

from mymodule import call_that_does_not_exist

As expected. This suggests that Python understands there is a "mymodule"
module and it contains "mycall", and not "call_that_doesnt_exist".

However, as mentioned, when I call bar() on a Foo object, it tries to call:

mymodule.mycall("a")

which worked when it was loaded into "__main__", but now I get:

"global name 'mymodule' is not defined"

With "from mymodule import mycall" in the script, I try:

mycall("a")

instead, and I get:

"global name 'mycall' is not defined"

A trace in PyInit_mymodule confirms it is being run, ie. mymodule is
being set up. The import calls seem to confirm that "mymodule" exists,
and "mycall" exists within it. When loaded into __main__, it works as
expected. When loaded into a different module, it doesn't.

I structured a pure Python test that had the main script load one
module, which imported another module, and called it in the same way. It
worked.

I'll also point out that whilst I'm set up to use multiple threads, I am
only using two threads at the point of the errors. I do the global setup
in the main thread, and then never use it again, and do one lot of
per-thread setup in a child thread, after which the errors occur. I'm
being pedantic about GIL locking in any case.

I've had to transcribe the above code by hand. Whilst I've checked and I
think it's fine, there is a small chance of typos.

Any ideas about what I might be doing wrong? Anything I can try on the
Python side or the C API side? My Python knowledge is a bit rusty so I
may have missed something obvious on the Python side. If there are any
resources online that show something similar to what I am doing, please
share, and I'll do the legwork. More info available if needed- just ask.

Cheers,
Garth

PS. Finishing off a test suite to illustrate, will post soon. It doesn't
appear to be a thread issue.