Embedded python loading .so files?

[jefishman]

I have a Python (2.3.x) interpreter running embedded in a C++
application on a host machine. I would like to run a specific package
on that host machine (numpy). I have managed to compile
(cross-compile) the library, so that I have the python modules and the
compiled .so files. I am relatively sure that these would import
normally in a standard interpreter (they give bad magic on the build
machine).

However, I would like to be able to import this package dynamically
within the application running on the host machine. When I attempted
to import the package within the already loaded python modules, I would
get errors that the C portions of the package could not be found (the
pure-python modules would load, call to import module_foo, which is
module_foo.so, but would error "Cannot find module_foo").

How can I get the application to load the package? Do I need to load
it through the C API? I would heavily prefer not to recompile the
interpreter, if at all possible.

Thanks,
Jeremy

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

However, I would like to be able to import this package dynamically
within the application running on the host machine. When I attempted
to import the package within the already loaded python modules, I would
get errors that the C portions of the package could not be found (the
pure-python modules would load, call to import module_foo, which is
module_foo.so, but would error "Cannot find module_foo").

How can I get the application to load the package?

You really need to do some debugging here. There must be more error
message than that, or there are some issues with libdl on the target
system. A common problem is that it gives an error message that
some symbols could not be found when loading the shared library,
and that these symbols originate from Python run-time. In this
case, the reason is that the executable does not export the Python
symbols; you need to pass -export-dynamic to the linker of the
container application.

This is all guessing, of course: you didn't even say whether
the target system is Unixish-ELF-based.

Regards,
Martin

 

[jefishman]

You really need to do some debugging here. There must be more error

message than that, or there are some issues with libdl on the target
system. A common problem is that it gives an error message that
some symbols could not be found when loading the shared library,
and that these symbols originate from Python run-time. In this
case, the reason is that the executable does not export the Python
symbols; you need to pass -export-dynamic to the linker of the
container application.

This is all guessing, of course: you didn't even say whether
the target system is Unixish-ELF-based.

Regards,
Martin

Sorry, I never really know how much information in enough, and when
it's just complicating the issue. That obviously wasn't, and didn't
even get to the root of the problem:

The target system is mipsel-linux, specifically Sony's OPEN-R platform
running on a Sony ERS-7 Aibo robotic dog. I have recompiled passing
-export-dynamic, but I realized this is not the issue. Here is the
output of the import attempt from within the module imported by
PyImport_ImportModule(const char *name):

import numpy # directory /MS/PyLib/numpy
# trying /MS/PyLib/numpy/__init__.py
# /MS/PyLib/numpy/__init__.pyc matches /MS/PyLib/numpy/__init__.py
import numpy # precompiled from /MS/PyLib/numpy/__init__.pyc
# trying /MS/PyLib/numpy/__config.py
# /MS/PyLib/numpy/__config.pyc matches /MS/PyLib/numpy/__config.py
import numpy.__config__ # precompiled from /MS/PyLib/numpy/__config.pyc
# trying /MS/PyLib/numpy/version.py
# /MS/PyLib/numpy/version.pyc matches /MS/PyLib/numpy/version.py
import numpy.version # precompiled from /MS/PyLib/numpy/version.pyc
# trying /MS/PyLib/numpy/_import_.py
# /MS/PyLib/numpy/_import_.pyc matches /MS/PyLib/numpy/_import_.py
import numpy._import_tools # precompiled from
/MS/PyLib/numpy/_import_.pyc
# trying /MS/PyLib/numpy/os.py
# trying /MS/PyLib/numpy/os.pyc
# trying /MS/PyLib/numpy/sys.py
# trying /MS/PyLib/numpy/sys.pyc
# trying /MS/PyLib/numpy/imp.py
# trying /MS/PyLib/numpy/imp.pyc
import imp # builtin
# trying /MS/PyLib/numpy/glob.py
# trying /MS/PyLib/numpy/glob.pyc
# trying /MS/PyLib/glob.py
# /MS/PyLib/glob.pyc matches /MS/PyLib/glob.py
import glob # precompiled from /MS/PyLib/glob.pyc
# trying /MS/PyLib/fnmatch.py
# /MS/PyLib/fnmatch.pyc matches /MS/PyLib/fnmatch.py
import fnmatch # precompiled from /MS/PyLib/fnmatch.pyc
# trying /MS/PyLib/re.py
# /MS/PyLib/re.pyc matches /MS/PyLib/re.py
import re # precompiled from /MS/PyLib/re.pyc
# trying /MS/PyLib/sre.py
# /MS/PyLib/sre.pyc matches /MS/PyLib/sre.py
import sre # precompiled from /MS/PyLib/sre.pyc
# trying /MS/PyLib/sre_comp.py
# /MS/PyLib/sre_comp.pyc matches /MS/PyLib/sre_comp.py
import sre_compile # precompiled from /MS/PyLib/sre_comp.pyc
import _sre # builtin
# trying /MS/PyLib/sre_cons.py
# /MS/PyLib/sre_cons.pyc matches /MS/PyLib/sre_cons.py
import sre_constants # precompiled from /MS/PyLib/sre_cons.pyc
# trying /MS/PyLib/sre_pars.py
# /MS/PyLib/sre_pars.pyc matches /MS/PyLib/sre_pars.py
import sre_parse # precompiled from /MS/PyLib/sre_pars.pyc
# trying /MS/PyLib/string.py
# /MS/PyLib/string.pyc matches /MS/PyLib/string.py
import string # precompiled from /MS/PyLib/string.pyc
import numpy.core # directory /MS/PyLib/numpy/core
# trying /MS/PyLib/numpy/core/__init__.py
# /MS/PyLib/numpy/core/__init__.pyc matches
/MS/PyLib/numpy/core/__init__.py
import numpy.core # precompiled from /MS/PyLib/numpy/core/__init__.pyc
# trying /MS/PyLib/numpy/core/info.py
# /MS/PyLib/numpy/core/info.pyc matches /MS/PyLib/numpy/core/info.py
import numpy.core.info # precompiled from /MS/PyLib/numpy/core/info.pyc
# trying /MS/PyLib/numpy/core/numpy.py
# trying /MS/PyLib/numpy/core/numpy.pyc
# trying /MS/PyLib/numpy/core/multiarr.py
# trying /MS/PyLib/numpy/core/multiarr.pyc
# trying /MS/PyLib/multiarr.py
# trying /MS/PyLib/multiarr.pyc
# trying /MS/PyCode/multiarr.py
# trying /MS/PyCode/multiarr.pyc
Error Importing Behaviours
Traceback (most recent call last):
File "/MS/PyCode/Behaviou.py", line 22, in ?
import Brain
File "/MS/PyCode/Brain.py", line 21, in ?
import Switch
File "/MS/PyCode/Switch.py", line 1, in ?
import pNone as selectedPlayer
File "/MS/PyCode/pNone.py", line 2, in ?
import numpy.oldnumeric
File "/MS/PyLib/numpy/__init__.py", line 34, in ?
import core
File "/MS/PyLib/numpy/core/__init__.py", line 5, in ?
import multiarray
ImportError: No module named multiarray


I saw that the import did not look for .so files. I verified this by
checking imp.get_suffixes():

import imp # builtin
[('.py', 'U', 1), ('.pyc', 'rb', 2)] #<-- result of `print
imp.get_suffixes()`


I realize that this should include a ('.so','rb',3) entry (should it
not? 3->imp.C_EXTENSION) if it were going to locate the c extension.
Thus, my revised question would be what sets the suffixes for import?
How do/Can I change this?

The interpreter is linked in from a library compiled from edited Python
2.3 source so that it will run on the OPEN-R system (strips things like
time, socket, etc., which are not supported).


Thanks again,
Jeremy

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

I realize that this should include a ('.so','rb',3) entry (should it
not? 3->imp.C_EXTENSION) if it were going to locate the c extension.
Thus, my revised question would be what sets the suffixes for import?
How do/Can I change this?

It depends on the target supporting dynamic loading, and, in particular
HAVE_DYNAMIC_LOADING being defined in pyconfig.h. This, in turn, is
set in configure[.in], if DYNLOADFILE isn't dynload_stub.o. DYNLOADFILE
should be dynload_shlib.o if the system isn't AIX, BeOS, hp, Darwin,
or atheos, and if ac_cv_func_dlopen is defined (i.e. if the system has
dlopen).

HTH,
Martin