python extensions: including project local headers

[J Kenneth King]

Hey everyone,

I'm working on a python extension wrapper around Rob Hess'
implementation of a SIFT feature detector. I'm working on a
computer-vision based project that requires interfacing with Python at
the higher layers, so I figured the best way to handle this would be in
C (since my initial implementation in python was ungodly and slow).

I can get distutils to compile the extension and install it in the
python path, but when I go to import it I get the wonderful exception:

ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
symbol: _sift_features

Of course, _sift_features is a function defined in his header that I'm
#including in my extension.

His sources are sitting in my project root under sift/ while my source
is under src/ -- My setup.py is as follows:

from distutils.core import setup, Extension

pysift = Extension('pysift',
                   include_dirs = ['sift/include'],
                   sources = ['src/pysift.c'],
                   extra_link_args = ['-lm', '-lcv', '-lcxcore',
                                      '-lhighgui', '-lcvaux'])

setup(name = 'pysift',
      version = '0.0',
      description = 'A SIFT feature detection package',
      author = 'James Kenneth King',
      author_email = "[email protected]",
      url = "http://agentultra.com/",
      long_description = """
      A python extension package for detecting SIFT
      features using Rob Hess' C implementation.

      http://web.engr.oregonstate.edu/~hess/

      Original SIFT feature descriptors by David Lowe
      and patented by the University of British Columbia.
      """,
      ext_modules = [pysift])

And the include to Rob's header file is on the second line of pysift.c:

#include "sift.h"

The weird thing (to me in my somewhat hackish knowledge of C) is that I
can use all the #defines from sift.h with no complaints from the
preprocessor (in fact, there are no complaints at all from the compiler
when compiling the extension module).

Once I get this bugger working, I'll be setting up a project page to
share sources and will also be releasing extension wrappers to the
OpenCV libraries.

I've never released any code before so any help getting this right and
proper for the community would be greatly appreciated.

Cheers.

 

[Philip Semanchuk]

Hey everyone,

I'm working on a python extension wrapper around Rob Hess'
implementation of a SIFT feature detector. I'm working on a
computer-vision based project that requires interfacing with Python at
the higher layers, so I figured the best way to handle this would be
in
C (since my initial implementation in python was ungodly and slow).

I can get distutils to compile the extension and install it in the
python path, but when I go to import it I get the wonderful exception:

ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
symbol: _sift_features

Kenneth,
You're close but not interpreting the error quite correctly. This
isn't an error from the compiler or preprocessor, it's a library
error. Assuming this is dynamically linked, your OS is reporting that,
at runtime, it can't find the library that contains _sift_features.
Make sure that it's somewhere where your OS can find it.

HTH
Philip

 

[Robert Kern]

Kenneth,
You're close but not interpreting the error quite correctly. This isn't
an error from the compiler or preprocessor, it's a library error.
Assuming this is dynamically linked, your OS is reporting that, at
runtime, it can't find the library that contains _sift_features. Make
sure that it's somewhere where your OS can find it.

It looks like the library implementing it was not linked into the extension.
sift_features() is not part of OpenCV.

James, are you including the source of Rob Hess's implementation with your
extension, or are you trying to link against an already installed version of the
library? If the former, you need to add the C sources to the pysift Extension().
If the latter, you need to add the name of the library to the list of libraries.

Also, you don't want to pass the list of libraries with extra_link_args.
Instead, use libraries=.

pysift = Extension('pysift',
include_dirs = ['sift/include'],
sources = ['src/pysift.c'],
libraries = ['feat', 'cv', 'cxcore', 'highgui',
'cvaux', 'm'])

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[J Kenneth King]

It looks like the library implementing it was not linked into the
extension. sift_features() is not part of OpenCV.

James, are you including the source of Rob Hess's implementation with
your extension, or are you trying to link against an already installed
version of the library? If the former, you need to add the C sources
to the pysift Extension(). If the latter, you need to add the name of
the library to the list of libraries.

I'm including Rob Hess' sources with the extension.

Would that mean I should add library_dirs to Extension() to point to the
sources in the project's path?

Also, you don't want to pass the list of libraries with
extra_link_args. Instead, use libraries=.

pysift = Extension('pysift',
include_dirs = ['sift/include'],
sources = ['src/pysift.c'],
libraries = ['feat', 'cv', 'cxcore', 'highgui',
'cvaux', 'm'])

Thanks for taking a moment to help me out.

 

[Robert Kern]

I'm including Rob Hess' sources with the extension.

Would that mean I should add library_dirs to Extension() to point to the
sources in the project's path?

No, you would add the source file names to the sources= list.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[J Kenneth King]

Kenneth,
You're close but not interpreting the error quite correctly. This
isn't an error from the compiler or preprocessor, it's a library
error. Assuming this is dynamically linked, your OS is reporting that,
at runtime, it can't find the library that contains _sift_features.
Make sure that it's somewhere where your OS can find it.

This is basically what I was looking for help with. So far the project
directory is:

/pysift
/sift
..
/include
..
sift.h
/src
..
sift.c
/src
pysift.c
setup.py

I thought I could just #include "sift.h" in pysift.c as long as
distutils passed the right -I path to gcc.

Maybe I should compile the sift code as a shared object and link it to
my extension? How would I get distutils to build the makefile and tell
gcc how to link it?

Thanks for the reply. Python has spoiled me and my C is rather
rusty.

 

[Philip Semanchuk]

This is basically what I was looking for help with. So far the project
directory is:

/pysift
/sift
..
/include
..
sift.h
/src
..
sift.c
/src
pysift.c
setup.py

I thought I could just #include "sift.h" in pysift.c as long as
distutils passed the right -I path to gcc.

That's true, and it sounds like you've got that part working.

Maybe I should compile the sift code as a shared object and link it to
my extension? How would I get distutils to build the makefile and tell
gcc how to link it?

Thanks for the reply. Python has spoiled me and my C is rather
rusty.

I don't know how to get setup.py to build a shared object separately.
I am in the same Python/C situation as you. I'm scrubbing the rust off
of my C skills and I'm also a n00b at developing extensions. I've
learned a lot from looking at other people's setup code, so maybe I
can help you there.

My posix_ipc module links to the realtime lib "rt" and here's the
relevant snippets of setup.py:

------------------------------
import distutils.core as duc

libraries = [ ]

libraries.append("rt")

source_files = ["posix_ipc_module.c"]

ext_modules = [ duc.Extension("posix_ipc",
source_files,
define_macros=define_macros,
libraries=libraries
)
]

duc.setup(name="posix_ipc", version=VERSION, ext_modules=ext_modules)

------------------------------

You can download the whole thing here if you want to examine all the
code:
http://semanchuk.com/philip/posix_ipc/

HTH
Philip

 

[Robert Kern]

This is basically what I was looking for help with. So far the project
directory is:

/pysift
/sift
..
/include
..
sift.h
/src
..
sift.c
/src
pysift.c
setup.py

I thought I could just #include "sift.h" in pysift.c as long as
distutils passed the right -I path to gcc.

Maybe I should compile the sift code as a shared object and link it to
my extension? How would I get distutils to build the makefile and tell
gcc how to link it?

I don't recommend doing that, if you can avoid it. distutils does not really
support that. If you can get the sift files to compile under distutils as part
of the Extension, that is by far the best option.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[J Kenneth King]

This is basically what I was looking for help with. So far the project
directory is:

/pysift
/sift
..
/include
..
sift.h
/src
..
sift.c
/src
pysift.c
setup.py

I thought I could just #include "sift.h" in pysift.c as long as
distutils passed the right -I path to gcc.

That's true, and it sounds like you've got that part working.

Maybe I should compile the sift code as a shared object and link it to
my extension? How would I get distutils to build the makefile and tell
gcc how to link it?

Thanks for the reply. Python has spoiled me and my C is rather
rusty.

I don't know how to get setup.py to build a shared object separately.
I am in the same Python/C situation as you. I'm scrubbing the rust off
of my C skills and I'm also a n00b at developing extensions. I've
learned a lot from looking at other people's setup code, so maybe I
can help you there.

My posix_ipc module links to the realtime lib "rt" and here's the
relevant snippets of setup.py:

------------------------------
import distutils.core as duc

libraries = [ ]

libraries.append("rt")

source_files = ["posix_ipc_module.c"]

ext_modules = [ duc.Extension("posix_ipc",
source_files,
define_macros=define_macros,
libraries=libraries
)
]

duc.setup(name="posix_ipc", version=VERSION, ext_modules=ext_modules)

------------------------------

You can download the whole thing here if you want to examine all the
code:
http://semanchuk.com/philip/posix_ipc/

HTH
Philip

I'll take a look, thanks!