Building things with setup.py

[James Stroud]

Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?

Also, why would this not be caught by the developers of scipy and numpy?
What factors go into needing to specify libraries and to properly link
..so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

 

[James Stroud]

Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?

Also, why would this not be caught by the developers of scipy and numpy?
What factors go into needing to specify libraries and to properly link
.so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

James

Well I added those arguments to $LDFLAGS, but it seemed like a pretty
miserable hack, especially the "-shared" part.

I'm still curious how to make a 3rd party setup.py work correctly.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

 

[Robert Kern]

Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?

If it's actually necessary,

python setup.py build_ext -lmylib

Also, why would this not be caught by the developers of scipy and numpy?

Because we've never seen it before. There's no reason that you should have to
add those libraries manually. Can you post your build log (the one that fails)
to scipy-dev?

http://www.scipy.org/Mailing_Lists

What factors go into needing to specify libraries and to properly link
.so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?
>
I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

Possibly. Won't know more until I see the build log.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Robert Kern]

Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

It's possible that distutils is not finding the information about how the python
interpreter was built. Did you build your own python(1)? Is the information in
$PREFIX/lib/python2.4/config/ accurate?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[James Stroud]

It's possible that distutils is not finding the information about how
the python interpreter was built. Did you build your own python(1)? Is
the information in $PREFIX/lib/python2.4/config/ accurate?

I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions. Everything I do is compiled by hand as joe-user. I'm in a
situation where I can't do RPM (and I don't have root on my work machine
(theoretically ;-)) so, to be a good joe-user, everything I add goes
into the prefix:

$HOME/Programs

This is the listing from $HOME/Programs/lib/python2.5/config:

euler 6% ls
total 4092
8 config.c 12 install-sh* 44 Makefile 8 python.o
8 Setup.config
8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
8 Setup.local

The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.

The text files Setup.config and Setup.local do not seem to have terribly
specific information in them. Which file in particular should I inspect?

I did not capture output from the build and I could not find a file with
the word "log" in it that appears to be a build log. The jist of the
problem is that first it can't find symbols from libpython2.5, then from
libthread, etc. Then, it complains about no "MAIN__" when linking the
..so files with g77 and no "main" with gcc (which is curious), so I must
include the -shared flag, after including -llibrary type flags for all
of the libraries it doesn't know about.

If its necessary, I can run setup.py build again and send the output to
a file and post that to scipy-dev if you think it might be helpful.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

 

[Robert Kern]

I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions.

Do other extensions build correctly? If so, it's beginning to look like a
problem in numpy.distutils .

Everything I do is compiled by hand as joe-user. I'm in a
situation where I can't do RPM (and I don't have root on my work machine
(theoretically ;-)) so, to be a good joe-user, everything I add goes
into the prefix:

$HOME/Programs

This is the listing from $HOME/Programs/lib/python2.5/config:

euler 6% ls
total 4092
8 config.c 12 install-sh* 44 Makefile 8 python.o
8 Setup.config
8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
8 Setup.local

The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.

Hmm. That doesn't quite sound right, but it's been a while since I compiled the
interpreter from source.

The text files Setup.config and Setup.local do not seem to have terribly
specific information in them. Which file in particular should I inspect?

Makefile has most of that information. You can verify that distutils is finding
it like so:
{'AR': 'ar',
...
}

I did not capture output from the build and I could not find a file with
the word "log" in it that appears to be a build log. The jist of the
problem is that first it can't find symbols from libpython2.5, then from
libthread, etc. Then, it complains about no "MAIN__" when linking the
.so files with g77 and no "main" with gcc (which is curious), so I must
include the -shared flag, after including -llibrary type flags for all
of the libraries it doesn't know about.

What versions of gcc and g77 are you using?

If its necessary, I can run setup.py build again and send the output to
a file and post that to scipy-dev if you think it might be helpful.

Yes, that is what I intended.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Robert Kern]

Well I added those arguments to $LDFLAGS, but it seemed like a pretty
miserable hack, especially the "-shared" part.

Hmmm. Did you have $LDFLAGS set to anything (or more likely, nothing) before?
I.e. did you have something like this?

export LDFLAGS=""
python setup.py build

If such environment variables are present, I think that numpy.distutils may
clobber the values that are inferred from the configuration.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

That shouldn't be necessary. Linking without this should work just fine.

That way, since you made libpython2.5.a a static library, you link an
entire Python interpreter into each and every extension module. This
should not be done.

What happens if you omit these flags?

Regards,
Martin

 

[James Stroud]

Do other extensions build correctly? If so, it's beginning to look like
a problem in numpy.distutils .

It seems that every thing has built without incident (numarray, numeric,
mxtexttools, pil, reportlab, just to name a few), with the exception,
strangely enough, of python2.5 itself. Python required hiding previous
python installations.

Hmm. That doesn't quite sound right, but it's been a while since I
compiled the interpreter from source.

Sorry. To clarify, making libpython2.5a available in a $LD_LIBRARY_PATH
was necessary to build numpy and scipy.

Makefile has most of that information. You can verify that distutils is
finding it like so:

'/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/config/Makefile'

{'AR': 'ar',
...
}

This output from the check is here:
http://www.jamesstroud.com/build-scipy-logs/distutil-check.txt

I'm not sure if this is relevant to what you see in the check, but I
installed scipy_distutils AFTER installing scipy.

What versions of gcc and g77 are you using?

euler 2% g77 -v
Reading specs from
/auto_nfs/data10/users/jstroud/Programs/bin/../lib/gcc/i686-pc-linux-gnu/3.4.2/specs
Configured with: ./configure --prefix=/data1/users/jstroud/Programs :
(reconfigured) ./configure --prefix=/data1/users/jstroud/Programs
Thread model: posix
gcc version 3.4.2
euler 3% gcc -v
Reading specs from
/auto_nfs/data10/users/jstroud/Programs/bin/../lib/gcc/i686-pc-linux-gnu/3.4.2/specs
Configured with: ./configure --prefix=/data1/users/jstroud/Programs :
(reconfigured) ./configure --prefix=/data1/users/jstroud/Programs
Thread model: posix
gcc version 3.4.2

Yes, that is what I intended.

This is posted here:
http://www.jamesstroud.com/build-scipy-logs/build-scipy-py2.5-euler.txt

For symmetry, similar output for numpy is posted here:
http://www.jamesstroud.com/build-scipy-logs/build-numpy-py2.5-euler.txt

Please let me know if I can be of any further help with these issues.

James

 

[James Stroud]

That shouldn't be necessary. Linking without this should work just fine.

Unfortunately, this was not my experience.

That way, since you made libpython2.5.a a static library, you link an
entire Python interpreter into each and every extension module. This
should not be done.

Actually, a straight configure, make, install of the python2.5 source
produced this library.

What happens if you omit these flags?

Please see my last message to Robert Kern.

James

 

[James Stroud]

Do other extensions build correctly? If so, it's beginning to look like
a problem in numpy.distutils .

Actually, I just found that MySQLdb had a similar problem with zlib, but
instead of bonking during linking, it complained about the compress
symbol when trying to import into python. The fix was adding "-lz" for
the linking step, after which it imported just fine.

James

 

[Robert Kern]

Sorry. To clarify, making libpython2.5a available in a $LD_LIBRARY_PATH
was necessary to build numpy and scipy.

I don't see how that could have affected anything. Static libraries aren't
looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.

I'm not sure if this is relevant to what you see in the check, but I
installed scipy_distutils AFTER installing scipy.

scipy_distutils is no longer used. All of its functionality has been folded into
numpy.

What do your environment variables look like when you build?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[James Stroud]

I don't see how that could have affected anything. Static libraries
aren't looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.

OK--new thing learned for the day. <rant>I try things until a build
works. I consider everything I do until acquiring a working build as
necessary. Going back to see exactly what I did to make things work is
not a viable option given that every new program I want requires my
building it and all of its dependencies--a consequence of the [adjective
deleted] beuracratic constraints (with regard to the machine on my
desktop) that have evolved here. These constraints are no doubt a
byproduct of the "Institute" part of my signature below, associated
paranoia, and over-worked sys-admins.</rant>

The upside is that my entire environment turns out to be roughly
transportable to other Linux machines on the network via an NFS home
directory--more or less independent of the configuration of said other
machines.

What do your environment variables look like when you build?

euler 2% echo $CPPFLAGS
-I/data10/users/jstroud/Programs/include
-I/data10/users/jstroud/Programs/qt/include -I/usr/include
-I/data10/users/jstroud/Programs/include/freetype2/freetype
euler 3% echo $LD_LIBRARY_PATH
/data10/users/jstroud/Programs/lib:/data10/users/jstroud/Programs/qt/lib:/usr/lib:/usr/X11R6/lib/:/usr/local/lib:/usr/lib/gcc-lib/i386-redhat-linux/3.2.3:/data10/users/jstroud/Programs/ccp4-5.0.2/lib
euler 4% echo $LDFLAGS
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3

Here is how I customize my ld.so.cache:

euler 9% cat ldconf
rm ld.so.cache
/sbin/ldconfig -v -f ./ld.so.conf -C ./ld.so.cache
echo source this

euler 10% cat ld.so.conf
/data10/users/jstroud/Programs/lib
/data10/users/jstroud/Programs/qt/lib
/data10/users/jstroud/Programs/newlib
/usr/lib
/usr/local/lib
/usr/X11R6/lib
/usr/lib/qt3/lib
/usr/lib/wine
/usr/include/pgsql

Again, see the above rant for an explanation about any needless paths or
flags--though advice and knowledge is welcome.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

 

[James Stroud]

Hmmm. Did you have $LDFLAGS set to anything (or more likely, nothing)
before? I.e. did you have something like this?

export LDFLAGS=""
python setup.py build

If such environment variables are present, I think that numpy.distutils
may clobber the values that are inferred from the configuration.

Didn't see this message last night.

No, my $LDFLAGS were set. I haven't tried unsetenv'ing them (I use tcsh
as a consequence of my anti-bash religious upbringing).

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Please see my last message to Robert Kern.

If you don't want me to help you, that's fine, then I won't.

I couldn't find an answer to this question in any of your
messages in this thread.

Still, it *should* work out of the box. If it doesn't, it
means there is an error somewhere. If you don't want to
help finding the error, that's fine, too (but please say
so, because that will constrain the solution space).

Regards,
Martin

 

[James Stroud]

If you don't want me to help you, that's fine, then I won't.

I couldn't find an answer to this question in any of your
messages in this thread.

Sorry, I didn't mean to sound dismissive. Perhaps I misunderstood what
"these" actually referred to. The results of omitting these flags
specifically:

"-lpython2.5 -lpthread -lm -lutil -ldl -shared"

Are posted here for both numpy and scipy:

http://www.jamesstroud.com/build-scipy-logs/build-scipy-py2.5-euler.txt
http://www.jamesstroud.com/build-scipy-logs/build-numpy-py2.5-euler.txt

I have not tried unsetenv'ing $LDFLAGS entirely, an option which did not
occur to me given their necessity for just about everything else. If you
think that this is a worthwhile experiment, I will run it.

> Still, it *should* work out of the box. If it doesn't, it
> means there is an error somewhere. If you don't want to
> help finding the error, that's fine, too (but please say
> so, because that will constrain the solution space).

I think I would like to know how to avoid or correct these sort of
issues in the future, which seem to be limited, for me at least, to
scipy and numpy, with the possible exception of MySQLdb and its
dependency on zlib. Ideally, I would like to understand exactly what
causes these errors in addition to knowing what I can do to correct them
in the future.


James


--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

 

[Robert Kern]

I don't see how that could have affected anything. Static libraries
aren't looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.

OK--new thing learned for the day. <rant>I try things until a build
works. I consider everything I do until acquiring a working build as
necessary. Going back to see exactly what I did to make things work is
not a viable option given that every new program I want requires my
building it and all of its dependencies--a consequence of the [adjective
deleted] beuracratic constraints (with regard to the machine on my
desktop) that have evolved here. These constraints are no doubt a
byproduct of the "Institute" part of my signature below, associated
paranoia, and over-worked sys-admins.</rant>

I've always thought that a Kubuntu LiveCD would be a useful path around much
of that nonsense, but I've fortunately not been forced to attempt it.

euler 2% echo $CPPFLAGS
-I/data10/users/jstroud/Programs/include
-I/data10/users/jstroud/Programs/qt/include -I/usr/include
-I/data10/users/jstroud/Programs/include/freetype2/freetype
euler 3% echo $LD_LIBRARY_PATH
/data10/users/jstroud/Programs/lib:/data10/users/jstroud/Programs/qt/lib:/usr/lib:/usr/X11R6/lib/:/usr/local/lib:/usr/lib/gcc-lib/i386-redhat-linux/3.2.3:/data10/users/jstroud/Programs/ccp4-5.0.2/lib
euler 4% echo $LDFLAGS
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3

Okay, this is possibly part of the problem. numpy.distutils handles FORTRAN code
differently than other extension modules; this is why pure C extension modules
both inside numpy and elsewehere were linking fine. Try unsetenving $CPPFLAGS
and $LDFLAGS and adding those arguments to build_ext, if necessary.

Looking on line 516 of numpy/fcompiler/__init__.py (why there's a class
definition in __init__.py, I'll never know), it does look like the linker flags
which are carefully constructed by numpy.distutils are indeed overridden by the
user's setting of $LDFLAGS.

The fact that this appears to be incompatible with how distutils does it for
everything else is a bug, IMO, although it *is* a useful feature if you really
do need to completely control the linker flags. That's probably what the
implementor was thinking. Although, sadly, not documenting. He should be back
from his honeymoon, now, so I'll raise the issue with him.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Ben Finney]

<rant>I try things until a build works. I consider everything I do
until acquiring a working build as necessary. Going back to see
exactly what I did to make things work is not a viable option

Before trying such "I don't know what I need to do, but I'm going to
want to know after I do it" procedures, I usually start 'script' in
the shell session and then start all my experimenting. ('script' is a
tool from BSD available on most GNU/Linux distributions as well.)

That way, the entire session, fumbling dead-ends and all, is available
for later examination and trimming back to a "what I did to get it
working" document.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

I think I would like to know how to avoid or correct these sort of
issues in the future, which seem to be limited, for me at least, to
scipy and numpy, with the possible exception of MySQLdb and its
dependency on zlib. Ideally, I would like to understand exactly what
causes these errors in addition to knowing what I can do to correct them
in the future.

Let's take a specific failure, namely the line

/auto_nfs/data10/users/jstroud/Programs/bin/g77
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o
-L/data10/users/jstroud/Programs/lib -lblas -lg2c -o
build/lib.linux-i686-2.5/numpy/core/_dotblas.so

This gives errors like

build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o(.text+0x758):numpy/core/blasdot/_dotblas.c:226:
undefined reference to `PyTuple_New'

That's not too surprising: this line tries to link the input
*as an executable program*, despite calling the output
_dotblas.so. In an executable program, all symbols need to
be defined; that's why it it complains about so many missing
symbols (including MAIN__ - which should never be missing
in a library). Even though adding python2.5.a to the linker
link makes these symbols appear, the result still won't
work, as you can't use an executable program file as if
it were a shared library.

Now, compare this to a succeeding build of a C extension
module,

gcc -pthread -shared -L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
-I/data10/users/jstroud/Programs/include
-I/data10/users/jstroud/Programs/qt/include -I/usr/include
-I/data10/users/jstroud/Programs/include/freetype2/freetype
build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core/src/umathmodule.o
-lm -o build/lib.linux-i686-2.5/numpy/core/umath.so

Notice that this passes -shared to the compiler, requesting
construction of a shared library. This is the problem with
the g77 linker line; try invoking

/auto_nfs/data10/users/jstroud/Programs/bin/g77 -shared
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o
-L/data10/users/jstroud/Programs/lib -lblas -lg2c -o
build/lib.linux-i686-2.5/numpy/core/_dotblas.so

manually (whether or not -pthread is also necessary
or supported for g77, I don't know). This should at least
make the complaints about missing symbols go away; you
should then see whether the resulting module can be
imported in Python.

If that solves the problem, the question is why the
-shared option isn't passed automatically; your setting
LDFLAGS might indeed be a cause.

Regards,
Martin

 

[James Stroud]

Before trying such "I don't know what I need to do, but I'm going to
want to know after I do it" procedures, I usually start 'script' in
the shell session and then start all my experimenting. ('script' is a
tool from BSD available on most GNU/Linux distributions as well.)

That way, the entire session, fumbling dead-ends and all, is available
for later examination and trimming back to a "what I did to get it
working" document.

Beautiful suggestion. I've never even heard of that before. With OS X
Terminal.app, I can buffer "unlimited lines" on my 6 year old ibook I
use at home and on the road. But at work, on my brand new dual pentium,
I've found myself limited to the thousand or so lines that konsole gives me.

James