Compiling Python 2.5.2 on AIX 5.2

[Randy.Galbraith]

I'm investigating the possible use of Mecurial SCM as a replacement
for CVS. Mecurial is written in Python. I have a background in GNU/
Linux, Solaris, sparc and Perl. However AIX, powerpc and Python are
new to me.

--uname output--
$ uname -rvp
2 5 powerpc
--end uname output--

I used this script to compile Python:
--script--
export PATH=/usr/bin:/usr/vacpp/bin
export CC=xlC_r
export OBJECT_MODE=32
gunzip -c Python-2.5.2.tar.gz | tar xvf -
cd Python-2.5.2
../configure --with-gcc="xlc_r" --with-cxx="xlC_r" \
--disable-ipv6 AR="ar" --prefix=$HOME
make
--end script--

My concern is when I run make test I get this output:
--make test output--
275 tests OK.
2 tests failed:
test_mmap test_wait4
45 tests skipped:
test_aepack test_al test_applesingle test_bsddb test_bsddb185
test_bsddb3 test_bz2 test_cd test_cl test_codecmaps_cn
test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr
test_codecmaps_tw test_ctypes test_curses test_dl test_gdbm
test_gl test_gzip test_imgfile test_largefile test_linuxaudiodev
test_macfs test_macostools test_nis test_normalization
test_ossaudiodev test_pep277 test_plistlib test_scriptpackages
test_socket_ssl test_socketserver test_sqlite test_startfile
test_sunaudiodev test_tcl test_timeout test_urllib2net
test_urllibnet test_winreg test_winsound test_zipfile64
test_zipimport test_zlib
2 skips unexpected on aix5:
test_largefile test_ctypes
make: *** [test] Error 1
--end make test output--

My question are:

(a) Have you successfully compiled Python 2.5.2 on AIX 5.2? If so,
which options did you place in the environment and send to ./
configure?

(b) Given the choice between xlc and gcc 4.2.2 (which we have on the
platform) which one is considered more suitable?

(c) I am concerned about the two failing test cases: test_mmap and
test_wait4. Are there good reasons why these failures can be safely
ignored?

(d) Should I be concerned with the skips of test_largefile and
test_ctypes?

Much thanks in advance.

Kind regards,
-Randy Galbraith

 

[M.-A. Lemburg]

I'm investigating the possible use of Mecurial SCM as a replacement
for CVS. Mecurial is written in Python. I have a background in GNU/
Linux, Solaris, sparc and Perl. However AIX, powerpc and Python are
new to me.

On AIX 5.3, Python 2.5.2 should build out of the box using gcc.

We've successfully build Python 2.3, 2.4 and 2.5 on AIX 5.3
using the gcc compiler suite and tools installed from the
AIX Linux Toolbox:

http://www-03.ibm.com/systems/p/os/aix/linux/toolbox/altlic.html

../configure --enable-unicode=ucs2 --with-gcc

For 2.3 and 2.4 you need to patch pyconfig.h after the
configure run:

*** pyconfig.h~ Wed Nov 14 17:22:01 2007
--- pyconfig.h Wed Nov 14 17:22:01 2007
***************
*** 49,55 ****
/* #undef HAVE_BROKEN_POLL */

/* Define if the Posix semaphores do not work on your system */
! /* #undef HAVE_BROKEN_POSIX_SEMAPHORES */

/* Define if pthread_sigmask() does not work on your system. */
/* #undef HAVE_BROKEN_PTHREAD_SIGMASK */
--- 49,55 ----
/* #undef HAVE_BROKEN_POLL */

/* Define if the Posix semaphores do not work on your system */
! #define HAVE_BROKEN_POSIX_SEMAPHORES 1

/* Define if pthread_sigmask() does not work on your system. */
/* #undef HAVE_BROKEN_PTHREAD_SIGMASK */


--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Apr 14 2008)________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611

 

[Randy Galbraith]

On AIX 5.3, Python 2.5.2 should build out of the box using gcc.

We've successfully build Python 2.3, 2.4 and 2.5 on AIX 5.3
using the gcc compiler suite and tools installed from the
AIX Linux Toolbox:

http://www-03.ibm.com/systems/p/os/aix/linux/toolbox/altlic.html

./configure --enable-unicode=ucs2 --with-gcc

For 2.3 and 2.4 you need to patch pyconfig.h after the
configure run:

Marc-Andre,

Thanks so much for responding. I had to add --disable-ipv6 to ./
configure to prevent this problem:

checking getaddrinfo bug... buggy
Fatal: You must get working getaddrinfo() function.
or you can specify "--disable-ipv6".

When I checked pyconfig.h to make the patch you suggested I noticed
line 60 already reads like this:

#define HAVE_BROKEN_POSIX_SEMAPHORES 1

When ran make I immediately ran into problems. I'll investigate and
post back the results. Any comments and direction here is
appreciated.

$ make
gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -
Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o
Modules/python.o ./Modules/python.c
In file included from Include/Python.h:83,
from ./Modules/python.c:3:
Include/unicodeobject.h:73:2: error: #error Must define
Py_UNICODE_SIZE

What is Py_UNICODE_SIZE and why was it not defined? There are current
questions I have.

Kind regards,
-Randy Galbraith
ps. Based on uname -rvp output of "2 5 powerpc" I'm assuming my OS is
AIX 5.2 on PowerPC. I noticed you said you're on AIX 5.3. I'm not
100% sure I'm understanding uname output correctly.

 

[Martin v. Löwis]

What is Py_UNICODE_SIZE and why was it not defined? There are current

questions I have.

Py_UNICODE_SIZE is the number of bytes that a Py_UNICODE value should
have in the interpreter. With --enable-unicode=ucs2, it should be 2.

I cannot guess why it is not defined; check pyconfig.h to find out
whether there is a definition. If not, look in your configure output
for the line

checking what type to use for unicode...

and perhaps edit configure to print out additional messages around
the place where it deals with Py_UNICODE.

Regards,
Martin

 

[Randy Galbraith]

Py_UNICODE_SIZE is the number of bytes that a Py_UNICODE value should
have in the interpreter. With --enable-unicode=ucs2, it should be 2.

Martin,

Thanks for your reply. I feel like a dummy, when I was following Marc-
Andre's instructions I incorrectly typed "--enable-
unicode=ucs24" (note the "4"). Once I fixed that the Py_UNICODE_SIZE
issue went away.

Alas, I still am having problems compiling and getting a clean run
through make test. I got this error:

Python-2.5.2/Modules/bz2module.c:12:19: error: bzlib.h: No such file
or directory

Which I solved by copying bzlib.h I have on the system to Python-2.5.2/
Include/.

The run of make test resulted in this error:

test_ctypes
find_library('c') -> None
find_library('m') -> None
make: *** [test] Segmentation fault (core dumped)

An examination of the core with gdb shows this:

$ gdb python core
GNU gdb 6.0
Copyright 2003 Free Software Foundation, Inc.
[snip]
This GDB was configured as "powerpc-ibm-aix5.2.0.0"...
Core was generated by `python'.
Program terminated with signal 11, Segmentation fault.
#0 0xdebccc5c in ffi_closure_helper_DARWIN (closure=0xdebcd050,
rvalue=0x2ff20230, pgr=0x2ff20258, pfr=0x2ff201c8)
at Python-2.5.2/Modules/_ctypes/libffi/src/powerpc/ffi_darwin.c:
626
626 avalue = alloca(cif->nargs * sizeof(void *));
(gdb) bt
#0 0xdebccc5c in ffi_closure_helper_DARWIN (closure=0xdebcd050,
rvalue=0x2ff20230, pgr=0x2ff20258, pfr=0x2ff201c8)
at Python-2.5.2/Modules/_ctypes/libffi/src/powerpc/ffi_darwin.c:
626
#1 0xdebcd2bc in ffi_closure_ASM () from build/lib.aix-5.2-2.5/
_ctypes.so
#2 0xdebcd458 in ffi_call_AIX () from build/lib.aix-5.2-2.5/
_ctypes.so
#3 0xdebccf24 in ffi_call (cif=0xdebcd050, fn=@0x2ff20288:
0x2ff20350,
rvalue=0x2ff20258, avalue=0x2ff201c8)
at Python-2.5.2/Modules/_ctypes/libffi/src/powerpc/ffi_darwin.c:
421
#4 0xdebcb5e8 in _CallProc (pProc=@0x30000fc8: 0xdebcd248
<ffi_closure_ASM>,
argtuple=0x20d5e9ac, flags=4097, argtypes=0x20d5eb2c,
restype=0x20e7dc44,
checker=0x0)
at Python-2.5.2/Modules/_ctypes/callproc.c:668

[snip frames #5 to #55]

#56 0x10063a70 in PyEval_EvalCode (co=0xdebcd050, globals=0x2ff20230,
locals=0x2ff20258) at Python/ceval.c:494
cals=0x2ff20258) at Python/ceval.c:494
(gdb) print cif
$1 = (ffi_cif *) 0x140
(gdb) print *cif
$3 = {abi = 1600485481, nargs = 1946157056, arg_types = 0x0, rtype =
0x400a,
bytes = 1, flags = 0}

Thus it would seem use cif here resulted in a segment violation. I'll
continue to research this issue and report back to the group as I know
more. Perhaps solving the issue with the 'c' and 'm' libraries
(whatever they might be) will make the core dump go away. However,
for tonight, I'll need to stop here.

Kind regards,
-Randy Galbraith

 

[Martin v. Löwis]

Thus it would seem use cif here resulted in a segment violation. I'll

continue to research this issue and report back to the group as I know
more. Perhaps solving the issue with the 'c' and 'm' libraries
(whatever they might be) will make the core dump go away. However,
for tonight, I'll need to stop here.

I recommend you disable compilation of ctypes (by removing the call
to detect_ctypes from setup.py). It's fairly unlikely that you can
manage to make ctypes work on your system.

Regards,
Martin

 

[Randy Galbraith]

I recommend you disable compilation of ctypes (by removing the call
to detect_ctypes from setup.py). It's fairly unlikely that you can
manage to make ctypes work on your system.

Martin, Thanks again. I'm much closer now. Here is the script I'm
using to compile Python on AIX:

--begin script--
cp /path/to/include/bzlib.h Include/.
cat <<EOF | patch setup.py
*** setup.py.xxreg042108 2008-04-21 13:19:24.000000000 -0700
--- setup.py 2008-04-21 13:20:56.000000000 -0700
***************
*** 1052,1058 ****
exts.append( Extension('dl', ['dlmodule.c']) )

# Thomas Heller's _ctypes module
! self.detect_ctypes(inc_dirs, lib_dirs)

# Platform-specific libraries
if platform == 'linux2':
--- 1052,1059 ----
exts.append( Extension('dl', ['dlmodule.c']) )

# Thomas Heller's _ctypes module
! #self.detect_ctypes(inc_dirs, lib_dirs)
! # Removed for AIX as per Martin v. Lowis 4/21/08

# Platform-specific libraries
if platform == 'linux2':
EOF
./configure --enable-unicode=ucs2 --with-gcc\
--disable-ipv6 --prefix=$HOME
--end script--

make then compiles relatively cleanly. However, make test results in
this:

276 tests OK.
2 tests failed:
test_mmap test_wait4
44 tests skipped:
test_aepack test_al test_applesingle test_bsddb test_bsddb185
[snip]
2 skips unexpected on aix5:
test_largefile test_ctypes
make: *** [test] Error 1

The concern of course is why mmap and wait4 failed. Earlier in the
make test output we see this for mmap:

test test_mmap crashed -- <type 'exceptions.EnvironmentError'>: [Errno
22] Invalid argument

and for wait4:

test test_wait4 failed -- Traceback (most recent call last):
File "Python-2.5.2/Lib/test/fork_wait.py", line 75, in test_wait
self.wait_impl(cpid)
File "Python-2.5.2/Lib/test/test_wait4.py", line 28, in wait_impl
self.assertEqual(spid, cpid)
AssertionError: 0 != 6840386

What do these failures indicate? Not sure of course, but will
research and report back.

Kind regards,
-Randy Galbraith
"We seem to be made to suffer. It's our lot in life." - C3PO of Star
Wars

 

[Martin v. Löwis]

22] Invalid argument

You should run this with -v. This is too little detail to know what
exactly failed.

self.assertEqual(spid, cpid)
AssertionError: 0 != 6840386

What do these failures indicate?

That suggests a bug in wait4: apparently, it fails to correctly return
the PID. Could be an OS bug, but more likely, it's a type problem in
Modules/posixmodule.c.

Regards,
Martin

 

[Randy Galbraith]

test test_mmap crashed -- <type 'exceptions.EnvironmentError'>: [Errno
22] Invalid argument

You should run this with -v. This is too little detail to know what
exactly failed.

Sorry it took so long to get back to you. At this point, I'm going to
attempt to use python on AIX 5.2 even with the mmap and wait4
failures. My main requirement running Mercurial SCM, so hopefully the
python binary I have will do the job.

Nonetheless, I wanted to quickly dump out the expanded results you
suggested might help.

Running this:
./python Lib/test/test_mmap.py -v

Resulted in this:
--results--
<type 'mmap.mmap'>
Position of foo: 1.0 pages
Length of file: 2.0 pages
Contents of byte 0: '\x00'
Contents of first 3 bytes: '\x00\x00\x00'

Modifying file's content...
Contents of byte 0: '3'
Contents of first 3 bytes: '3\x00\x00'
Contents of second page: '\x00foobar\x00'
Regex match on mmap (page start, length of match): 1.0 6
Seek to zeroth byte
Seek to 42nd byte
Seek to last byte
Try to seek to negative position...
Try to seek beyond end of mmap...
Try to seek to negative position...
Attempting resize()
Creating 10 byte test data file.
Opening mmap with access=ACCESS_READ
Ensuring that readonly mmap can't be slice assigned.
Ensuring that readonly mmap can't be item assigned.
Ensuring that readonly mmap can't be write() to.
Ensuring that readonly mmap can't be write_byte() to.
Ensuring that readonly mmap can't be resized.
Opening mmap with size too big
Opening mmap with access=ACCESS_WRITE
Modifying write-through memory map.
Opening mmap with access=ACCESS_COPY
Modifying copy-on-write memory map.
Traceback (most recent call last):
File "Lib/test/test_mmap.py", line 393, in <module>
test_both()
File "Lib/test/test_mmap.py", line 247, in test_both
m.flush()
EnvironmentError: [Errno 22] Invalid argument
--end results--

btw I'm not sure if -v was required in this direct run. Running w/o -
v gave the same results.

Now, over to wait4. Your comment was:

That suggests a bug in wait4: apparently, it fails to correctly return
the PID. Could be an OS bug, but more likely, it's a type problem in
Modules/posixmodule.c.

Running this:
./python Lib/test/test_wait4.py

Resulted in this:
--results--
test_wait (__main__.Wait4Test) ... FAIL

======================================================================
FAIL: test_wait (__main__.Wait4Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/..<snip>../Python-2.5.2/Lib/test/fork_wait.py", line 75, in
test_wait
self.wait_impl(cpid)
File "Lib/test/test_wait4.py", line 28, in wait_impl
self.assertEqual(spid, cpid)
AssertionError: 0 != 8417358

----------------------------------------------------------------------
Ran 1 test in 12.066s

FAILED (failures=1)
Traceback (most recent call last):
File "Lib/test/test_wait4.py", line 37, in <module>
test_main()
File "Lib/test/test_wait4.py", line 33, in test_main
run_unittest(Wait4Test)
File "/..<snip>../Python-2.5.2/Lib/test/test_support.py", line 451,
in run_unittest
run_suite(suite, testclass)
File "/..<snip>../Python-2.5.2/Lib/test/test_support.py", line 436,
in run_suite
raise TestFailed(err)
test.test_support.TestFailed: Traceback (most recent call last):
File "/..<snip>../Python-2.5.2/Lib/test/fork_wait.py", line 75, in
test_wait
self.wait_impl(cpid)
File "Lib/test/test_wait4.py", line 28, in wait_impl
self.assertEqual(spid, cpid)
AssertionError: 0 != 8417358
--end results--

Thanks for taking your time to respond. It is truly appreciated at
this end.

Kind regards,
-Randy Galbraith

 

[Martin v. Löwis]

Opening mmap with access=ACCESS_COPY

Modifying copy-on-write memory map.
Traceback (most recent call last):
File "Lib/test/test_mmap.py", line 393, in <module>
test_both()
File "Lib/test/test_mmap.py", line 247, in test_both
m.flush()
EnvironmentError: [Errno 22] Invalid argument

I see. This is

http://bugs.python.org/issue678250

Regards,
Martin