ANN: GMPY 1.10 alpha with support for Python 3

[casevh]

An alpha release of GMPY that supports Python 2 and 3 is available.
GMPY is a wrapper for the GMP multiple-precision arithmetic
library. The MPIR multiple-precision arithmetic library is also
supported. GMPY is available for download from
http://code.google.com/p/gmpy/

Support for Python 3 required many changes to the logic used to
convert between different numerical types. The result type of some
combinations has changed. For example, 'mpz' + 'float' now returns
an 'mpf' instead of a 'float'. See the file "changes.txt" for more
information.

In addition to support for Python 3, there are several other
changes and bug fixes:

- Bug fixes in mpz.binary() and mpq.binary().

- Unicode strings are accepted as input on Python 2.
(Known bug: works for mpz, fails for mpq and mpf)

- The overhead for calling GMPY routines has been reduced.
If one operand in a small integer, it is not converted to mpz.

- 'mpf' and 'mpq' now support % and divmod.

Comments on provided binaries

The 32-bit Windows installers were compiled using MPIR 1.2.1 and
will automatically recognize the CPU type and use code optimized for
that CPU.

Please test with your applications and report any issues found!

casevh

 

[casevh]

I discovered a serious bug with comparisons and have posted alpha2
which fixes that bug and adds Unicode support for Python 2.x

casevh

 

[Mensanator]

I discovered a serious bug with comparisons and have posted alpha2
which fixes that bug and adds Unicode support for Python 2.x

casevh

Damn! I was just congatulating myself for pulling off
a hat trick (there had been no point in downloading
3.x without gmpy so I have been putting it off):

- installing Python 3.1
- installing gmpy 1.10
- converting my Collatz Function library to 3.1 syntax

And it all worked smoothly, just had to add parentheses
to my print statements, change xrange to range and all
my / to // (the library is exclusively integer). I had
gmpy running in my library on 3.1 in about 10 minutes.

So I'll have to re-do the gmpy install. Shouldn't be
any big deal.

I started doing some real world tests. Generally, things
look good (nothing crashes, timing looks not bad) but
I'm getting some funny results on one of my tests, so
I'll report back when I have more information.

 

[Mensanator]

Damn! I was just congatulating myself for pulling off
a hat trick (there had been no point in downloading
3.x without gmpy so I have been putting it off):

- installing Python 3.1
- installing gmpy 1.10
- converting my Collatz Function library to 3.1 syntax

And it all worked smoothly, just had to add parentheses
to my print statements, change xrange to range and all
my / to // (the library is exclusively integer). I had
gmpy running in my library on 3.1 in about 10 minutes.

So I'll have to re-do the gmpy install. Shouldn't be
any big deal.

I started doing some real world tests. Generally, things
look good (nothing crashes, timing looks not bad) but
I'm getting some funny results on one of my tests, so
I'll report back when I have more information.

As I said, I was getting funny results from one of my tests.

It seemed to work ok, but timing varied from 2 to 88 seconds,
which seemed odd. The other tests were generally consistent
for a given environment (cpu speed, OS, Python version, gmpy
version).

At some point I watched the memory usage profile from Windows.
On the same machine I have both Python 2.6 and 3.1 installed,
with the appropriate gmpy 1.10 version loaded for each.

In Python 2.6, it looks like this:

memory usage profile Python 2.6 gmpy 1.1 Vista

/--------------\ /-------\ .....RESTART SHELL
/ .\/ \
____/ . \________
. .
. .
start of RUN start of RUN

The first "start of RUN" is the first time the test is run
(from IDLE). That change in usage represents about 700 MB
(I'm testing really BIG numbers, up to 500 million digits).

The memory remains allocated after the program terminates
(the flat plateau). When I run a second time, we see the
allocation dip, then climb back up to the plateau, so it
appears that the allocation never climbs above 1.1 GB.

Finally, doing a RESTART SHELL seems to completely free
the allocated memory. I assume this is normal behaviour.

With Python 3.1, it get this profile:

memory usage profile Python 3.1 gmpy 1.1 Vista

/-
/ |
/----------------/ \------\ .....RESTART SHELL
/ . \
____/ . \___________
. .
. .
start of RUN start of RUN

Here, at the start of the second RUN, it appears that new
memory is allocated BEFORE the previous is cleared. Is this
a quirk in the way 3.1 behaves? Here, the peak usage climbs
to 1.8 GB which I think causes VM thrashing accounting for
the increased execution times.

My guess is that gmpy is provoking, but not causing this
behaviour.

The actual test is:

t0 = time.time()
n=10
for k in range(1,n):
for i in range(1,n-2):
print((str(cf.gmpy.numdigits(cf.Type12MH(k,i))).zfill(n)),end=' ')
print()
print()
t1 = time.time()


The library function Type12MH is:

def Type12MH(k,i):
"""Find ith, kth Generation Type [1,2] Mersenne Hailstone using
the closed form equation

Type12MH(k,i)
k: generation
i: member of generation
returns Hailstone (a)
"""
ONE = gmpy.mpz(1)
TWO = gmpy.mpz(2)
SIX = gmpy.mpz(6)
NIN = gmpy.mpz(9)

if (k<1) or (i<1): return 0

i = gmpy.mpz(i)
k = gmpy.mpz(k)

# a = (i-1)*9**(k-1) + (9**(k-1) - 1)//2 + 1
# return 2**(6*a - 1) - 1

a = (i-ONE)*NIN**(k-ONE) + (NIN**(k-ONE) - ONE)//TWO + ONE
return TWO**(SIX*a - ONE) - ONE

## Sample runs
##
## Test 1 - create numbers up to 500 million digits
##
## 0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
## 0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
## 0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
## 0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
## 0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
## 0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
## 0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
## 0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
## 0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
## 15.5460000038
## >>> ================================ RESTART
================================
## >>>
## 0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
## 0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
## 0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
## 0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
## 0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
## 0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
## 0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
## 0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
## 0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
## 3.06299996376

 

[casevh]

As I said, I was getting funny results from one of my tests.

It seemed to work ok, but timing varied from 2 to 88 seconds,
which seemed odd. The other tests were generally consistent
for a given environment (cpu speed, OS, Python version, gmpy
version).

At some point I watched the memory usage profile from Windows.
On the same machine I have both Python 2.6 and 3.1 installed,
with the appropriate gmpy 1.10 version loaded for each.

In Python 2.6, it looks like this:

memory usage profile Python 2.6 gmpy 1.1 Vista

      /--------------\  /-------\ .....RESTART SHELL
     /               .\/         \
____/                .            \________
   .                 .
   .                 .
   start of RUN      start of RUN

The first "start of RUN" is the first time the test is run
(from IDLE). That change in usage represents about 700 MB
(I'm testing really BIG numbers, up to 500 million digits).

The memory remains allocated after the program terminates
(the flat plateau). When I run a second time, we see the
allocation dip, then climb back up to the plateau, so it
appears that the allocation never climbs above 1.1 GB.

Finally, doing a RESTART SHELL seems to completely free
the allocated memory. I assume this is normal behaviour.

With Python 3.1, it get this profile:

memory usage profile Python 3.1 gmpy 1.1 Vista

                         /-
                        / |
      /----------------/   \------\ .....RESTART SHELL
     /                 .           \
____/                  .            \___________
   .                   .
   .                   .
   start of RUN        start of RUN

Here, at the start of the second RUN, it appears that new
memory is allocated BEFORE the previous is cleared. Is this
a quirk in the way 3.1 behaves? Here, the peak usage climbs
to 1.8 GB which I think causes VM thrashing accounting for
the increased execution times.

Hmmm. It looks like memory is not being release properly. I don't see
that behavior under Linux. The running time is a very consistent 1.35
seconds. I'm traveling at the moment so it will be at least a week
before I can test under Windows.

Thanks for the report. I'll try to see if I can figure out what is
going on.

casevh

My guess is that gmpy is provoking, but not causing this
behaviour.

The actual test is:

t0 = time.time()
n=10
for k in range(1,n):
  for i in range(1,n-2):
    print((str(cf.gmpy.numdigits(cf.Type12MH(k,i))).zfill(n)),end=' ')
  print()
print()
t1 = time.time()

The library function Type12MH is:

def Type12MH(k,i):
    """Find ith, kth Generation Type [1,2] Mersenne Hailstone using
the closed form equation

    Type12MH(k,i)
    k: generation
    i: member of generation
    returns Hailstone (a)
    """
    ONE = gmpy.mpz(1)
    TWO = gmpy.mpz(2)
    SIX = gmpy.mpz(6)
    NIN = gmpy.mpz(9)

    if (k<1) or (i<1): return 0

    i = gmpy.mpz(i)
    k = gmpy.mpz(k)

    # a = (i-1)*9**(k-1) + (9**(k-1) - 1)//2 + 1
    # return 2**(6*a - 1) - 1

    a = (i-ONE)*NIN**(k-ONE) + (NIN**(k-ONE) - ONE)//TWO + ONE
    return TWO**(SIX*a - ONE) - ONE

##  Sample runs
##
##  Test 1 - create numbers up to 500 million digits
##
##  0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
##  0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
##  0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
##  0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
##  0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
##  0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
##  0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
##  0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
##  0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
##  15.5460000038
##  >>> ================================ RESTART
================================
##  >>>
##  0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
##  0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
##  0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
##  0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
##  0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
##  0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
##  0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
##  0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
##  0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
##  3.06299996376

 

[casevh]

GMPY 1.10 beta is now available. This version fixes an issue where
very large objects would be cached for reuse instead of being freed.

Source code and Windows installers may be found at
http://code.google.com/p/gmpy/downloads/list

casevh

 

[Mensanator]

GMPY 1.10 beta is now available. This version fixes an issue where
very large objects would be cached for reuse instead of being freed.

Excellent! That explains the funny memory usage graph.

Source code and Windows installers may be found athttp://code.google.com/p/gmpy/downloads/list

Test results:

import collatz_functions as cf
# imports gmpy
# the cf.Type12MH function does the equivalent
# of the pure Python version of the function
# using gmpy
import time

#3.1 gmpy 1.10 beta
#
t0 = time.time()
n=10
for k in range(1,n):
for i in range(1,n-2):
print((str(cf.gmpy.numdigits(cf.Type12MH(k,i))).zfill(n)),end=' ')
print()
print()
t1 = time.time()

print(t1-t0)

# 3.1 pure Python
#
##t0 = time.time()
##n=10
##for k in range(1,n):
## for i in range(1,n-2):
## print((str(cf.gmpy.numdigits( \
## 2**(6*((i-1)*9**(k-1)+(9**(k-1)-1)//2+1)-1)-1 \
## )).zfill(n)),end=' ')
## print()
##print()
##t1 = time.time()
##
##print(t1-t0)


## 3.1 gmpy 1.10 alpha
##
##0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
##0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
##0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
##0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
##0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
##0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
##0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
##0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
##0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
##3.32299995422
## funny memory usage display, different between 2.6 & 3.1, but funny
regardless,
## memory remains allocated (~700 MB) when program halts

## 3.1 pure Python
##
##0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
##0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
##0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
##0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
##0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
##0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
##0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
##0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
##0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
##338.832000017
## memory usage seems normal, runs slow enough to observe memory being
allocated
## and freed and remains freed when program halts

## 2.6 pure Python
##
##0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
##0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
##0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
##0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
##0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
##0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
##0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
##0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
##aborted after ONE FULL WEEK of processing (only 8 of 9 rows
completed)

## Something funny happened here. By the time I ran this test, there
## were 14 orphaned copies of pythonw.exe present, at least one of
which
## was still running along with the real pair executing the program.
Yet
## the Performance only registered ~50%. Normally, this would be 100%
if
## two programs were running. Pure Python is a LOT slower than gmpy,
## but not THAT slow. These orphaned processes may be a Python 3.1
issue.

##===================================================================

## 3.1 gmpy 1.10 beta
##
## 0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
## 0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
## 0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
## 0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
## 0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
## 0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
## 0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
## 0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
## 0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
## 2.41799998283
## memory usage now appears normal, no more allocated memory when
program halts
## and it's so fast all we see on Performance graph is a small blip,
much better
## than the 700-1400 MB it was allocating before.

## 3.1 pure Python
##
##0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
##0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
##0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
##0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
##0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
##0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
##0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
##0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
##0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
##339.611999989
## looks like gmpy about a 100 times faster. I knew there was a reason
I
## won't use Python without gmpy.

## 2.6 gmpy 1.10 beta
##
##0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
##0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
##0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
##0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
##0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
##0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
##0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
##0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
##0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
##2.68400001526

## 2.6 pure Python
##
##0000000002 0000000004 0000000006 0000000007 0000000009 0000000011
0000000013
##0000000009 0000000025 0000000042 0000000058 0000000074 0000000091
0000000107
##0000000074 0000000221 0000000367 0000000513 0000000659 0000000806
0000000952
##0000000659 0000001976 0000003293 0000004610 0000005926 0000007243
0000008560
##0000005926 0000017777 0000029627 0000041477 0000053328 0000065178
0000077028
##0000053328 0000159981 0000266634 0000373287 0000479940 0000586593
0000693246
##0000479940 0001439818 0002399696 0003359574 0004319453 0005279331
0006239209
##0004319453 0012958355 0021597258 0030236161 0038875064 0047513967
0056152869
##0038875064 0116625189 0194375315 0272125440 0349875565 0427625691
0505375816
##
##338.987999916
## ah, that's more like it. no more pythonw.exe thrashing. I was
almost ready
## to drop version 2.6 like a live grenade, but that was based on some
other
## fault apparently. it may be 3.1 I have to drop.

I still got a single instance of an orphaned pythonw.exe, but this
happened during a quick attempt at provoking it during a 3.1 pure
Python test, so I have no reason to suspect gmpy (previously, I
thought it may be related to the memory leak).

I've got an idea on how to reliably provoke this which I'll try to
work up later (and won't even load gmpy to make sure it's not
involved).