Questions on 64 bit versions of Python

[python]

Background: I'm going to be processing some raw transaction logs that
are 30G in size. As part of this processing I may need to create some
very large dictionary structures. I will be running my scripts on a
version of Windows 2003 Server Enterprise Edition that supports 16G of
RAM. Yes, I could use a database in place of dictionaries, but I'm
looking for maximum performance.

The following page lists two 64 bit versions of Python for Windows:
http://www.python.org/download/releases/2.5.2/

For Win64-Itanium users: python-2.5.2.ia64.msi
For Win64-AMD64 users: python-2.5.2.amd64.msi

1. It looks like the 64 bit versions of Python for Windows are CPU
vendor specific, eg. it doesn't look like there's a single, universal
executable for Windows 64 bit platforms. Is this true?

2. Are there limitations to the using the 64 bit versions of Python? I
seem to remember reading that many 3rd party modules (especially Windows
OS specific modules) may not be compatible with the 64 bit versions of
Python for Windows.

3. If I wanted to run a 64 bit version of Python under Linux, would I
need to recompile from source on 64 bit version of Linux or do 64 bit
versions of Linux automatically ship with 64 bit versions of Python?
(Any recommendations on a flavor of 64 bit of Linux for the Intel
architecture would be appreciated)

4. Is there a stable version of IronPython compiled under a 64 bit
version of .NET? Anyone have experience with such a beast?

Thank you,
Malcolm

 

[Martin v. Löwis]

(Any recommendations on a flavor of 64 bit of Linux for the Intel

architecture would be appreciated)

My recommendation is to use Debian or Ubuntu, as that's my personal
preference. As MAL said, any recent distribution that supports AMD64
should be fine (assuming you are not interested in Itanium, SPARC64,
HPPA (2.0), PPC64, or Alpha, which are all 64-bit flavors of Linux).

Regards,
Martin

 

[Martin v. Löwis]

(Any recommendations on a flavor of 64 bit of Linux for the Intel

architecture would be appreciated)

My recommendation is to use Debian or Ubuntu, as that's my personal
preference. As MAL said, any recent distribution that supports AMD64
should be fine (assuming you are not interested in Itanium, SPARC64,
HPPA (2.0), PPC64, or Alpha, which are all 64-bit flavors of Linux).

Regards,
Martin

 

[Tim Roberts]

For Win64-Itanium users: python-2.5.2.ia64.msi
For Win64-AMD64 users: python-2.5.2.amd64.msi

1. It looks like the 64 bit versions of Python for Windows are CPU
vendor specific, eg. it doesn't look like there's a single, universal
executable for Windows 64 bit platforms. Is this true?

It's true for ALL operating systems, not just Windows. The ia64 (Itanium)
and the amd64 are completely separate processors with VERY different
instruction sets.

 

[Roel Schroeven]

Tim Roberts schreef:

It's true for ALL operating systems, not just Windows. The ia64 (Itanium)
and the amd64 are completely separate processors with VERY different
instruction sets.

Maybe I'm wrong, but I have the impression that Malcolm is a bit
confused about the different 64-bit processors from Intel and AMD. So,
to clear things up a bit:

- The Itanium (ia64) is a 64-bit processor made by Intel, mostly for use
in high-end systems (servers etc.) IIRC. It's instruction set is indeed
completely different from the i386 instruction set (though I think
there's an emulation mode)

- AMD64 (or x86-64 or x64 or EMT64 or Intel64) is a 64-bit instruction
set from AMD which is an extension to the i386 instruction set, and runs
32-bit (and 16-bit) i386-code natively. But, and this is important,
despite the name the instruction set is also used by Intel (though they
call it EMT64 and made a few minor changes).

If you have or buy a 64-bit computer, it almost certainly uses the AMD64
instruction set, even if it has an Intel CPU (if you have an Itanium or
other 64-bit CPU, I suppose you would know; it would not be something
you buy from the shelf in a regular computer store). So you almost
certainly need the AMD64 version of Python.

--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
-- Isaac Asimov

Roel Schroeven

 

[Martin v. Löwis]

- AMD64 (or x86-64 or x64 or EMT64 or Intel64) is a 64-bit instruction

set from AMD which is an extension to the i386 instruction set, and runs
32-bit (and 16-bit) i386-code natively. But, and this is important,
despite the name the instruction set is also used by Intel (though they
call it EMT64 and made a few minor changes).

Indeed, there are (unfortunately) many names for the architecture.
Originally, AMD called it x86-64, and later renamed it to AMD64. Intel
originally implemented it under the name EM64T (for Extended Memory 64
Technology), and now calls the architecture Intel 64.

Microsoft (apparently not wanting to take sides) calls it x64.

I personally believe that whoever invents a technology also
gets to name it, so I encourage use of the name AMD gives it
(which is AMD64). That's why the Python installer has that label
in its name.

Regards,
Martin

 

[William McBrine]

Originally, AMD called it x86-64, and later renamed it to AMD64. Intel
originally implemented it under the name EM64T (for Extended Memory 64
Technology), and now calls the architecture Intel 64.

I hadn't heard "Intel 64" before. That's a bit nervy, isn't it? Plus it
seems to conflict with their own use of "IA-64" (Intel Architecture 64)
for the Itanium (vs. "IA-32" for traditional x86).

 

[Martin v. Löwis]

I hadn't heard "Intel 64" before. That's a bit nervy, isn't it? Plus it

seems to conflict with their own use of "IA-64" (Intel Architecture 64)
for the Itanium (vs. "IA-32" for traditional x86).

Indeed. Microsoft Installer has an architecture string for the MSI file;
"Intel64" there means Itanium (and AMD64 can be denoted by either
"AMD64" or "x64"). This was from a time when Intel still told people
that the future of 64-bit computing is in Itanium (and architecturally,
I still think this would have been the right choice).

Regards,
Martin

 

[python]

Dear List,

Thanks for everyone's feedback - excellent detail - all my questions
have been answered.

BTW: Roel was correct that I got confused over the AMD and Intel naming
conventions regarding the 64 bit versions of Python for Windows. (I
missed that nuance that the Intel build refered to the Itanium vs. the
standard off-the-rack 64 bit version of Intel's 586/686 CPU)

Best regards,
Malcolm

 

[Tim Golden]

Thanks for everyone's feedback - excellent detail - all my questions
have been answered.

BTW: Roel was correct that I got confused over the AMD and Intel naming
conventions regarding the 64 bit versions of Python for Windows. (I
missed that nuance that the Intel build refered to the Itanium vs. the
standard off-the-rack 64 bit version of Intel's 586/686 CPU)

Frankly, I'm very glad you did get confused: I've learnt more
about the various 64-bit architectures in this thread than
I ever thought I'd need to know

TJG