Python Extension Building Network

[kyosohma]

Hi,

I am trying to get a small group of volunteers together to create
Windows binaries for any Python extension developer that needs them,
much like the package/extension builders who volunteer their time to
create Linux RPMs.

The main thing I need are people willing to test the binaries to make
sure the extension is stable. This would require installing the binary
and probably downloading the source too to get the developer's test
code. I've been able to get some of the tests to run great while
others are pretty finicky and some extensions don't come with tests.
It would be nice to know which extensions are most in need of this
too.

While I can create the binaries on my own for a while, if I get too
many requests, there will be a backlog, so it would be nice to have
help with that too. I'm also looking for knowledgeable people to be
sounding boards (i.e. give advice).

Developers: all I would require is a request, a link to the source,
and a well-written setup.py file for a cross-platform extension.

You can find the few that I've already done here:http://
www.pythonlibrary.org/python_modules.htm

I have also posted a way to create the binaries using the MinGW
compiler. I have VS2003 installed on my PC and MinGW is installed in a
VM, so I can compile the extensions both ways.

Thanks in advance for any feedback.

Mike

 

[Tim Golden]

Hi,

I am trying to get a small group of volunteers together to create
Windows binaries for any Python extension developer that needs them,
much like the package/extension builders who volunteer their time to
create Linux RPMs.

The main thing I need are people willing to test the binaries to make
sure the extension is stable. This would require installing the binary
and probably downloading the source too to get the developer's test
code. I've been able to get some of the tests to run great while
others are pretty finicky and some extensions don't come with tests.
It would be nice to know which extensions are most in need of this
too.

While I can create the binaries on my own for a while, if I get too
many requests, there will be a backlog, so it would be nice to have
help with that too. I'm also looking for knowledgeable people to be
sounding boards (i.e. give advice).

Developers: all I would require is a request, a link to the source,
and a well-written setup.py file for a cross-platform extension.

You can find the few that I've already done here:http://
www.pythonlibrary.org/python_modules.htm

I have also posted a way to create the binaries using the MinGW
compiler. I have VS2003 installed on my PC and MinGW is installed in a
VM, so I can compile the extensions both ways.

Mike, this is great news. Whenever I have time <laughs, but
means it sincerely> I'll try to run through some of the modules
you've compiled.

As a slight aside, the main problem I've found when I've tried
to build extensions (and I've been doing it recently with AVBin and
Pyglet) is that Windows just doesn't have the build environment, the
directory structures, the env vars and all that that a ./configure or
even a python setup.py install sometimes expects. eg if I were to
offer to build a MySQL extension (as someone who doesn't use MySQL
and wouldn't have the source libs installed if I did) there would
be a fair bit of pain to go through. You've obviously gone through
that pain barrier for at least some of the extensions on the modules
page. Was it tough?

TJG

(PS SendKeys link on this page is dead:
http://www.pythonlibrary.org/automation.htm)

 

[kyosohma]

Mike, this is great news. Whenever I have time <laughs, but
means it sincerely> I'll try to run through some of the modules
you've compiled.

As a slight aside, the main problem I've found when I've tried
to build extensions (and I've been doing it recently with AVBin and
Pyglet) is that Windows just doesn't have the build environment, the
directory structures, the env vars and all that that a ./configure or
even a python setup.py install sometimes expects. eg if I were to
offer to build a MySQL extension (as someone who doesn't use MySQL
and wouldn't have the source libs installed if I did) there would
be a fair bit of pain to go through. You've obviously gone through
that pain barrier for at least some of the extensions on the modules
page. Was it tough?

The hardest part was finding accurate information. Most people on the
user groups have been unhelpful or sarcastic. I had better luck
contacting developers directly who had already created Windows
binaries. They didn't mind giving me some pointers.

The directions for MinGW were usually only partially correct. So I
went through the two sets of directions I found (links on the site)
and mixed and matched until I got it right.

There are no directions on how to use Visual Studio 2003 that I've
found, just some old free edition. those directions were incompatible
with VS2003. I'll post VS2003's correct usage eventually, but it's
basically just installing it and then using distutils.

TJG

(PS SendKeys link on this page is dead:http://www.pythonlibrary.org/automation.htm)

I've noticed some of the stuff I thought I uploaded seems to have gone
MIA. I'll get that fixed tonight. Thanks for the bug report and offer
of help.

Mike

 

[Tim Golden]

The hardest part was finding accurate information. Most people on the
user groups have been unhelpful or sarcastic.

That's a shame to hear. Because you were building on Windows?
Or for some other reason? (I ask because, even here on the
Python lists, reactions like "Get a working O/S" are not unknown
in answer to questions like "How do I... on Windows?")

The directions for MinGW were usually only partially correct.

The gripe I've had MingW -- which is obviously tempered by the
fact of its existence and the huge amount of effort which has
gone into it -- is the difficulty of finding a version of all
the tools which pleases everyone. And/or of knowing whether it's
safe to mix "Stable", "Candidate" etc. release packages.

TJG

 

[kyosohma]

That's a shame to hear. Because you were building on Windows?
Or for some other reason? (I ask because, even here on the
Python lists, reactions like "Get a working O/S" are not unknown
in answer to questions like "How do I... on Windows?")

I don't think it was because of Windows, but because I was asking
about how to use Visual Studio. I've had classes in it, but intro
classes in Comp Sci don't teach you how to compile. One of the people
on this list told me to go read Microsoft's docs.

Well, those docs are uniformly unhelpful until you actually know what
you're doing. And they were useless since the actual way to use the
compiler was to use the python command:

python setup.py bdist_wininst

Which of course won't be found in any docs produced from the venerable
Microsoft.

The gripe I've had MingW -- which is obviously tempered by the
fact of its existence and the huge amount of effort which has
gone into it -- is the difficulty of finding a version of all
the tools which pleases everyone. And/or of knowing whether it's
safe to mix "Stable", "Candidate" etc. release packages.

TJG

I used Candidate. At some point, I'll have to try uninstalling MinGW
and try Stable. Mixing them sounds interesting too.

I'm no expert in either one yet, but I hope to be soon.

Mike

 

[John Nagle]

Hi,

I am trying to get a small group of volunteers together to create
Windows binaries for any Python extension developer that needs them,
much like the package/extension builders who volunteer their time to
create Linux RPMs.

Really good idea.

Can you get some corporate support? It would be good to have
some organization behind this. Binaries are a security issue;
you need an organization or a reputation to distribute binaries.

John Nagle

 

[kyosohma]

Really good idea.

Can you get some corporate support? It would be good to have
some organization behind this. Binaries are a security issue;
you need an organization or a reputation to distribute binaries.

John Nagle

Right now all I have is Steve Holden's backing (and now Golden's too).
If you have some suggestions beyond Python luminaries, let me know.

Mike

 

[kyosohma]

Really good idea.

Can you get some corporate support? It would be good to have
some organization behind this. Binaries are a security issue;
you need an organization or a reputation to distribute binaries.

John Nagle

I forgot to ask, but what would that look like? Some kind of message
like "these binaries are backed by the Blah Blah Organization" ? I
can't get a reputation until I start doing it...

Mike

 

[M.-A. Lemburg]

Are you aware of the repository that ActiveState created for
its version of Python (ActivePython) ? It comes with a
set of pre-compiled Python extensions (PPMs) and an easy
to use installer.

Perhaps getting ActiveState to open up the repo would be good
idea - something like Ubuntu does with the universe repo.

The hardest part was finding accurate information. Most people on the
user groups have been unhelpful or sarcastic. I had better luck
contacting developers directly who had already created Windows
binaries. They didn't mind giving me some pointers.

Interesting: Python seems to be "growing up" in all kinds of
ways ...

The directions for MinGW were usually only partially correct. So I
went through the two sets of directions I found (links on the site)
and mixed and matched until I got it right.

There are no directions on how to use Visual Studio 2003 that I've
found, just some old free edition. those directions were incompatible
with VS2003. I'll post VS2003's correct usage eventually, but it's
basically just installing it and then using distutils.

Getting VS2003 ready to compile Python extensions is really easy:

1. open a command shell
2. run vcvars32.bat
3. make sure the Python version you are targetting is on the
PATH
4. "python setup.py bdist_wininst" or "python setup.py bdist_msi"
5. pick up the installer in the build\ directory.

Note: bdist_msi is only available in Python 2.5 and later.

You need VC6 if you want to compile extensions for Python 1.5-2.3
and VC7.1 for Python 2.4 and later.

I've noticed some of the stuff I thought I uploaded seems to have gone
MIA. I'll get that fixed tonight. Thanks for the bug report and offer
of help.

Mike

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Nov 09 2007)________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611

 

[kyosohma]

Are you aware of the repository that ActiveState created for
its version of Python (ActivePython) ? It comes with a
set of pre-compiled Python extensions (PPMs) and an easy
to use installer.

Perhaps getting ActiveState to open up the repo would be good
idea - something like Ubuntu does with the universe repo.





Interesting: Python seems to be "growing up" in all kinds of
ways ...



Getting VS2003 ready to compile Python extensions is really easy:

1. open a command shell
2. run vcvars32.bat
3. make sure the Python version you are targetting is on the
PATH
4. "python setup.py bdist_wininst" or "python setup.py bdist_msi"
5. pick up the installer in the build\ directory.

I didn't need to run vcvars32.bat to make mine work. But that's good
to know...I think.

Note: bdist_msi is only available in Python 2.5 and later.

You need VC6 if you want to compile extensions for Python 1.5-2.3
and VC7.1 for Python 2.4 and later.

I was aware of that you needed VC6 for 2.3, but I didn't realize it
went that far back. And I knew you needed 7.1 for 2.4 and 2.5, but
I've heard that they're moving to VS2005 soon.

Thanks for the feedback, Marc-Andre!

Mike

 

[Gabriel Genellina]

I don't think it was because of Windows, but because I was asking
about how to use Visual Studio. I've had classes in it, but intro
classes in Comp Sci don't teach you how to compile. One of the people
on this list told me to go read Microsoft's docs.

Was that me? In this message?
<http://groups.google.com/group/comp...read/thread/da5e45553acccb9d/82b376b6cca35acf>
(Note that you were asking "how do I use VS to compile an exe?", not "how
do I build a Python extension?")
I'm sorry if you feel the response was somewhat rude, I apologize in that
case. It was not intended at all.

 

[kyosohma]

Was that me? In this message?
<http://groups.google.com/group/comp.lang.python/browse_thread/thread/...>
(Note that you were asking "how do I use VS to compile an exe?", not "how
do I build a Python extension?")
I'm sorry if you feel the response was somewhat rude, I apologize in that
case. It was not intended at all.

I am at fault as well. What I thought was clear obviously wasn't to
you. I almost always like your answers, so when you said that, it
seemed pretty snippy to me. I try to be as unambiguous and clear as I
can be when posting, but I sometimes lapse in this regard. Sorry!

Mike

 

[M.-A. Lemburg]

I didn't need to run vcvars32.bat to make mine work. But that's good
to know...I think.



I was aware of that you needed VC6 for 2.3, but I didn't realize it
went that far back. And I knew you needed 7.1 for 2.4 and 2.5, but
I've heard that they're moving to VS2005 soon.

AFAIK, Martin (who's building the Windows installers for Python)
is going to skip VS2005 and go straight for VS2008, but I may be
wrong.

Thanks for the feedback, Marc-Andre!

You're welcome

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Nov 10 2007)________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611

 

[kyosohma]

AFAIK, Martin (who's building the Windows installers for Python)
is going to skip VS2005 and go straight for VS2008, but I may be
wrong.


You're welcome

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Nov 10 2007)>>> Python/Zope Consulting and Support ... http://www.egenix.com/

________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611

Is it okay if I take your instructions for VS2003 and use them on my
website? I'll modify them slightly, but it'll be mostly the same.

Mike

 

[Yu-Xi Lim]

Hi,

I am trying to get a small group of volunteers together to create
Windows binaries for any Python extension developer that needs them,
much like the package/extension builders who volunteer their time to
create Linux RPMs.

Mike

It's not entirely clear to me what you're compiling. Most of the modules
you have there do not have native code. I'm guessing your main focus is
building .exe installers for Windows users.

If it's just installers of pure Python code, why the extra effort of
using both VS and MingW? AFAIK, they both yield functionally identical
results---the exact same .py files get installed.

I'm also not sure how big a task this is or the issues involved, but
here's my take based on what I've read:

Throwing more manpower at the task is not the solution. The actual
process can be and SHOULD BE highly automated. All that's needed are a
proper distutils script, and proper tests. Both responsibilities should
fall on the shoulders of the module authors, though I guess experienced
volunteers can help out with the former.

If the module is pure Python, there should be little need for testing
the installer specifically on Windows. If there's a failure, then the
module itself is buggy (e.g. making platform-specific assumptions) or
there's similar problem with the install script.

For a large number of modules on PyPI, building the installer is trivial
(assuming pure Python, and a setup script). Compiling them can be easily
automated: 1) check PyPI for updates 2) download the latest update 3)
build the .exe 3) upload to a website. All that remains is the need for
some hardware (a build farm) and the occasional manual intervention.

If I'm not mistaken, distutils can build Windows installers on other
platforms. Maybe you could set up a Linux LiveCD with the necessary
tools installed and distribute that. Module authors who want to build
Windows installers can easily use that CD. Volunteers wanting to help
can do so easily without having to repeat the task of setting up the
toolchain separately.


I've done some work as a port maintainer (aka package maintainer) for
FreeBSD. The BSD port system is somewhat closer to PEAK's Easy Install,
however binary packages (similar to RPMs or DEBs) are also available.
The port maintainer's job is solely to ensure that the Makefile (build
script) is working and up to date. The actual testing of the code is
done by the authors of the software itself and the port maintainer only
ensures that the install script works right. Binary packages are built
by an automated system using a cluster of computers for various
architectures and supported OS versions. Errors during builds are sent
to to port maintainers.

A similar system is used for most Linux distributions with a central
repository for packages.

 

[kyosohma]

It's not entirely clear to me what you're compiling. Most of the modules
you have there do not have native code. I'm guessing your main focus is
building .exe installers for Windows users.

The main objective is to make it easier for Windows users to install
the modules, so that's what the .exe is for. Admittedly developers
should know how to install from source, but the newbs don't always
know how and I've had some trouble with some of the more complex ones
myself.

If it's just installers of pure Python code, why the extra effort of
using both VS and MingW? AFAIK, they both yield functionally identical
results---the exact same .py files get installed.

I am using both because Steve Holden asked me to try MinGW. I was just
going to use Visual Studio. However, I thought it might be a good idea
to try it both ways to make sure it could be done in a completely open
source environment. I'm pretty sure you can use MinGW on impure Python
extensions too, although it takes more work.

I'm also not sure how big a task this is or the issues involved, but
here's my take based on what I've read:

Throwing more manpower at the task is not the solution. The actual
process can be and SHOULD BE highly automated. All that's needed are a
proper distutils script, and proper tests. Both responsibilities should
fall on the shoulders of the module authors, though I guess experienced
volunteers can help out with the former.

I would like to do this and I was trying to come up with a way to do
just that. For some reason, when I mentioned that idea to one of my
more knowledgeable contacts, he didn't see the need for it.

If the module is pure Python, there should be little need for testing
the installer specifically on Windows. If there's a failure, then the
module itself is buggy (e.g. making platform-specific assumptions) or
there's similar problem with the install script.

For a large number of modules on PyPI, building the installer is trivial
(assuming pure Python, and a setup script). Compiling them can be easily
automated: 1) check PyPI for updates 2) download the latest update 3)
build the .exe 3) upload to a website. All that remains is the need for
some hardware (a build farm) and the occasional manual intervention.

If I'm not mistaken, distutils can build Windows installers on other
platforms. Maybe you could set up a Linux LiveCD with the necessary
tools installed and distribute that. Module authors who want to build
Windows installers can easily use that CD. Volunteers wanting to help
can do so easily without having to repeat the task of setting up the
toolchain separately.

I've been told that distutils can build Windows installers on other
platforms, but I think that may apply only to pure Python extensions
only.

I've done some work as a port maintainer (aka package maintainer) for
FreeBSD. The BSD port system is somewhat closer to PEAK's Easy Install,
however binary packages (similar to RPMs or DEBs) are also available.
The port maintainer's job is solely to ensure that the Makefile (build
script) is working and up to date. The actual testing of the code is
done by the authors of the software itself and the port maintainer only
ensures that the install script works right. Binary packages are built
by an automated system using a cluster of computers for various
architectures and supported OS versions. Errors during builds are sent
to to port maintainers.

A similar system is used for most Linux distributions with a central
repository for packages.

I like this idea quite a bit. It's nice to know that other people are
thinking along the same lines as I am. However, I am not sure how to
do this myself. I assume when you mean by creating a Linux Live CD
with the proper tools, you mean that it should include MinGW and
Python. I suppose the only problem with that is the dependency issue.

There's a number of modules that require arbitrary modules for the
setup.py file to run. Mechanize is one such module as it require
ClientForm. Some of the others I've done required various versions of
Easy Setup or ElementTree. I'm not sure how this is resolved unless
you can just download these items during the Live CD session and
install them to memory. However, I don't think this is currently
possible, correct?

I'll float your ideas by Holden and see what he thinks though. Thanks
for the ideas.

Mike