MinGW and Python

[Edward Elliott]

At the same time, if the 14% slowdown is representative, then it's not
true that the compiler responsible for it "optimizes as well" as the
other; indeed, "does not optimize particularly well", under such a
hypothesis, would be far from a "beyond BS" assertion.

Maybe someone with access to Windows and some free time could compare
pybench compiled under msvc and mingw. The best way to test the compiler
is to eliminate all other variables.

 

[Edward Elliott]

about MacOSX, which also uses gcc: 14% faster pybench using Python 2.4.3

this is the second time I've seen that 14% figure. OOC, where does it come
from? the data sets you posted show an average 12.6% speedup. 14 is an
odd way to round.

I don't think it's very useful to talk about average speedups from a
benchmark of equally-weighted feature tests. the data shows wildly varying
differences in performance for each test. a real-world application could
be much slower or much faster on either platform depending on its feature
mix. not the type of thing that's amenable to expression as a single
value.

 

[Edward Elliott]

Great, then I tend to agree that there is no reason for building it
with mingw.

wouldn't the gcc/cygwin build of python rely on the cygwin dll? cygwin
implements the standard unix system calls on top of the windows platform.
iirc, some operations run much slower through cygwin than native windows
calls (file and terminal i/o come to mind).

if so, i do see good reason to build with mingw instead of gcc/cygwin.

 

[sturlamolden]

if so, i do see good reason to build with mingw instead of gcc/cygwin.

MinGW and Cygwin GCC is actually the same compiler. On Cygwin you can
remove the dependency on the cygwin dll by compiling with -mno-cygwin.
But as long as the cygwin dll is there, it creates an overhead for any
system call. The worst overhead is associated with the Unix fork()
system call, which Cygwin must emulate as there are no Win32
equivalent. In particular, a fork() on Unix will be optimized with
copy-on-write semantics in the kernel, but Cygwin cannot emulate that
as it has no access to the Windows kernel.

Another issue is that MinGW and Cygwin implements C++ exceptions with
setjmp/longjmp. Other verions of GCC (including DJGPP for DOS) does not
do this and as a result produces faster code for C++. I don't know if
there are any C++ in Python, but if there is, it will be an issue. it
is not an issue for plain C though.

Finally, MinGW (I am not sure about Cygwin) statically links the C++
standard library. VC++ and GCC on other platforms (e.g. Linux) does not
(normally) do this, and as a result the executables get a smaller
footprint.

 

[Edward Elliott]

MinGW and Cygwin GCC is actually the same compiler. On Cygwin you can
remove the dependency on the cygwin dll by compiling with -mno-cygwin.
But as long as the cygwin dll is there, it creates an overhead for any
system call.

Thanks for that very informative post! To clarify, mingw (aka gcc
-mno-cygwin) has no POSIX layer like cygwin. Because your post could also
be (incorrectly) interpreted to mean mingw removes the cygwin dll
dependency by just linking it in statically. But I googled and this is not
the case.

Back to the original point about whether mingw compilation is necessary
given the python compiles under cygwin: Yes, it is. There can't be an
apples-to-apples comparison between gcc and msvc-compiled python with the
cygwin POSIX layer in the way.

Couldn't you compile the msvc-python code under gcc/mingw? If the code
sticks to C standards, and mingw can link to native libs, it should work.
The only problem is if python relies on some msvc-specific weirdness under
msvc. While there's much weirdness in msvc, I would expect a
cross-platform app like python to stay away from it.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Actually, you answered me then too. I misunderstood after reading
http://sebsauvage.net/python/mingw.html.

Is the information on that page not correct? Has it never been?

It's not correct, to the best of my knowledge. However, since
very few people use it, support for mingw keeps breaking
(inadvertently), so there might be bugs that prevent it from
working *for you*.

At the point this was written (apparently around Python 2.2)
it was possible to build Python extensions with MingW (the page
actually explains you how to do that), so "Under Windows, if you
do not have the costly Microsoft Visual C++, you cannot
install Python extensions written in C" wasn't even true
at the time it was written. You couldn't use distutils, but
you could certainly have built extensions by invoking the
compiler manually.

There were times when cygwin couldn't link with .DLLs unless
they were created with GNU ld, but these times are long past
history.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Thanks for that very informative post! To clarify, mingw (aka gcc
-mno-cygwin) has no POSIX layer like cygwin. Because your post could also
be (incorrectly) interpreted to mean mingw removes the cygwin dll
dependency by just linking it in statically. But I googled and this is not
the case.

Well, you are not compiling with neither mingw, nor cygwin; you are
compiling with gcc in either case.

The question is what C library (and header files) you use: with cygwin,
you use cygwin1.dll (and its header files); with mingw, you use
msvcrt.dll (or some other MS CRT), along with the a GNU version of the
Microsoft header files.

Couldn't you compile the msvc-python code under gcc/mingw? If the code
sticks to C standards, and mingw can link to native libs, it should work.

Well, there is no native C library on Microsoft Windows: the system
simply doesn't include an official C library (I know there is crtdll.dll
and msvcrt.dll, but these aren't "endorsed" system C libraries).

The only problem is if python relies on some msvc-specific weirdness under
msvc. While there's much weirdness in msvc, I would expect a
cross-platform app like python to stay away from it.

For Windows, that would require not to use any of the standard C
functionality, since the system doesn't provide that functionality out
of the box. So Python itself never uses and msvcrt weirdness (well,
some, but that could be dropped easily) - yet still, the Python
binary will depend on a specific version of the MS CRT.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

In MSYS:

$ ./configure --prefix=/c/mingw
$ make
$ make install

This should be obvious to any with Unix experience.

MinGW actually distribute precompiled Python binaries as well (in
MSYS-DTK).

So how does that deal with the various extension modules that
PCbuild/readme.txt mentions? Do I get winsound.pyd? _msi.pyd
(in 2.5)?

Please believe me: there is currently no build process that
gives the same results as the build process used. It might
be fairly easy to create one, but none exists as of today.

MinGW can compile MFC. Download Windows Platform SDK and you get the
MFC source.

Did you actually try that? Last I tried, GCC would refuse to compile
the MinGW header files.

Even if you succeed, I don't think the SDK license will allow you
to distribute your binary, and it *will* be incompatible the official
MFC binary. MSVC++ has its own object layout, vtable layout, name
mangling, so you can't use GCC to link against MSVC-compiled
C++ libraries.

Regards,
Martin

 

[Ross Ridge]

MinGW and Cygwin GCC is actually the same compiler.

Not exactly. They're both GCC, but the MinGW compiler that you can
download from MinGW WWW site is a native Win32 appliction, while the
"MinGW" compiler included with Cygwin and invoked by "-mno-cygwin" is a
Cygwin application.

Ross Ridge

 

[Ross Ridge]

MinGW can compile MFC. Download Windows Platform SDK and you get the
MFC source.

Do not do this. The Platform SDK's EULA does not permit you to
redistribute anything you build from the MFC sources included in the
SDK. The only way to get a copy of MFC that you can legitimately use
in anything you distribute you need to buy a Microsoft C++ compiler.

Ross Ridge

 

[sturlamolden]

Couldn't you compile the msvc-python code under gcc/mingw?

Yes I could, but I cannot compile the code under msvc for comparison. I
only have MinGW. If build the mingw binary then someone else has to
build the msvc binary for comparison. Then we could do pybenches on the
same computer.

Uh .. I actually think it could be an EULA violation to publish mingw
vs. msvc benchmarks without permission from Microsoft. I don't want to
part of anything illegal or have M$ lawyers breathing down my back. If
we are going to do this, then we must do it properly and get the
permissions we need.

 

[Ross Ridge]

Well, there is no native C library on Microsoft Windows: the system
simply doesn't include an official C library (I know there is crtdll.dll
and msvcrt.dll, but these aren't "endorsed" system C libraries).

MSVCRT.DLL has been a standard system compent of Windows since at least
Windows 98. Many other system components depend on it. Essentially,
MSVCRT.DLL is an "undocumented" part of the Windows API. It's not
exactly "endorsed", Microsoft would rather you use it's current
compiler and runtime, but it is the standard "official" Windows system
C library.

Ross Ridge

 

[sjdevnull]

But as long as the cygwin dll is there, it creates an overhead for any
system call. The worst overhead is associated with the Unix fork()
system call, which Cygwin must emulate as there are no Win32
equivalent. In particular, a fork() on Unix will be optimized with
copy-on-write semantics in the kernel, but Cygwin cannot emulate that
as it has no access to the Windows kernel.

Cygwin doesn't do COW fork() for historical reasons. Windows 95/98
didn't expose support for such a thing, but NT, XP, etc do (pass NULL
as the SectionHandle parameter to ZwCreateProcess or the older
NtCreateProcess to do a COW duplicate of the calling process).

Cygwin at one point had a fork() implementation using this in the
development tree (or being discussed on the mailing lists) but decided
not to use it since it wasn't much faster than full-copy fork for some
reason, and it would've split the code for NT/95.

 

[Edward Elliott]

Yes I could, but I cannot compile the code under msvc for comparison. I
only have MinGW. If build the mingw binary then someone else has to
build the msvc binary for comparison. Then we could do pybenches on the
same computer.

Sorry, I didn't mean you personally, I meant "you" in the general sense.

Uh .. I actually think it could be an EULA violation to publish mingw
vs. msvc benchmarks without permission from Microsoft. I don't want to
part of anything illegal or have M$ lawyers breathing down my back. If
we are going to do this, then we must do it properly and get the
permissions we need.

Why, because of a "no benchmarks" clause in the EULA? First, I'm fairly
certain that clause is uneforceable. Second, it's not worth MS's time to
come after you. Third, even if they did, they'd have an incredibly tough
time showing that any harm occured from your actions, leaving them with no
recovery. Fourth, if it somehow came to that, you'd have organizations
like the EFF lining up at your door to take the case gratis. I'm assuming
US law here.

Not saying you should do it, just that the risk in my opinion is extremely
low. I am a law student, but this is not legal advice.

 

[Edward Elliott]

Well, you are not compiling with neither mingw, nor cygwin; you are
compiling with gcc in either case.

touche, mr. pedant.

Well, there is no native C library on Microsoft Windows: the system
simply doesn't include an official C library (I know there is crtdll.dll
and msvcrt.dll, but these aren't "endorsed" system C libraries).

don't know what you mean by "endorsed". does it lack features of the C89
ANSI standard?

For Windows, that would require not to use any of the standard C
functionality, since the system doesn't provide that functionality out
of the box.

That would be a problem then. So what happens when you compile python with
msvc, and why can't mingw just replicate that?

 

[Edward Elliott]

MSVCRT.DLL ... It's not
exactly "endorsed", Microsoft would rather you use it's current
compiler and runtime, but it is the standard "official" Windows system
C library.

Does it comply with the ANSI C89 standard? I'm still not seeing why mingw
can't just link python to it.

 

[sturlamolden]

Sorry, I didn't mean you personally, I meant "you" in the general sense.

OK

I've just tried to build Python 2.4.3 with MinGW (MSYS 1.0.10, GCC
3.4.2):

$ ./configure --prefix=/c/Python243-mingw
$ make

The build then failed on posixmodule.c.


Sturla Molden


../Modules/posixmodule.c:308: warning: function declaration isn't a
prototype
../Modules/posixmodule.c: In function `posix_ttyname':
../Modules/posixmodule.c:1162: warning: implicit declaration of function
`ttyname'
../Modules/posixmodule.c:1162: warning: assignment makes pointer from
integer without a cast
../Modules/posixmodule.c: In function `posix_mkdir':
../Modules/posixmodule.c:1791: error: too many arguments to function
`mkdir'
../Modules/posixmodule.c: In function `posix_execv':
../Modules/posixmodule.c:2199: warning: passing arg 2 of `execv' from
incompatible pointer type
../Modules/posixmodule.c: In function `posix_execve':
../Modules/posixmodule.c:2332: warning: passing arg 2 of `execve' from
incompatible pointer type
../Modules/posixmodule.c:2332: warning: passing arg 3 of `execve' from
incompatible pointer type
../Modules/posixmodule.c: In function `posix_fork':
../Modules/posixmodule.c:2847: warning: implicit declaration of function
`fork'
../Modules/posixmodule.c: In function `posix_openpty':
../Modules/posixmodule.c:2909: error: `O_NOCTTY' undeclared (first use
in this function)
../Modules/posixmodule.c:2909: error: (Each undeclared identifier is
reported only once
../Modules/posixmodule.c:2909: error: for each function it appears in.)
../Modules/posixmodule.c:2912: error: `SIGCHLD' undeclared (first use in
this function)
../Modules/posixmodule.c:2914: warning: implicit declaration of function
`grantpt'
../Modules/posixmodule.c:2919: warning: implicit declaration of function
`unlockpt'
../Modules/posixmodule.c:2924: warning: implicit declaration of function
`ptsname'
../Modules/posixmodule.c:2924: warning: assignment makes pointer from
integer without a cast
../Modules/posixmodule.c:2931: warning: implicit declaration of function
`ioctl'
../Modules/posixmodule.c:2931: error: `I_PUSH' undeclared (first use in
this function)
../Modules/posixmodule.c: In function `posix_getegid':
../Modules/posixmodule.c:2973: warning: implicit declaration of function
`getegid'
../Modules/posixmodule.c: In function `posix_geteuid':
../Modules/posixmodule.c:2986: warning: implicit declaration of function
`geteuid'
../Modules/posixmodule.c: In function `posix_getgid':
../Modules/posixmodule.c:2999: warning: implicit declaration of function
`getgid'
../Modules/posixmodule.c: In function `posix_getppid':
../Modules/posixmodule.c:3121: warning: implicit declaration of function
`getppid'
../Modules/posixmodule.c: In function `posix_getuid':
../Modules/posixmodule.c:3163: warning: implicit declaration of function
`getuid'
../Modules/posixmodule.c: In function `posix_kill':
../Modules/posixmodule.c:3193: warning: implicit declaration of function
`kill'
../Modules/posixmodule.c: In function `posix_wait':
../Modules/posixmodule.c:4970: warning: implicit declaration of function
`wait'
../Modules/posixmodule.c: In function `posix_pipe':
../Modules/posixmodule.c:5511: warning: implicit declaration of function
`pipe'
../Modules/posixmodule.c: At top level:
../Modules/posixmodule.c:382: warning: 'posix_error_with_filename'
defined but not used
../Modules/posixmodule.c:534: warning: 'posix_fildes' defined but not
used
../Modules/posixmodule.c:6223: warning: 'conv_confname' defined but not
used
../Modules/posixmodule.c:7126: warning: 'setup_confname_table' defined
but not used
make: *** [Modules/posixmodule.o] Error 1

 

[sturlamolden]

Please believe me: there is currently no build process that
gives the same results as the build process used. It might
be fairly easy to create one, but none exists as of today.

I tried to build with MinGW this eveing and it failed. I believe you We
need to make a build process for MinGW.

 

[Neil Hodgson]

sturlamolden:

Uh .. I actually think it could be an EULA violation to publish mingw
vs. msvc benchmarks without permission from Microsoft. I don't want to
part of anything illegal or have M$ lawyers breathing down my back. If
we are going to do this, then we must do it properly and get the
permissions we need.

The EULA attempted to stop disclosing benchmarks of .NET code, not
C++ code. The most recent version of the EULA has replaced that with
permission to disclose benchmarks of .NET code provided some conditions
are met: full disclosure of benchmark code and environment; latest
release of .NET; compliance with 'best practices'. Its still stupid as
it would be in Microsoft's interest to encourage benchmarking .NET but
not quite as stupid as before.

Neil

 

[Robert Kern]

Does it comply with the ANSI C89 standard? I'm still not seeing why mingw
can't just link python to it.

It can.

--
Robert Kern
(e-mail address removed)

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco