Problem with msvcrt60 vs. msvcr71 vs. strdup/free

[Gerhard Haering]

Hello,

I used to build Python extension modules with mingw. Now, Python has
switched to the MSVCR71 runtime with version 2.4, and I thought mingw
has support for this. But I get problems with symbols being referenced
from the wrong DLLs.

You can see the problem by compiling this:

##################
#include <string.h>

int main()
{
char* s;
int i;


for (i = 0; i < 10; i++) {
s = strdup("foo");
free(s);
}

return 0;
}
##################

with gcc x.c -lmsvcr71

Then if you run a.exe it crashes.

If you use depends.exe on it, you see that it resolves strdup() via
msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
the one malloc, but free() a different free() from the other DLL,
which is undoubtedly the reason for the crash.

Is there any way I can force mingw to not link in msvcr for things
like strdup?

I think I found a very strange and ugly workaround: if I use _strdup()
instead of strdup(), it is resolved using the msvcr71 DLL.

I hope it is (soon) possible to use only the msvcr71 runtime from the
mingw compiler.

-- Gerhard
--
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFByCmWdIO4ozGCH14RAu0nAJwKYhMLulwl52UltcFk00sy55yEpQCfaTsS
Ba8Q5W8g23lRe/VM/K/kBCU=
=L8+w
-----END PGP SIGNATURE-----

 

[Thomas Heller]

Hello,

I used to build Python extension modules with mingw. Now, Python has
switched to the MSVCR71 runtime with version 2.4, and I thought mingw
has support for this. But I get problems with symbols being referenced
from the wrong DLLs.

You can see the problem by compiling this:

##################
#include <string.h>

int main()
{
char* s;
int i;


for (i = 0; i < 10; i++) {
s = strdup("foo");
free(s);
}

return 0;
}
##################

with gcc x.c -lmsvcr71

Then if you run a.exe it crashes.

If you use depends.exe on it, you see that it resolves strdup() via
msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
the one malloc, but free() a different free() from the other DLL,
which is undoubtedly the reason for the crash.

Is there any way I can force mingw to not link in msvcr for things
like strdup?

Only guesswork, but replacing this section in the
lib\gcc-lib\mingw32\3.2.3\specs file (I can only guess wht this file does)

*libgcc:
%{mthreads:-lmingwthrd} -lmingw32 -lgcc -lmoldname -lmingwex -lmsvcrt

with this one:

*libgcc:
%{mthreads:-lmingwthrd} -lmingw32 -lgcc -lmoldname -lmingwex -lmsvcr71

seemed to do the trick.

Thomas

 

[abkhd]

From: "" <[email protected]>
Newsgroups: comp.lang.python
Subject: Re: Problem with msvcrt60 vs. msvcr71 vs. strdup/free
Date: Wed, 22 Dec 2004 11:07:02 -0800

Hello,

I used to build Python extension modules with mingw. Now, Python has
switched to the MSVCR71 runtime with version 2.4, and I thought mingw
has support for this. But I get problems with symbols being referenced
from the wrong DLLs.

You can see the problem by compiling this:

##################
#include <string.h>

int main()
{
char* s;
int i;


for (i = 0; i < 10; i++) {
s = strdup("foo");
free(s);
}

return 0;
}
##################

with gcc x.c -lmsvcr71

Then if you run a.exe it crashes.

If you use depends.exe on it, you see that it resolves strdup() via
msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
the one malloc, but free() a different free() from the other DLL,
which is undoubtedly the reason for the crash.

Is there any way I can force mingw to not link in msvcr for things
like strdup?

Hello


Believe it or not I do not get a crash here.

Using objdump on a.exe I get:

[CUT]

00005014 00005084 00000000 00000000 000052b8 00005100

DLL Name: msvcrt.dll
vma: Hint/Ord Member-Name Bound-To
51b4 81 _strdup

00005028 00005090 00000000 00000000 000052f8 0000510c

DLL Name: msvcr71.dll
vma: Hint/Ord Member-Name Bound-To
51c0 59 __getmainargs
51d0 82 __p__environ
51e0 105 __set_app_type
51f4 154 _cexit
5200 211 _fileno
520c 222 _fmode
5218 225 _fpreset
5224 273 _iob
522c 390 _onexit
5238 435 _setmode
5244 599 atexit
5250 634 free
5258 715 signal

0000503c 000050cc 00000000 00000000 00005314 00005148

DLL Name: msvcrt.dll
vma: Hint/Ord Member-Name Bound-To
5264 510 abort
526c 537 fflush
5278 546 fprintf
5284 603 malloc

00005050 00000000 00000000 00000000 00000000 00000000


[CUT]


Having said that, let me add that I know where you are coming from. And
even though your a.exe did not crash with me now, I know that other
dlls and programs, in my case, did.

And let it be known that unfortunately the measures suggested by the
good people here do not always work. Changing the entry in the specs
file works most of the time, but not on mmap.pyd (of the official
Python source distribution) for example, where a call to some _lseek of
some sort (I forgot the exact call) is still linked from msvcrt.dll not
from msvcr71. And yes the test_mmap crashes if I compile it that way.

Also changing the specs file seems to work only when the linking is
simple. But if the linking gets complex (as in linking many objects
files to build, say, pythonXY.dll) then even the specs change will not
help. To illustrate, and in trying to get the pyMinGW built Python to
link with msvcr71 that way, one ends up with a pythonXY.dll that
references both runtime libraries: msvcrt, and msvcr71 with strange
consequences. This particular version of Python crashes on site import
hallucinating about (and I paraphrase!) "Null result from PY...XXX
without Error."

On MSDN, it is stated: "If your DLLs pass CRT resources across the
msvcrt.dll and msvcr71.dll boundary, you will encounter issues with
mismatched CRTs and need to recompile your project with Visual C++
..NET." [1]

I have a gut feeling that "DLLs" here does not mean inter-dll passing
of resources, but intra-dll passing of resources. In other words that
if the self-same DLL uses both run-times then you are in trouble; but I
don't think that this relates to the situation where one DLL is using
msvcrt and another DLL is using msvcr71. Please bear with me.

In this light, I truly don't know why would we try, or recommend
others, to link with libmsvcr71.a, if some mix-up is still possible. I
have yet to see a crash occur when Python24.dll or an extension of the
same uses the old way of linking with libmsvcrt.a. That way we are
assured that no references are made to msvcr71 when msvcrt is the one
used when linking.

I could be mistaken, but I think people are confusing the behavior of
the MS compilers with MinGW's. The old MS compiler (V. 6) uses
msvcrt.lib by default, and even when the newer MSVS uses the same name
for the its default runtime, that msvcrt link implicitly means the
newer msvcr71 [2]. This behind the scenes behavior is what seems to
have people mixed up. You see if one has older projects compiled using
V. 6, then linking these object files to other object files compiled
with the newer MSVS could effectively mean that the resulting DLL will
have references to both run-times, something by the way is also
mentioned in the MSDN resources [3].

But as far as I know this is not the case with regards to MinGW. MinGW
explicitly uses msvcrt, and so old and newer projects should in theory
have no problems if they stick to msvcrt, as opposed to going the MS
compilers' path-- shooting ourselves in the foot-- and risking linking
to two run-times at the same time.

I have tried changing the specs, and changing the location of msvcr71
and the results are never 100% re-assuring. And so I have resorted to
linking with the good old msvcrt and I have not had a crash due to this
to date. Of course this is my humble experience and I could be mistaken
big time.

For what it is worth, however, my extension DLLs are clean in that they
reference msvcrt only and not msvcr71, and hence all the malloc, and
free and what have you are never shared with any other runtime. My
point being that lacking empirical evidence to the contrary, I think
this is how making extensions to Python 2.4 with MinGW should be: using
the default runtime libmsvcrt.a. It would be fantastic if the good
people at MinGW, or others, can get that hard-wiring issue of the
msvcrt resolved. But until that happens I think it is safer to extend
Python 2.4 the old way. It seems to work fine.



Regards,
Khalid






[1] C-RunTime Libraries from MSDN:
http://tinyurl.com/q20a

[2] Ibid.

[3] Ibid.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Believe it or not I do not get a crash here.

I can believe this. strdup does malloc, i.e. gets some
memory from the heap, and free puts it back into the heap.
Since this is a different CRT, it puts it back into
a different heap. This is a leak, but it should not cause
a crash.

I have a gut feeling that "DLLs" here does not mean inter-dll passing
of resources, but intra-dll passing of resources. In other words that
if the self-same DLL uses both run-times then you are in trouble; but I
don't think that this relates to the situation where one DLL is using
msvcrt and another DLL is using msvcr71. Please bear with me.

I think you are wrong. What DLL invokes the CRT function does not matter
- that the CRT functions come from different DLLs matters. Each CRT
is loaded exactly once into an address space, regardless of how many
DLLs refer to it. However, different CRTs have different sets of global
variables, and this is what hurts.

E.g. malloc use a pointer to the free list in the heap, and so does
free. Each CRT has its own heap, and each has its own pointer to the
free list. Then, doing malloc in one CRT and free in the other does
not cause crashes (perhaps unless you use the debug CRT, which tries
to determine whether it owns the pointer being freed). The CRT will
put the memory into its own free list, and allocate it again if its
own malloc asks for it. However, if you repeatedly do malloc() in one
CRT and free() in another, you get a memory leak, since malloc() will
not find the memory that has been freed.

In this light, I truly don't know why would we try, or recommend
others, to link with libmsvcr71.a, if some mix-up is still possible. I
have yet to see a crash occur when Python24.dll or an extension of the
same uses the old way of linking with libmsvcrt.a. That way we are
assured that no references are made to msvcr71 when msvcrt is the one
used when linking.

This is very easy to create. Just do fopen() in an extension linked
with msvcrt.dll, then do PyRun_AnyFile. This will crash, because fread()
on a file opened in a different CRT will always crash.

I could be mistaken, but I think people are confusing the behavior of
the MS compilers with MinGW's. The old MS compiler (V. 6) uses
msvcrt.lib by default, and even when the newer MSVS uses the same name
for the its default runtime, that msvcrt link implicitly means the
newer msvcr71 [2].

You are mistaken. The name of the import library is, and always was,
msvcrt.lib. However, different versions of msvcrt.lib refer to different
DLLs: msvcrt40.dll, msvcr70.dll, and msvcr71.dll.

> This behind the scenes behavior is what seems to
have people mixed up. You see if one has older projects compiled using
V. 6, then linking these object files to other object files compiled
with the newer MSVS could effectively mean that the resulting DLL will
have references to both run-times, something by the way is also
mentioned in the MSDN resources [3].

I think you are misreading [3]: It explicitly says

"If you have a .lib or .obj file that needs to link to msvcrt.lib, then
you should *not* have to recompile it to work with the new msvcrt.lib in
Visual C++ .NET." (emphasis mine)

The only case where you can link a single DLL with different CRTs using
MS tools is when you combine different versions of the CRTs, e.g. debug
and non-debug, or DLL and non-DLL.

But as far as I know this is not the case with regards to MinGW. MinGW
explicitly uses msvcrt, and so old and newer projects should in theory
have no problems if they stick to msvcrt, as opposed to going the MS
compilers' path-- shooting ourselves in the foot-- and risking linking
to two run-times at the same time.

Python extensions should never ever link with msvcrt.dll. As your
reference says:

'The msvcrt.dll is now a "known DLL," meaning that it is a system
component owned and built by Windows. It is intended for future use only
by system-level components. An application should use and redistribute
msvcr71.dll'

Now, a Python extension is released from the need to distribute
msvcr71.dll, since Python already does that. It should still link
with that DLL.

I have tried changing the specs, and changing the location of msvcr71
and the results are never 100% re-assuring. And so I have resorted to
linking with the good old msvcrt and I have not had a crash due to this
to date. Of course this is my humble experience and I could be mistaken
big time.

You are.

For what it is worth, however, my extension DLLs are clean in that they
reference msvcrt only and not msvcr71, and hence all the malloc, and
free and what have you are never shared with any other runtime.

How do you know? What if your extension returns a pointer malloc'ed by
msvcrt.dll, and returns it to python24.dll, and then python24.dll
frees the pointer, using free() from msvcr71.dll?

My
point being that lacking empirical evidence to the contrary, I think
this is how making extensions to Python 2.4 with MinGW should be: using
the default runtime libmsvcrt.a. It would be fantastic if the good
people at MinGW, or others, can get that hard-wiring issue of the
msvcrt resolved. But until that happens I think it is safer to extend
Python 2.4 the old way. It seems to work fine.

Indeed, it *will* work most of the time, as you rarely pass resources
across CRTs in a typical Python extension. However, it is very hard
to tell from the source code of the extension if such resource passing
could ever happen (e.g. what about setlocale(), atexit(), malloc(),
....), so if you need something better than "seems to work", you really
need to make sure that you understand all the issues, and have a build
process that resolves them all.

Regards,
Martin

 

[abkhd]

Indeed, it *will* work most of the time, as you rarely pass resources
across CRTs in a typical Python extension. However, it is very hard
to tell from the source code of the extension if such resource passing
could ever happen (e.g. what about setlocale(), atexit(), malloc(),
...), so if you need something better than "seems to work", you really
need to make sure that you understand all the issues, and have a build
process that resolves them all.

Regards,
Martin

You know if I had not tried the example you gave, I'd probably still be
arguing: "but it works." Oh well. Not anymore.

For those who are interested, here is a patch to demo.c of the
\python\dist\src\Demo\embed\demo.c demonstrating what Martin has argued
about above. With such evidence at hand I guess it is safe to say that
the MinGW die-hards will have to improvise.

But seriously, and since I don't have the official Python 2.4
distribution yet, and since the pyMinGW compiled Python I use relies on
msvcrt, as opposed to msvcr71, I needed to reverse the roles here to
see the problem first hand: making the "extension" demo use msvcr71 to
see how it reacts with the home-made python24.dll which uses msvcrt.
And sure enough: crash.

Having said that, this seems to me not to be an issue for those who
have or can build Python and/or the extensions they need from sources.
Indeed recompiling the same "demo" to use msvcrt (to use the same
run-time library that my python24.dll uses) instead of msvcr71 yields
the expected correct results without crashes.

And so I suspect this approach (compiling sources yourself) can shield
people from this ugly mess, if they can build Python 2.4 and the
extensions and have them (core and extension) use one and the same
run-time library. But indeed this might be an issue for those who [1]
use MinGW as their first choice and [2] cannot or won't bother to build
their Python and/or extensions from sources, and hence have to rely on
binary distributions.


Regards
Khalid



--- demo.c Sat Nov 17 08:30:20 2001
+++ demo2.c Thu Dec 23 02:02:40 2004
@@ -34,6 +34,20 @@
/* Note that you can call any public function of the Python
interpreter here, e.g. call_object(). */

+ FILE *in;
+ int sts;
+
+ if ((in = fopen(argv[1], "rt")) == NULL) {
+ fprintf(stderr, "Cannot open program file.\n");
+ return 1;
+ }
+
+ printf("Working on %s\n", argv[1]);
+ sts = PyRun_AnyFile(in, argv[1]) != 0;
+ printf("sts: %d\n", sts);
+ fclose(in);
+
+
/* Some more application specific code */
printf("\nGoodbye, cruel world\n");

 

[Scott David Daniels]

And so I suspect this approach (compiling sources yourself) can shield
people from this ugly mess, if they can build Python 2.4 and the
extensions and have them (core and extension) use one and the same
run-time library. But indeed this might be an issue for those who [1]
use MinGW as their first choice and [2] cannot or won't bother to build
their Python and/or extensions from sources, and hence have to rely on
binary distributions.

I encourage anyone who gets further into solving the "How do I use
MinGW to build Python2.4 (and later) standard-distribution compatible
modules (.pyd s)?" question to share any clues they have. The MS
free compiler is useful to many, but not all of us.

I expect that MinGW is the most frequent choice for making Python
modules in the "free compiler outside of MS" universe. Let's make
that one work well, and (once we've done that) we can pass the lessons
on for other compiler is desired.

--Scott David Daniels
(e-mail address removed)

p.s. I am able (and willing) to use the MS free compiler, but I have
sympathy for those who aren't. I'd like to see it possible to use
_lots_ of different compilers, but we are at the beginning of the
problem, not the end. If Intel's compiler is as much faster than
MS's as it used to be, I may eventually wind up using theirs anyway.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

I encourage anyone who gets further into solving the "How do I use
MinGW to build Python2.4 (and later) standard-distribution compatible
modules (.pyd s)?" question to share any clues they have. The MS
free compiler is useful to many, but not all of us.

I think the simplest answer is "It works out of the box". Just do

python setup.py build --compiler=mingw32

If it fails, report what the failure is (the first failure should
be lack of libpython24.a, for which you can find build instructions
in Google).

Regards,
Martin