embedding Python: how to avoid memory leaks?

[Andrew Trevorrow]

No response to my last message, so I'll try a different tack...

Does anyone know of, or even better, has anyone here written a
C++ application for Mac/Windows that allows users to run Python
scripts from within the app? Not just once, but many times in
a single session, and without leaking memory. Preferably an
open source app so I can see how it's done.

Our app (http://golly.sourceforge.net/) currently uses calls
like these every time a user decides to run a script:

Py_Initialize();
PyRun_SimpleString("execfile('foo.py')");
Py_Finalize();

But even if foo.py is *empty* the above calls result in a memory
leak of about 12K on Mac OS 10.3.9 (using Python 2.3) and about
11K on Windows 2000 (using Python 2.4.2). I wouldn't mind if
there was a one-off cost due to calling Py_Initialize the very
first time, but we see leaks every time a script is executed.

I've tried calling Py_Initialize just once (at app startup)
and Py_Finalize once on exit, but that doesn't really solve
anything. It avoids leaks when using trivial scripts (like
an empty .py file!) but we want to run some rather complicated
scripts that consume lots of memory, so we need a reliable way
to release that memory. I was surprised to discover that
Py_Finalize doesn't always do that.

Is there some magic Python code that can restore memory usage
to what it was before each execfile call? Something like
PostScript's save and restore.

I've been struggling with this problem for about a week now.
Having been seduced by Python's power and beauty I'd hate to
have to abandon it and switch to Perl or some other crappy
scripting language! Please help...

Andrew

 

[Torsten Bronger]

Hallöchen!

[...]

[...] Not just once, but many times in a single session, and
without leaking memory. Preferably an open source app so I can
see how it's done.

Our app (http://golly.sourceforge.net/) currently uses calls
like these every time a user decides to run a script:

Py_Initialize();
PyRun_SimpleString("execfile('foo.py')");

Does PyRun_AnyFile show the same effect? That's the way I'm about
to go.

Tschö,
Torsten.

 

[Andrew Trevorrow]

Our app (http://golly.sourceforge.net/) currently uses calls

Does PyRun_AnyFile show the same effect? That's the way I'm about
to go.

I couldn't get the PyRun_*File* calls to work on Windows, presumably
because of the FILE* problem mentioned in the docs.

I'll be very surprised if it makes any difference to the memory
leak problem. Let me know how you get on!

Andrew

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Our app (http://golly.sourceforge.net/) currently uses calls
like these every time a user decides to run a script:

Py_Initialize();
PyRun_SimpleString("execfile('foo.py')");
Py_Finalize();

But even if foo.py is *empty* the above calls result in a memory
leak of about 12K on Mac OS 10.3.9 (using Python 2.3) and about
11K on Windows 2000 (using Python 2.4.2).

I could reproduce a memory leak with the code

#include <Python.h>
int main()
{
while(1){
Py_Initialize();
PyRun_SimpleString("execfile('foo.py')");
Py_Finalize();
}
}

However, I could not reproduce a memory leak with the code

#include <Python.h>
int main()
{
Py_Initialize();
while(1){
PyRun_SimpleString("execfile('foo.py')");
}
Py_Finalize();
}

So I recommend you do Py_Initialize only once. It is well-known
that initializing the Python interpreter allocates memory that
can never be freed, e.g. global variables in extension modules
(there just isn't any API to tell all the modules to release their
memory). So a cycle of Py_Initialize/Py_Finalize will certainly
leak.

OTOH, PyRun_SimpleString shouldn't leak, and didn't when I
tried it.

Regards,
Martin

 

[Torsten Bronger]

Hallöchen!

[...]

I couldn't get the PyRun_*File* calls to work on Windows, presumably
because of the FILE* problem mentioned in the docs.

Which compiler do you use?

Tschö,
Torsten.

 

[Andrew Trevorrow]

[...]

I couldn't get the PyRun_*File* calls to work on Windows, presumably
because of the FILE* problem mentioned in the docs.

Which compiler do you use?

MSVC++ (version 6 from memory -- I do most of my development on the
Mac and fire up Virtual PC occasionally to test Win builds).

Andrew

 

[Torsten Bronger]

Hallöchen!

[...]

I couldn't get the PyRun_*File* calls to work on Windows,
presumably because of the FILE* problem mentioned in the docs.

Which compiler do you use?

MSVC++ (version 6 from memory -- I do most of my development on
the Mac and fire up Virtual PC occasionally to test Win builds).

Well, I don't really *know*, but it's hard to believe to me that the
file descriptor format changed within the Microsoft product series.

Tschö,
Torsten.

 

[Andrew Trevorrow]

I could reproduce a memory leak with the code

#include <Python.h>
int main()
{
while(1){
Py_Initialize();
PyRun_SimpleString("execfile('foo.py')");
Py_Finalize();
}
}

However, I could not reproduce a memory leak with the code

#include <Python.h>
int main()
{
Py_Initialize();
while(1){
PyRun_SimpleString("execfile('foo.py')");
}
Py_Finalize();
}

So I recommend you do Py_Initialize only once. It is well-known
that initializing the Python interpreter allocates memory that
can never be freed, e.g. global variables in extension modules
(there just isn't any API to tell all the modules to release their
memory). So a cycle of Py_Initialize/Py_Finalize will certainly
leak.

Surely that's a bug that should be fixed. There should be some way
to tell Python "release all the memory you've ever allocated and
start again with a clean slate".

OTOH, PyRun_SimpleString shouldn't leak, and didn't when I
tried it.

Ok, after reading http://evanjones.ca/python-memory.html I think I
understand what's happening. Apparently the Python memory allocator
never releases memory back to the OS! So if a complicated script
happens to consume 100MB of interpreter memory then that amount is
no longer available to the app in which Python is embededded.
Even worse, if a script has a (Python) memory leak then there's
nothing the app can do about it. It would be great if wrapping
each script inside Py_Initialize/Py_Finalize could avoid all that.

I've been told that the next version of Python will release memory,
so that's good news. You can get it now if you're willing to build
Python from the latest source code.

Andrew

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Surely that's a bug that should be fixed. There should be some way
to tell Python "release all the memory you've ever allocated and
start again with a clean slate".

This bug cannot be fixed in any foreseeable future.

I've been told that the next version of Python will release memory,
so that's good news. You can get it now if you're willing to build
Python from the latest source code.

That still won't release all memory - only the arenas that don't
have live Python objects on them anymore.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Well, I don't really *know*, but it's hard to believe to me that the
file descriptor format changed within the Microsoft product series.

The layout of the FILE type indeed didn't change. However, passing
FILE* across different versions of msvcrt will still crash; google
for details. In particular, if you build an application with VC6,
it will be linked with msvcrt4.dll. If you combine this with Python
2.4 (which is linked with msvcr71.dll), you cannot use PyRun_*File*.

Regards,
Martin

 

[Wolfgang]

Hallo,

Well, I don't really *know*, but it's hard to believe to me that the
file descriptor format changed within the Microsoft product series.

Yes it is hard to believe but true.
Even debug and release versions have incompatible file pointer.
It's not possible to pass af FILE* from a debug app to the embedded
release python.dll.

A workaround is to create the FILE* with the python api functions and
pass this pointer back to python.