Memory leak issue with complex data structure

[Alan Franzoni]

Hello,
I've got a complex data structure (graph-like). Each of my nodes may
reference zero or more other nodes, internally managed by a set() object.

I have a root node which is not referenced by any other node. So, I
created a "clear()" method which is called on all children (by calling
their clear() method" and then clears the set with the references of the
node itself.

I have a serious "leak" issue; even though I clear all those sets and I
delete all the references I can have to the current namespace, memory is
not freed.

I have tried the gc module and it couldn't help (I can't determine where
the references are) and I tried using weakref.ref as well to connect
object, but then I have problems (objects have no more references to
them beside their parents, hence they get deleted).

I understand it's impossibile to tell what's my problem without seeing
the actual code (which I can't post) but I'm asking if there's any
tool/debugger I could employ in order to track what's happening.

Thanks!

--
Alan Franzoni <[email protected]>
-
Togli .xyz dalla mia email per contattarmi.
Remove .xyz from my address in order to contact me.
-
GPG Key Fingerprint (Key ID = FE068F3E):
5C77 9DC3 BD5B 3A28 E7BC 921A 0255 42AA FE06 8F3E

 

[Josiah Carlson]

I have a root node which is not referenced by any other node. So, I
created a "clear()" method which is called on all children (by calling
their clear() method" and then clears the set with the references of the
node itself.

Using the .clear() method on sets (or dictionaries) does not reduce the
memory that the underlying hash tables use, it only removes references
to the keys (and values) that the sets (and dictionaries) contain(s).
If your sets used to be large, I could have sworn that you could force a
resize down by adding some items to the set, but I seem to be mistaken:

#memory use is 3440k
s = set(xrange(1000000))
#memory use is 31,864k
s.clear()
#memory use is 15,460k
s.add(0)
s.remove(0)
#memory use is 15,460k

Then again, this may be an artifact of my operating system memory
freeing semantics.

As is usually the case, read the C source from svn.python.org/view (or
some other means) for more information.

- Josiah

 

[Hrvoje Niksic]

I have a serious "leak" issue; even though I clear all those sets
and I delete all the references I can have to the current namespace,
memory is not freed.

Maybe the memory is freed (marked as available for further use by
Python), just not released to the operating system.[1] To test against
that, try to allocate more Python structures and see if they reuse the
freed memory or if they allocate even more memory. Even better, run
code like this:

while 1:
... populate your data structures ...
clear()

If this causes Python to allocate more and more memory, it means you
have a real leak. If not, it means that the GC is working fine, but
it's not possible to release the memory to the OS.


[1]
Not giving freed memory back to the system is not (necessarily) a
Python bug; the same thing happens in C and is a consequence of
managed memory being assigned to the process as a contiguous block.