Python server locks up

[Zac Burns]

I have a server running Python 2.6x64 which after running for about a
month decides to lock up and become unresponsive to all threads for
several minutes at a time. While it is locked up Python proceeds to
consume large amounts of continually increasing memory.

The basic function of the server is to serve a large dictionary -
somewhat like a database. I have a couple theories as to why it locks
up, but I'm not sure how to test them.

Theories:
Python is resizing the large dictionary
Python is garbage collecting

How would you suggest to figure out what is the problem?

--
Zachary Burns
(407)590-4814
Aim - Zac256FL
Production Engineer (Digital Overlord)
Zindagi Games

 

[sturlamolden]

Theories:
   Python is resizing the large dictionary
   Python is garbage collecting

Python uses reference counting, not a generational GC like Java. A
Python object is destroyed when the refcount drops to 0. The GC only
collects cyclic references. If you create none, there are no GC delays
(you can in fact safely turn the GC off). Python does not share Java's
nasty habit of having long GC delays.

 

[Paul Rubin]

Python uses reference counting, not a generational GC like Java. A
Python object is destroyed when the refcount drops to 0. The GC only
collects cyclic references. If you create none, there are no GC delays
(you can in fact safely turn the GC off). Python does not share Java's
nasty habit of having long GC delays.

If you drop the last reference to a large object (say a billion item
dictionary), then Python can pause for quite a long time freeing
all the constituents of that object.

 

[Terry Reedy]

Python uses reference counting, not a generational GC like Java.

The CPython implementation, that is. Jython, built on top of Java, uses
Java's GC. Ditto for IronPython implementation. PyPY may allow some choice.

A

 

[Dave Angel]

If you drop the last reference to a large object (say a billion item
dictionary), then Python can pause for quite a long time freeing
all the constituents of that object.

(Both of these messages refer to an implementation, CPython, not the
language Python.)

The key difference is that CPython "pauses" at a deterministic point in
the code, because of something the program has specified, rather than at
a point when some system measurement decides that now would be a good
time for gc.