limiting memory consumption of Python itself or a dict in a Python program

[Jonas Maurus]

Hello everybody,

I'm pondering the following problem:

I want to write a Python program that receives messages via SMTP and
stores them in a dict or an array. For my purposes it would be
important that all received mail would be kept in RAM and not cached
out to disk. If a new message comes in that can't fit in the allocated
memory, a number of old messages would be discarded.

As the server needs to have room for other tasks, I'd like to limit
the overall memory consumption to a certain amount.

Is this possible? How would I go about implementing it? By imposing
"ulimit"s and catching MemoryError?

Thanks for your help,
Jonas

 

[Kay Schluehr]

Hello everybody,

I'm pondering the following problem:

I want to write a Python program that receives messages via SMTP and
stores them in a dict or an array. For my purposes it would be
important that all received mail would be kept in RAM and not cached
out to disk. If a new message comes in that can't fit in the allocated
memory, a number of old messages would be discarded.

As the server needs to have room for other tasks, I'd like to limit
the overall memory consumption to a certain amount.

Is this possible? How would I go about implementing it? By imposing
"ulimit"s and catching MemoryError?

Thanks for your help,
Jonas

When your application pre-allocates a chunk of memory and use
datatypes like array and struct you might write your own memory
manager. You can't control the total memory consumption of Python
though, since Python implements an own malloc and it might allocate
memory in large chunks. But this is somewhat orthogonal to your
problem and applies also when Python reads/writes buffers from/to
disk.

 

[Jeremy Sanders]

I want to write a Python program that receives messages via SMTP and
stores them in a dict or an array. For my purposes it would be
important that all received mail would be kept in RAM and not cached
out to disk. If a new message comes in that can't fit in the allocated
memory, a number of old messages would be discarded.

As the server needs to have room for other tasks, I'd like to limit
the overall memory consumption to a certain amount.

Since your data is all in one place, why not write a dict or list wrapper
which keeps track of the total space of the items stored (using len on the
strings)?

It could automatically clean out old entries when the memory usage becomes
too much.

Jeremy