Why on CentOS, python consumes too much memory ?

[snowingbear]

Hi:

Previously, we found that our python scripts consume too much memory. SoI use python's resource module to restrict RLIMIT_AS's soft limit and hardlimit to 200M.
On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 6.2(x86_64)+python2.6.6, it reports memory error(exceeding 200M).

And I tested with a very small script, and result is out of my expect, it still use too much memory on my CentOS 6.2 python:
import time
time.sleep(200)

I use guppy and memory_profiler to see the memory usage and see that python objects just use about 6M memory both on RHEL5.3(i386)+python2.6 and CentOS 6.2(x86_64)+python2.6. But when I cat /proc/<pid>/status. I found that though VmRss is not very large on both machines. But the VmSize on CentOS6.2(x86_64)+python2.6 is 140M-180M, while on my RHEL5.3+python2.6, the VmSize is just 6M. And I tested on CentOS 5.7(x86_64)+python2.4.3, the VmSize is 70M.

I could understand that 64 bit machines will occupy more virtual memory than that on 32 bit, because the length of some types are not the same. But I don't know why they differs so greatly(6M to 180M), Or is this only caused by that python2.6 on CentOS 6.2's memory allocation is different from python's default one? Could you kindly give me some clues? Thank you very much.

Best Regards
William

 

[William Bai]

I found that it was caused by not by python but by /usr/lib/locale/locale-archive, the same problem as that described in
http://illiterat.livejournal.com/4615.html.

William

在 2013å¹´7月18日星期四UTC+8下åˆ12æ—¶45分01秒，William Bai写é“：

 

[Antoine Pitrou]

Hi:

Previously, we found that our python scripts consume too much memory.

So I use
python's resource module to

restrict RLIMIT_AS's soft limit and hard limit to 200M.
On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS

6.2(x86_64)+python2.6.6, it reports memory

error(exceeding 200M).

Take a look at http://www.selenic.com/smem/ for accurate measurement of
actual memory consumption under Linux. Virtual memory size is generally
useless for this purpose.

Regards

Antoine.

 

[Michael Torrie]

I found that it was caused by not by python but by
/usr/lib/locale/locale-archive, the same problem as that described
in http://illiterat.livejournal.com/4615.html.

Too funny. So in other words there isn't a problem at all. What you
thought was RAM usage was really just a memory-mapped file. That's why
Antoine said that using VSS to determine memory usage is not valid at
all in determining memory footprint. VSS shows all kinds of things from
shared libraries to locale archives that have zero impact on RAM usage.
Every app will show at least the size of glibc in its VSS number.

What is "too much memory" anyway? What do you mean by "consume too much
memory?" Now if your script has a resource leak and is long-running,
then that's a problem, but the solution is to fix your resource leak,
not have the OS kill your app when it exceeds the RLIMIT.