memory usage, temporary and otherwise

[mk]

Obviously, don't try this on low-memory machine:
.... a='spam'*10
....

>>> import sys
>>> sys.getsizeof(a) 201326728
>>> id(a[1]) 3085643936L
>>> id(a[100])

3085713568L

.... ids[id(a)]=True
....10000000

Hm, apparently Python didn't spot that 'spam'*10 in a's values is really
the same string, right?

So sys.getsizeof returns some 200MB for this dictionary. But according
to top RSS of the python process is 300MB. ps auxw says the same thing
(more or less).

Why the 50% overhead? (and I would swear that a couple of times RSS
according to top grew to 800MB).

Regards,
mk

 

[Bruno Desthuilliers]

mk a écrit :

Obviously, don't try this on low-memory machine:

Note that in Python 2, this will build a list of 10000000 int objects.
You may want to use xrange instead...

... a='spam'*10
...

import sys
sys.getsizeof(a) 201326728
id(a[1]) 3085643936L
id(a[100])

3085713568L

ids={}
for i in range(len(a)):

And this build yet another list of 10000000 int objects.

... ids[id(a)]=True
...10000000

Hm, apparently Python didn't spot that 'spam'*10 in a's values is really
the same string, right?

Seems not. FWIW, Python does some caching on some values of some
immutable types (small ints, some strings etc), but this is
implementation dependant so you shouldn't rely on it.

So sys.getsizeof returns some 200MB for this dictionary. But according
to top RSS of the python process is 300MB. ps auxw says the same thing
(more or less).

Why the 50% overhead? (and I would swear that a couple of times RSS
according to top grew to 800MB).

(overly simplified)

When an object is garbage-collected, the memory is not necessarily
"returned" to the system - and the system doesn't necessarily claim it
back neither until it _really_ needs it.

This avoid a _lot_ of possibly useless work for both the python
interpreter (keeping already allocated memory costs less than immediatly
returning it, just to try and allocate some more memory a couple
instructions later) and the system (ditto - FWIW, how linux handles
memory allocations is somewhat funny, if you ever programmed in C).

HTH

 

[Bruno Desthuilliers]

Bruno Desthuilliers a écrit :

mk a écrit : (snip)

Oh, and yes - the interpreter itself, the builtins, and all imported
modules also eat some space...

(snip)

 

[mk]

mk a écrit :

Note that in Python 2, this will build a list of 10000000 int objects.
You may want to use xrange instead...

Huh? I was under impression that some time after 2.0 range was made to
work "under the covers" like xrange when used in a loop? Or is it 3.0
that does that?

And this build yet another list of 10000000 int objects.

Well this explains much of the overhead.

(overly simplified)

When an object is garbage-collected, the memory is not necessarily
"returned" to the system - and the system doesn't necessarily claim it
back neither until it _really_ needs it.

This avoid a _lot_ of possibly useless work for both the python
interpreter (keeping already allocated memory costs less than immediatly
returning it, just to try and allocate some more memory a couple
instructions later) and the system (ditto - FWIW, how linux handles
memory allocations is somewhat funny, if you ever programmed in C).

Ah! That explains a lot. Thanks to you, I have again expanded my
knowledge of Python!

Hmm I would definitely like to read smth on how CPython handles memory
on Python wiki. Thanks for that doc on wiki on "functions & methods" to
you and John Posner, I'm reading it every day like a bible. ;-)


Regards,
mk

 

[lbolla]

 >>> a={}
 >>> for i in range(10000000):
...     a=intern('spam'*10)

"intern": another name borrowed from Lisp?

 

[Terry Reedy]

Bruno Desthuilliers wrote:

Huh? I was under impression that some time after 2.0 range was made to
work "under the covers" like xrange when used in a loop? Or is it 3.0
that does that?

3.0.

 

[Steve Holden]

Hm, apparently Python didn't spot that 'spam'*10 in a's values is really
the same string, right?

If you want it to spot that then give it a hint that it should be looking
for identical strings:
... a=intern('spam'*10)

should reduce your memory use somewhat.

Better still, hoist the constant value out of the loop:
... a = const


regards
Steve