Memory Usage

[rbt]

Would a Python process consume more memory on a PC with lots of memory?

For example, say I have the same Python script running on two WinXP
computers that both have Python 2.4.0. One computer has 256 MB of Ram
while the other has 2 GB of Ram. On the machine with less Ram, the
process takes about 1 MB of Ram. On the machine with more Ram, it uses 9
MB of Ram.

Is this normal and expected behavior?

Thanks,

rbt

 

[Peter Hansen]

Would a Python process consume more memory on a PC with lots of memory?

For example, say I have the same Python script running on two WinXP
computers that both have Python 2.4.0. One computer has 256 MB of Ram
while the other has 2 GB of Ram. On the machine with less Ram, the
process takes about 1 MB of Ram. On the machine with more Ram, it uses 9
MB of Ram.

Is this normal and expected behavior?

It's probably not normal if this is *really* the memory usage, but
I would expect to see such behaviour, given how difficult it is
to measure *actual* memory usage. How are you measuring it?
Just by looking at the Mem Usage column in the Task Manager?

-Peter

 

[Fredrik Lundh]

For example, say I have the same Python script running on two WinXP computers that both have
Python 2.4.0. One computer has 256 MB of Ram while the other has 2 GB of Ram. On the machine with
less Ram, the process takes about 1 MB of Ram. On the machine with more Ram, it uses 9 MB of Ram.

Is this normal and expected behavior?

1 MB sounds low, 9 MB sounds more reasonable for a script that uses a few mega-
bytes of data. what tool are you using to determine the process size?

</F>

 

[rbt]

It's probably not normal if this is *really* the memory usage, but
I would expect to see such behaviour, given how difficult it is
to measure *actual* memory usage. How are you measuring it?
Just by looking at the Mem Usage column in the Task Manager?

-Peter

That's right. I look at that column. Should I measue mem usage in some
other way?

 

[Peter Hansen]

That's right. I look at that column. Should I measue mem usage in some
other way?

Probably, but for a start, have you noticed that even just
switching to another window can drastically affect the
memory apparently used by an application? For example,
running the wxPython demo, clicking on a few controls and
getting memory usage up to 27MB, then minimizing the window
will drop it down aboiut 2MB. Restoring the window will
bring the amount back to only about 4MB, at least until
you click on stuff. Even then, it might climb to only
about 14MB or so.

This particular phenomenon might not be affecting your
application, but it's an indication of how bad this
measurement technique can be.

Inside the Control Panel, you will find "Administrative Tools".
In there is a "Performance" gadget. It's not trivial to
use, and I can't give a tutorial here, but if you can
manage to display the Working Set for the specific Process
in which you are interested, that's probably a better
way to view the information. (To be honest, I think it's
actually this particular value which the "Mem Usage"
field shows in the Task Manager, but at least this way
you get a real-time graph and more precision, and a
record of the usage.) There are also dozens of other
parameters you can examine to help you track down the
actual usage, or to help find out what is going if it turns
out that you really are using such different amounts on
the two machines.

More important than any of this, however, might be making
sure you have similar conditions on the two machines. Are
you terminating as many other processes as you can? Making
sure there is ample Physical Memory Available (see the
Performance tab of Task Manager, for example)? If you
have one of the machines running out of memory because
of other applications running, it is quite possible that
the OS will steal memory from the Python process to feed
the other apps, and that can show up in the working set size.

As I said, this stuff isn't exactly straightforward, so
it's not necessarily surprising you are seeing this behaviour.
There's probably a relatively simple explanation, however,
but it might come only after a bit of exploration.

-Peter

 

[Stuart McGarrity]

Do you have a page file?

The Mem column should be RAM usage and not total working set. Some of it
could be swapped to the page file. A free tool like process explorer can
give you better informaton than the task manager.

 

[Tim Peters]

Would a Python process consume more memory on a PC with lots of
memory?

For example, say I have the same Python script running on two WinXP
computers that both have Python 2.4.0. One computer has 256 MB of Ram
while the other has 2 GB of Ram. On the machine with less Ram, the
process takes about 1 MB of Ram. On the machine with more Ram, it uses
9 MB of Ram.

Is this normal and expected behavior?

[and later confirms he's looking at Mem Usage in Task Manager]

256MB is on the low end for an XP system, and will generally cause
lots of swap space (virtual memory residing on disk -- kinda) to get
used. Mem Usage doesn't count swap space, just RAM currently in use.
Under Task Manager it's better to look at the VM Size column, which
tells roughly how much virtual address space is assigned to the
process, and regardless of how it's currently split between RAM and
disk. You may need to use View -> Select Columns to get this
statistic displayed. If your process is in fact using a fixed amount
of address space, the VM Size column will stay steady but Mem Usage
may jump all over the place as time goes on. In general, I would
expect VM Size to be about the same on your two boxes; I would not
expect Mem Usage to be the same.

 

[Stephen Kellett]

That's right. I look at that column. Should I measue mem usage in some
other way?

Try VM Validator, a free memory visualization tool from Software
Verification.

http://www.softwareverify.com
http://www.softwareverify.com/vmValidator/index.html

It shows paged memory usage and also Virtual Memory manager usage on
separate tabs. Colour coded visual representation of each 4K page of
memory.

Probably more use on more memory intensive applications than yours, but
may still shed some light all the same. Either launch Python from VM
Validator, or inject VM Validator into your running Python.exe process.

Stephen

 

[Nick Coghlan]

Do you have a page file?

The Mem column should be RAM usage and not total working set. Some of it
could be swapped to the page file. A free tool like process explorer can
give you better informaton than the task manager.

As Tim pointed out, "View->Select Columns" and activating "VM Size" is enough to
find out how much memory that program actually has *mapped* (rather than
currently loaded into RAM).

Cheers,
Nick.