Python memory use (psyco, C++)

[Roy Smith]

I understand that psyco significantly increases memory use. Is that for
code or data? More specifically, if I've got a memory intensive
application (it might use 100's of Mbytes of data), should I expect
memory use to go up significantly under psyco?

Also, for that memory intensive application, how should I expect Python
memory use to compare with C++? I'm really only interested in data; the
memory needed to store the code is almost certainly insignificant in
either case.

The data is a large number of small objects, interconnected in various
data structures, not just one huge block of raw data.

I know all of the above is very vague, but I'm just trying to get a
rough idea if a Python implementation is feasable (or at least
plausable). If a C++ version takes 300 Mbytes and a Python version
takes 1 Gig, that's probably not going to work. Are there any rules of
thumb I could use to get a first-order estimate?

 

[Tyler Eaves]

I know all of the above is very vague, but I'm just trying to get a
rough idea if a Python implementation is feasable (or at least
plausable). If a C++ version takes 300 Mbytes and a Python version
takes 1 Gig, that's probably not going to work. Are there any rules of
thumb I could use to get a first-order estimate?

Why wouldn't it work? Memory is still quite cheap. If this is an app of
any complexity, the development time saved vs C++ will probably far more
than additional memory would cost.

 

[Roy Smith]

Why wouldn't it work? Memory is still quite cheap. If this is an app of
any complexity, the development time saved vs C++ will probably far more
than additional memory would cost.

Because memory use is one of the constraints I was given to work with.

 

[Paul Rubin]

I understand that psyco significantly increases memory use. Is that for
code or data?

It's for code. Psyco generates code on the fly each time it hits a new
combination of datatypes anyplace in the code.

More specifically, if I've got a memory intensive
application (it might use 100's of Mbytes of data), should I expect
memory use to go up significantly under psyco?

Probably not.

Also, for that memory intensive application, how should I expect Python
memory use to compare with C++? I'm really only interested in data; the
memory needed to store the code is almost certainly insignificant in
either case.

The data is a large number of small objects, interconnected in various
data structures, not just one huge block of raw data.

If you just code straightforwardly, Python will use a lot more memory.
There may be simple, application-specific things you can do to reduce
the memory footprint.

I know all of the above is very vague, but I'm just trying to get a
rough idea if a Python implementation is feasable (or at least
plausable). If a C++ version takes 300 Mbytes and a Python version
takes 1 Gig, that's probably not going to work. Are there any rules of
thumb I could use to get a first-order estimate?

There's a minimum of around 12 (?) bytes of overhead for any Python
object including integers. So if your objects are all very small, you
will get killed.

 

[Terry Reedy]

I understand that psyco significantly increases memory use. Is that for
code or data?

Code. The amount depends on how much code is psycoized and how many
specialized versions of each block of code it ends up compiling. I have
the impression that the amount is perhaps megabytes, not 10s of megabytes.

> Also, for that memory intensive application, how should I expect Python
memory use to compare with C++?

If you have large arrays of numbers and use Numerical Python or Numarray,
space and speed are comparable. If you have large chunks of text (millions
of bytes each), space is about same. If you have millions of separate
objects (network, dict with millions of 6 byte words), then I expect memory
overhead of Python to be larger, perhaps enough to be a problem. A more
specific description of your situation will get more specific answers,
possibly with specific ways to reduce bloat.

Terry J. Reedy