Total maximal size of data

[Alexander Moibenko]

I have a simple question to which I could not find an answer.
What is the total maximal size of list including size of its elements?
I do not like to look into python source.
Here is a code example:
import struct
KB=1024
MB=KB*KB
GB=MB*KB
buf=[]
bs=32*KB
n=4*GB/bs
print "N",n
i=0
size=0L
while i < n:
data = struct.pack("%ss" % (bs,), "")
buf.append(data)
size = size+bs
if size % (100*MB) == 0:
print "SIZE", size/MB, "MB"
i=i+1
while 1:
# to keep script running while I am looking at the machine status
pass

Here is what I get on 32-bit architecture:
cat /proc/meminfo
MemTotal: 8309860 kB
MemFree: 5964888 kB
Buffers: 84396 kB
Cached: 865644 kB
SwapCached: 0 kB
.......
The program output:
N 131072
SIZE 100 MB
SIZE 200 MB
SIZE 300 MB
SIZE 400 MB
SIZE 500 MB
SIZE 600 MB
SIZE 700 MB
SIZE 800 MB
SIZE 900 MB
SIZE 1000 MB
SIZE 1100 MB
SIZE 1200 MB
SIZE 1300 MB
SIZE 1400 MB
SIZE 1500 MB
SIZE 1600 MB
SIZE 1700 MB
SIZE 1800 MB
SIZE 1900 MB
SIZE 2000 MB
SIZE 2100 MB
SIZE 2200 MB
SIZE 2300 MB
SIZE 2400 MB
SIZE 2500 MB
SIZE 2600 MB
SIZE 2700 MB
SIZE 2800 MB
SIZE 2900 MB
SIZE 3000 MB
Traceback (most recent call last):
File "bs.py", line 14, in ?
data = struct.pack("%ss" % (bs,), "")
MemoryError
++++++++++++++++++++
The number of list elements for a given block size is 131072.
If I change block size the script traces back at the same total size 3000MB.
Somewhere I read that list could have
2147483647 items, on most platforms.
Somewhere else that it is
*536,870,912
(http://stackoverflow.com/questions/855191/how-big-can-a-python-array-get)**
*But what is the maximal size of the whole list including the size of
its elements?
Thanks.

 

[Diez B. Roggisch]

Am 25.01.10 20:05, schrieb Alexander Moibenko:

I have a simple question to which I could not find an answer.
What is the total maximal size of list including size of its elements?
I do not like to look into python source.

But it would answer that question pretty fast. Because then you'd see
that all list-object-methods are defined in terms of Py_ssize_t, which
is an alias for ssize_t of your platform. 64bit that should be a 64bit long.

Diez

 

[Diez B. Roggisch]

Am 25.01.10 20:05, schrieb Alexander Moibenko:

I have a simple question to which I could not find an answer.
What is the total maximal size of list including size of its elements?
I do not like to look into python source.

But it would answer that question pretty fast. Because then you'd see
that all list-object-methods are defined in terms of Py_ssize_t, which
is an alias for ssize_t of your platform. 64bit that should be a 64bit long.

Diez

 

[AlexM]

Am 25.01.10 20:05, schrieb Alexander Moibenko:


But it would answer that question pretty fast. Because then you'd see
that all list-object-methods are defined in terms of Py_ssize_t, which
is an alias for ssize_t of your platform. 64bit that should be a 64bit long.

Diez

Then how do explain the program output?
Alex.

 

[Diez B. Roggisch]

Am 25.01.10 20:39, schrieb AlexM:

Then how do explain the program output?

What exactly? That after 3GB it ran out of memory? Because you don't
have 4GB memory available for processes.

Diez

 

[AlexM]

Am 25.01.10 20:39, schrieb AlexM:



What exactly? That after 3GB it ran out of memory? Because you don't
have 4GB memory available for processes.

Diez

Did you see my posting?
.....
Here is what I get on 32-bit architecture:
cat /proc/meminfo
MemTotal: 8309860 kB
MemFree: 5964888 kB
Buffers: 84396 kB
Cached: 865644 kB
SwapCached: 0 kB
......

I have more than 5G in memory not speaking of swap space.

 

[Diez B. Roggisch]

Am 25.01.10 21:15, schrieb AlexM:

Did you see my posting?
....
Here is what I get on 32-bit architecture:
cat /proc/meminfo
MemTotal: 8309860 kB
MemFree: 5964888 kB
Buffers: 84396 kB
Cached: 865644 kB
SwapCached: 0 kB
.....

I have more than 5G in memory not speaking of swap space.

Yes, I saw your posting. 32Bit is 32Bit. Do you know about PAE?

http://de.wikipedia.org/wiki/Physical_Address_Extension

Just because the system can deal with more overall memory - one process
can't get more than 4 GB (or even less, through re-mapped memory).
Except it uses specific APIs like the old hi-mem-stuff under DOS.

Diez

 

[AlexM]

Am 25.01.10 21:15, schrieb AlexM:






Yes, I saw your posting. 32Bit is 32Bit. Do you know about PAE?

   http://de.wikipedia.org/wiki/Physical_Address_Extension

Just because the system can deal with more overall memory - one process
can't get more than 4 GB (or even less, through re-mapped memory).
Except it uses specific APIs like the old hi-mem-stuff under DOS.

Diez

Yes, I do. Good catch! I have PAE enabled, but I guess I have compiled
python without extended memory. So I was looking in the wrong place.
Thanks!
AlexM

 

[Diez B. Roggisch]

Am 25.01.10 22:22, schrieb AlexM:

Yes, I do. Good catch! I have PAE enabled, but I guess I have compiled
python without extended memory. So I was looking in the wrong place.

You can't compile it with PAE. It's an extension that doesn't make sense
in a general purpose language. It is used by Databases or some such,
that can hold large structures in memory that don't need random access,
but can cope with windowing.

Diez

 

[AlexM]

Am 25.01.10 22:22, schrieb AlexM:





You can't compile it with PAE. It's an extension that doesn't make sense
in a general purpose language. It is used by Databases or some such,
that can hold large structures in memory that don't need random access,
but can cope with windowing.

Diez

Well, there actually is a way of building programs that may use more
than 4GB of memory on 32 machines for Linux with higmem kernels, but I
guess this would not work for python.
I'll just switch to 64-bit architecture.
Thanks again.
AlexM

 

[Diez B. Roggisch]

Well, there actually is a way of building programs that may use more
than 4GB of memory on 32 machines for Linux with higmem kernels, but I
guess this would not work for python.

As I said, it's essentially paging:

http://kerneltrap.org/node/2450

And it's not something you can just compile in, you need explicit
code-support for it. Which python hasn't. And most other programs. So
there is not a magic compile option.

I'll just switch to 64-bit architecture.

That's the solution, yes

Diez

 

[denis]

I have a simple question to which I could not find an answer.
What is the [total maximal] size of list including size of its elements?

Beware, sys.getsizeof(alist) is 4*len(alist) + a bit, regardless of
alists's contents ?!
See http://stackoverflow.com/questions/2117255/python-deep-getsizeof-list-with-contents

cheers
-- denis