Update on Memory problem with NumPy arrays

[sonjaa]

Hi

last week I posted a problem with running out of memory when changing
values in NumPy arrays. Since then I have tried many different
approaches and
work-arounds but to no avail.

I was able to reduce the code (see below) to its smallest size and
still
have the problem, albeit at a slower rate. The problem appears to come
from changing values in the array. Does this create another reference
to the
array, which can't be released?

Also, are there other python methods/extensions that can create
multi-deminsional
arrays?

thanks again to those who repsonded to the last post
Sonja

PS. to watch the memory usage I just used task manager

the code:
from numpy import *

y = ones((501,501))
z = zeros((501,501))
it = 50

for kk in xrange(it):
y[1,1] = 4
y[1,2] = 4
y[1,0] = 4
y[2,1] = 6

print "Iteration #:%s" %(kk)
for ee in xrange(0,501):
for ff in xrange(0,501):
if y[ee,ff] == 4 or y[ee,ff] == 6:
y[ee,ff] = 2
else:
pass

 

[Robert Kern]

Hi

last week I posted a problem with running out of memory when changing
values in NumPy arrays. Since then I have tried many different
approaches and
work-arounds but to no avail.

I was able to reduce the code (see below) to its smallest size and
still
have the problem, albeit at a slower rate.

Please post this to numpy-discussion instead of here. Also, please create a
ticket in our Trac:

http://projects.scipy.org/scipy/numpy

The problem appears to come
from changing values in the array. Does this create another reference
to the
array, which can't be released?

Since the array shouldn't be going away and no new arrays should be created that
wouldn't cause a problem.

It's possible that there is a bug with the scalar objects that are being created
when indexing into the arrays. I can reproduce abnormal memory consumption even
when I remove the line where you are setting value in the if: clause.

Also, are there other python methods/extensions that can create
multi-deminsional
arrays?

A few. numpy's predecessors, Numeric and numarray are still usable, but aren't
being actively developed. There's a pure-Python array package somewhere, but I
forget the name.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Fredrik Lundh]

Also, are there other python methods/extensions that can create
multi-deminsional arrays?

if this example is typical for the code you're writing, you might as
well use nested Python lists:

def make_array(width, height, value):
out = []
for y in range(height):
out.append([value] * width)
return

y = make_array(501, 501, 1)
z = make_array(501, 501, 0)

y[ee][ff] = 4

etc

</F>

 

[Filip Wasilewski]

Hi

last week I posted a problem with running out of memory when changing
values in NumPy arrays. Since then I have tried many different
approaches and
work-arounds but to no avail.

[...]

Based on the numpy-discussion this seems to be fixed in the SVN now(?).

Anyway, you can use 'where' function to eliminate the loops:

from numpy import *

y = ones((501,501))
z = zeros((501,501))
it = 50

for kk in xrange(it):
y[1,1] = 4
y[1,2] = 4
y[1,0] = 4
y[2,1] = 6

print "Iteration #:%s" %(kk)
y = where((y == 4) | (y == 6), 2, y)


best,
fw

 

[sonjaa]

I've been in contact with Travis O, and he said it was fixed in the
SVN.
thanks for the suggestions, I'll try them out now.

best
Sonja

Hi

last week I posted a problem with running out of memory when changing
values in NumPy arrays. Since then I have tried many different
approaches and
work-arounds but to no avail.

[...]

Based on the numpy-discussion this seems to be fixed in the SVN now(?).

Anyway, you can use 'where' function to eliminate the loops:

from numpy import *

y = ones((501,501))
z = zeros((501,501))
it = 50

for kk in xrange(it):
y[1,1] = 4
y[1,2] = 4
y[1,0] = 4
y[2,1] = 6

print "Iteration #:%s" %(kk)
y = where((y == 4) | (y == 6), 2, y)


best,
fw