segmentation fault in scipy?

[conor.robinson]

I'm running operations large arrays of floats, approx 25,000 x 80.
Python (scipy) does not seem to come close to using 4GB of wired mem,
but segments at around a gig. Everything works fine on smaller batches
of data around 10,000 x 80 and uses a max of ~600mb of mem. Any Ideas?
Is this just too much data for scipy?

Thanks Conor

Traceback (most recent call last):
File "C:\Temp\CR_2\run.py", line 68, in ?
net.rProp(1.2, .5, .000001, 50.0, input, output, 1)
File "/Users/conorrob/Desktop/CR_2/Network.py", line 230, in rProp
print scipy.trace(error*scipy.transpose(error))
File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line
149, in
__mul__
return N.dot(self, other)
MemoryError

 

[Travis E. Oliphant]

I'm running operations large arrays of floats, approx 25,000 x 80.
Python (scipy) does not seem to come close to using 4GB of wired mem,
but segments at around a gig. Everything works fine on smaller batches
of data around 10,000 x 80 and uses a max of ~600mb of mem. Any Ideas?
Is this just too much data for scipy?

Thanks Conor

Traceback (most recent call last):
File "C:\Temp\CR_2\run.py", line 68, in ?
net.rProp(1.2, .5, .000001, 50.0, input, output, 1)
File "/Users/conorrob/Desktop/CR_2/Network.py", line 230, in rProp
print scipy.trace(error*scipy.transpose(error))
File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line
149, in
__mul__
return N.dot(self, other)
MemoryError

You should ask this question on the numpy-discussion list for better
feedback.


Does it actually segfault or give you this Memory Error?


Temporary arrays that need to be created could be the source of the
extra memory.


Generally, you should be able to use all the memory on your system
(unless you are on a 64-bit system and are not using Python 2.5).



-Travis

 

[conor.robinson]

If I run it from the shell (unix) I get: Segmentation fault and see a
core dump in my processes. If I run it in the python shell I get as
above:
File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line
149, in
__mul__
return N.dot(self, other)
MemoryError

I your experience as one of the dev of scipy, is this too much data?

thank you

 

[Robert Kern]

I'm running operations large arrays of floats, approx 25,000 x 80.
Python (scipy) does not seem to come close to using 4GB of wired mem,
but segments at around a gig. Everything works fine on smaller batches
of data around 10,000 x 80 and uses a max of ~600mb of mem. Any Ideas?
Is this just too much data for scipy?

Thanks Conor

Traceback (most recent call last):
File "C:\Temp\CR_2\run.py", line 68, in ?
net.rProp(1.2, .5, .000001, 50.0, input, output, 1)
File "/Users/conorrob/Desktop/CR_2/Network.py", line 230, in rProp
print scipy.trace(error*scipy.transpose(error))
File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line
149, in
__mul__
return N.dot(self, other)
MemoryError

This is not a segfault. Is this the only error you see? Or are you actually
seeing a segfault somewhere?

If error.shape == (25000, 80), then dot(error, transpose(error)) will be
returning an array of shape (25000, 25000). Assuming double precision floats,
that array will take up about 4768 megabytes of memory, more than you have. The
memory usage doesn't go up near 4 gigabytes because the allocation of the very
large returned array fails, so the large chunk of memory never gets allocated.

There are two possibilities:

1. As Travis mentioned, numpy won't create the array because it is still
32-bit-limited due to the Python 2.4 C API. This has been resolved with Python 2.5.

2. The default build of numpy uses plain-old malloc(3) to allocate memory, and
it may be failing to create such large chunks of memory.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[conor.robinson]

Good point. Finding the SSE using an absolute error matrix of (25000 x
1) is insane. I pulled out the error function (for now) and I'm back
in business. Thanks for all the great advise.

 

[Robert Kern]

Good point. Finding the SSE using an absolute error matrix of (25000 x
1) is insane. I pulled out the error function (for now) and I'm back
in business. Thanks for all the great advise.

Could you go back for a second and describe your problem a little bit more. It
sounds like you were doing the wrong operation. By SSE, you mean "Sum of Squared
Errors" of the 25000 length-80 vectors, right? In that case, using matrix
multiplication won't give you that. That will, in essence calculate the
dot-product of each of the 25000 length-80 vectors with *each* of the other
25000 length-80 vectors in addition to themselves. It seems to me like you want
something like this:

SSE = sum(error * error, axis=-1)

Then SSE.shape == (25000,).

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[conor.robinson]

Im using rprop (not dependent on error function in this case ie.
standard rprop vs. irprop or arprop) for an MLP tanh, sigmod nnet as
part of a hybrid model. I guess I was using a little Matlab thought
when I wrote the SSE funtion. My batches are about 25,000 x 80 so my
absolute error (diff between net outputs and desired outputs) when
using *one* output unit is shape(~25000,), am I wrong to assume
trace(error*transpose(error)) is the sum of the squared errors which
should be an shape(1,)? I'm just now starting to dig a little deeper
into scipy, and I need to get the full doc.

Thanks for all your input.

 

[Robert Kern]

Im using rprop (not dependent on error function in this case ie.
standard rprop vs. irprop or arprop) for an MLP tanh, sigmod nnet as
part of a hybrid model. I guess I was using a little Matlab thought
when I wrote the SSE funtion. My batches are about 25,000 x 80 so my
absolute error (diff between net outputs and desired outputs) when
using *one* output unit is shape(~25000,), am I wrong to assume
trace(error*transpose(error)) is the sum of the squared errors which
should be an shape(1,)?

I'm afraid you're using terminology (and abbreviations!) that I can't follow.
Let me try to restate what's going on and you can correct me as I screw up. You
have a neural net that has 80 output units. You have 25000 observations that you
are using to train the neural net. Each observation vector (and consequently,
each error vector) has 80 elements.

Judging by your notation, you are using the matrix subclass of array to change *
to matrix multiplication. In my message you are responding to (btw, please quote
the emails you respond to so we can maintain some context), I gave an answer for
you using regular arrays which have * as elementwise multiplication. The matrix
object's behavior gets in the way of the most natural way to do these
calculations, so I do recommend avoiding the matrix object and learning to use
the dot() function to do matrix multiplication instead. But if you want to
continue using matrix objects, then you can use the multiply() function to do
element-wise multiplication.

The answer I gave also used the wrong name for the result. It seems that you
want the sum of the squared errors across all of the observations. In this case,
you can use axis=None to specify that every element should be summed:

SSE = sum(multiply(error, error), axis=None)

trace(dot(error, transpose(error))) wastes a *huge* amount of time and memory
since you are calculating (if your machine was capable of it) a gigantic matrix,
then throwing away all of the off-diagonal elements. The method I give above
wastes a little memory; there is one temporary matrix the size of error.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Dennis Lee Bieber]

If I run it from the shell (unix) I get: Segmentation fault and see a
core dump in my processes. If I run it in the python shell I get as
above:
File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line
149, in

That's a Window's path... Does Windows even make full use of 4GB?
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[conor.robinson]

If I run it from the shell (unix) I get: Segmentation fault and see a

core dump in my processes. If I run it in the python shell I get as
above:
File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line
149, in

That's a Window's path... Does Windows even make full use of
4GB?

Im developing on a unix machine. However, you do have a sharp eye...
So you win the blue ribbon and a free roundtable pizza! I'm actually
running experiments on an HP workstation while I work (emailed myself
the error), that's one of the beauties of python. Yes, Dennis there is
a max of 4gb of addressable mem on that machine and 2 cpus. This is
what a unix path looks like. Can you tell me what OS and shell Im
running (both are obvious from the prompt) for the bonus prize?

[gremlins:~/desktop/CR_2] conorrob% python run.py
AL
fold: 0
Done Loading
Input Ready
Network Training
Initial SSE:
Segmentation fault
[gremlins:~/desktop/CR_2] conorrob%

 

[conor.robinson]

I'm afraid you're using terminology (and abbreviations!) that I can't follow.

Let me try to restate what's going on and you can correct me as I screw up. You
have a neural net that has 80 output units. You have 25000 observations that you
are using to train the neural net. Each observation vector (and consequently,
each error vector) has 80 elements.

First, sorry for not quoting and using abbreviations. Second, your
obsevation is correct, except that I have *one* output unit, not 80. I
have ~80 input units + bias (for each of the 25000 observations), x +
bias number of hidden units and one output unit leaving me with an
output array/matrix of shape =(25000,), as well as my desired output
having the same shape.

RPROP = Resilliant BackPropogation, uses chages in the error gradiant
ignores the magnitude of the gradiant, which can be harmful. See "A
Direct Adaptive Method for Faster Backpropogation Learning: The RPROP
Algorithm" -Martin Riedmiller and Heinrich Braun, just google it and
read Section D, it's well written.

tanh = hyperbolic tangent function [-1, 1] y vals, often times better
for its steeper derivative and wider range.

sigmoid function = popular for its [0,1] range and can calculate
posterior probabilities when using a the cross entropy error function
which I have commented out since it takes more time to process and I'm
not using my error function in this specific case at this point in
time, thus SSE is not really needed, however I'd like use it as a
check. Also, popular for its easily calculatable derivative.

The answer I gave also used the wrong name for the result. It seems that you
want the sum of the squared errors across all of the observations. In this case,
you can use axis=None to specify that every element should be summed:

SSE = sum(multiply(error, error), axis=None)

trace(dot(error, transpose(error))) wastes a *huge* amount of time and memory
since you are calculating (if your machine was capable of it) a gigantic matrix,
then throwing away all of the off-diagonal elements. The method I give above
wastes a little memory; there is one temporary matrix the size of error.

This is great advise and much appreciated. It was the answer to my
problem, thank you. However, isn't this faster...
scipy.sum(scipy.array(scipy.matrix(error)*scipy.matrix(error)), axis =
None)
as we you explained in my other posting?

Thanks again.

 

[Robert Kern]

This is great advise and much appreciated. It was the answer to my
problem, thank you. However, isn't this faster...
scipy.sum(scipy.array(scipy.matrix(error)*scipy.matrix(error)), axis =
None)
as we you explained in my other posting?

No! matrix objects use matrix multiplication for *. You seem to need elementwise
multiplication.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[py_genetic]

No! matrix objects use matrix multiplication for *. You seem to need elementwise

multiplication.

No! when you mult a vector with itself transposed, the diagonal of the
resulting matrix is the squares of each error (albeit you do a lot of
extra calc), then sum the squares, ie trace(). Its a nifty trick, if
you don't have too much data 25000x25000 matrix in mem and youre using
matricies ie. batch learning. The actual equation includes multiply by
1/2*(sum of the squares), but mean squared error can be more telling
about error and cross entropy is even better, becuase it tells you how
well youre predicting the posterior probabilies...

 

[Robert Kern]

No! when you mult a vector with itself transposed, the diagonal of the
resulting matrix is the squares of each error (albeit you do a lot of
extra calc), then sum the squares, ie trace(). Its a nifty trick, if
you don't have too much data 25000x25000 matrix in mem and youre using
matricies ie. batch learning. The actual equation includes multiply by
1/2*(sum of the squares), but mean squared error can be more telling
about error and cross entropy is even better, becuase it tells you how
well youre predicting the posterior probabilies...

But, again, why bother calculating an enormous matrix just to take its trace
when you can use elementwise multiplication and not waste any time or memory
calculating values that you don't need? It's not a trick. It's just wrong.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Robert Kern]

No! when you mult a vector with itself transposed, the diagonal of the
resulting matrix is the squares of each error (albeit you do a lot of
extra calc), then sum the squares, ie trace(). Its a nifty trick, if
you don't have too much data 25000x25000 matrix in mem and youre using
matricies ie. batch learning. The actual equation includes multiply by
1/2*(sum of the squares), but mean squared error can be more telling
about error and cross entropy is even better, becuase it tells you how
well youre predicting the posterior probabilies...

Now I'm even more confused. What kind of array is "error" here? First you tell
me it's a (25000, 80) array and now you are telling me it is a (25000,) array.

Once you've defined what "error" is, then please tell me what the quantity is
that you want to calculate. I think I told you several different wrong things,
previously, based on wrong assumptions.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[py_genetic]

True! it is rediculous/insane as I mentioned and noted and agreed with
you (in all your responses) and was my problem, however, not wrong
(same result), as I was just simply noting (not to be right), although,
yes, insane. Thanks again.

 

[py_genetic]

Now I'm even more confused. What kind of array is "error" here? First you tell

me it's a (25000, 80) array and now you are telling me it is a (25000,) array.
Once you've defined what "error" is, then please tell me what the quantity is
that you want to calculate. I think I told you several different wrong things,
previously, based on wrong assumptions.

It's just in my original post I was trying to get across maximum size
of the arrays I'm using. sorry for the confusion, I didn't state actual
size of my output vectors. I discovered the probelm when your first
stated:

If error.shape == (25000, 80), then dot(error, transpose(error)) will be
returning an array of shape (25000, 25000)

Which was exactly related to the excessive calculation I was running
and set off the red flags and made it very clear.

Later I was somewhat confused and believed that you we were talking
about two different things regarding SSE when you said:

SSE = sum(multiply(error, error), axis=None)

And didn't realize that multipy() was an efficient element mult method
for matricies, thinking it was like matrixmultiply() and gave you back
the old trace sse method but with the casting (not meaning to
contradict you). However all is now very clear, and I agree with
element wise mult or even squaring the output error, and have no real
reason why I was using trace, except I had a faint memory of using it
in a class for small experiments in matlab (i guess the idea was to
keep everything in linear algebra) and I spit it up for some reason
when I was doing a quicky error function.