Problems allocating memory

[Neclepsio]

Hi everyone.
I've made a class Matrix, which contains a pointer to the data and some
methods which all return a copy of the matrix modified in some way. The
program works quite well for small matrices, but as matrix dimension
grows up it crashes in deterministic locations depending on data,
machine and OS.
In particular, this happens when I test it with 3000x3000 matrices
(which take up more than 30MB) under three different machines/OSes. The
crash happens when much memory is still available (for example, under
cygwin, more than 1GB of virtual memory available), and I cannot think
there is no contiguous space for allocating the matrix.
I always use references and never pointers to matrices, so Matrix
objects are not dynamically allocated (the data, however, is allocated
by the constructor). Could this be a problem with stack/heap limitations?
Can someone help me with some idea on how to further investigate the
problem?

Thank you,
Ignazio

 

[Ivan Vecerina]

: Hi everyone.
: I've made a class Matrix, which contains a pointer to the data and
some
: methods which all return a copy of the matrix modified in some way.
The
: program works quite well for small matrices, but as matrix dimension
: grows up it crashes in deterministic locations depending on data,
: machine and OS.
: In particular, this happens when I test it with 3000x3000 matrices
: (which take up more than 30MB) under three different machines/OSes.
The
: crash happens when much memory is still available (for example, under
: cygwin, more than 1GB of virtual memory available), and I cannot think
: there is no contiguous space for allocating the matrix.
: I always use references and never pointers to matrices, so Matrix
: objects are not dynamically allocated (the data, however, is allocated
: by the constructor). Could this be a problem with stack/heap
limitations?
: Can someone help me with some idea on how to further investigate the
: problem?

The first thing to check is that you have provided correct
implementations of the destructor, the copy-constructor, and the
copy-assignment operators.

But to make your life easier in the first place, you should consider
using std::vector instead of a raw pointer to store the matrix data.

hth -Ivan

 

[Neclepsio]

Thank you for replying.

The first thing to check is that you have provided correct
implementations of the destructor, the copy-constructor, and the
copy-assignment operators.

Everything is ok. The second?

But to make your life easier in the first place, you should consider
using std::vector instead of a raw pointer to store the matrix data.

I needed raw pointers to use the BLAS library.

Thank you,
Ignazio

 

[John Harrison]

Hi everyone.
I've made a class Matrix, which contains a pointer to the data and some
methods which all return a copy of the matrix modified in some way. The
program works quite well for small matrices, but as matrix dimension
grows up it crashes in deterministic locations depending on data,
machine and OS.
In particular, this happens when I test it with 3000x3000 matrices
(which take up more than 30MB) under three different machines/OSes. The
crash happens when much memory is still available (for example, under
cygwin, more than 1GB of virtual memory available), and I cannot think
there is no contiguous space for allocating the matrix.
I always use references and never pointers to matrices, so Matrix
objects are not dynamically allocated (the data, however, is allocated
by the constructor). Could this be a problem with stack/heap limitations?

The fact that it crashes under three different platforms suggests not.

Can someone help me with some idea on how to further investigate the
problem?

If you can reproduce the problem in a small program then post the code
here. Plently will be willing to take a look.

Thank you,
Ignazio

john

 

[Jacek Dziedzic]

Hi everyone.
I've made a class Matrix, which contains a pointer to the data and some
methods which all return a copy of the matrix modified in some way. The
program works quite well for small matrices, but as matrix dimension
grows up it crashes in deterministic locations depending on data,
machine and OS.
In particular, this happens when I test it with 3000x3000 matrices
(which take up more than 30MB) under three different machines/OSes. The
crash happens when much memory is still available (for example, under
cygwin, more than 1GB of virtual memory available), and I cannot think
there is no contiguous space for allocating the matrix.
I always use references and never pointers to matrices, so Matrix
objects are not dynamically allocated (the data, however, is allocated
by the constructor). Could this be a problem with stack/heap limitations?

There is no "heap limitation", unless you impose it yourself
(hint: try "ulimit -a"). The stack is definitely limited, in
the order of a few MB usually (try "ulimit -s") -- but you only
hit this limitation if you allocate too many/much _local_ variables.

Can someone help me with some idea on how to further investigate the
problem?

If this is an x86 platform, I highly recommend valgrind,
this a utility that will point out many bugs to you, and does
not require a recompile. Other than that you may try
electric fence ("efence"), though this requires a recompile
and is not as useful as valgrind.

And the obvious way out -- recompile your code with debug
information on (usually the -g option), run it under a
debugger (like gdb), let it crash, then backtrace and
check where the problem occurs. Put tons of debug printouts
there and hopefully all will become clear.

HTH,
- J.

 

[John Harrison]

I needed raw pointers to use the BLAS library.

std::vector does not prevent you from getting a raw pointer to the data.

john

 

[Neclepsio]

The stack is definitely limited, in
the order of a few MB usually (try "ulimit -s") -- but you only
hit this limitation if you allocate too many/much _local_ variables.

Yes, I know. In my despair, I was wondering if something like:

Matrix A(3000, 3000);
Matrix B(3000, 3000);
Matrix C(3000, 3000);

or

Matrix B = A.Transpose() * B.SubMatrix(Vector::All, Vector::All);
//(each method returns a new Matrix object)

Would take up much stack space... Anyway, since I get a bad_alloc in
Matrix constructor (which dynamically allocates the space), I don't
think it's the point.

If this is an x86 platform, I highly recommend valgrind,
this a utility that will point out many bugs to you, and does
not require a recompile. Other than that you may try
electric fence ("efence"), though this requires a recompile
and is not as useful as valgrind.

Thank you, I'm downloading it.

And the obvious way out -- recompile your code with debug
information on (usually the -g option), run it under a
debugger (like gdb), let it crash, then backtrace and
check where the problem occurs. Put tons of debug printouts
there and hopefully all will become clear.

Yes, I've done... On the cygwin machine (the most accessable for me) it
crashes on the line:

Matrix temp = P.Transpose();

which just allocates and fills a new Matrix with the same dimensions as
caller (just swapped).



Thank you,
Ignazio

 

[Neclepsio]

If you can reproduce the problem in a small program then post the code
here. Plently will be willing to take a look.

I'll try to write it as small as I can.

Thank you,
Ignazio

 

[Neclepsio]

std::vector does not prevent you from getting a raw pointer to the data.

So, if new can't allocate the memory, why should std::vector be able to?

Thank you,
Ignazio

 

[Alf P. Steinbach]

* Neclepsio:

So, if new can't allocate the memory, why should std::vector be able to?

The std::vector code is more likely to be correct.

 

[Martijn van Buul]

* Alf P. Steinbach:

The std::vector code is more likely to be correct.

Care to elaborate on this hollow claim?

stl::vector is an array with all kinds of nasty strings attached - much
like the rest of this troublesome library. It's an array with an identity
crisis, it's an array with an attitude, but in the end it is still an array.
If you can't new that array yourself, then STL surely can't either. STL *will*
end up doing the new itself.

There's an enormous amount of uncalled for STL zealotry going on, with people
"suggesting" to use STL without even the *slightest* bit of validation to
their claims. The STL is *not* The One Answer. In fact, often, it's exactly
the WRONG answer.

 

[Alf P. Steinbach]

* Martijn van Buul:

* Alf P. Steinbach:

Care to elaborate

no.

 

[John Harrison]

So, if new can't allocate the memory, why should std::vector be able to?

Thank you,
Ignazio

You are changing the subject, you said that you couldn't use std::vector
because you needed raw pointers. I pointed out that this wasn't true. I
didn't say std::vector would cure your problem.

john

 

[Neclepsio]

You are changing the subject, you said that you couldn't use std::vector
because you needed raw pointers. I pointed out that this wasn't true. I
didn't say std::vector would cure your problem.

Ah, ok, I misunderstood your point. Thank you.

 

[Neclepsio]

The std::vector code is more likely to be correct.

This would not solve my problem: understand where I'm wrong.

Ignazio

 

[Alf P. Steinbach]

* Neclepsio:

This would not solve my problem: understand where I'm wrong.

See Ivan Vecerina's reply earlier in the thread.

 

[Kai-Uwe Bux]

This would not solve my problem: understand where I'm wrong.

As long as we don't see the code that is giving you trouble, we won't be
able to help you solve _that_ problem. See the FAQ [5.8] for helpful hints
on how to post about code that does not work.


Best

Kai-Uwe Bux

 

[bnonaj]

Hi everyone.
I've made a class Matrix, which contains a pointer to the data and some
methods which all return a copy of the matrix modified in some way. The
program works quite well for small matrices, but as matrix dimension
grows up it crashes in deterministic locations depending on data,
machine and OS.
In particular, this happens when I test it with 3000x3000 matrices
(which take up more than 30MB) under three different machines/OSes. The
crash happens when much memory is still available (for example, under
cygwin, more than 1GB of virtual memory available), and I cannot think
there is no contiguous space for allocating the matrix.
I always use references and never pointers to matrices, so Matrix
objects are not dynamically allocated (the data, however, is allocated
by the constructor). Could this be a problem with stack/heap limitations?
Can someone help me with some idea on how to further investigate the
problem?

Thank you,
Ignazio

Have you got exception handling so failed allocation will cause a throw?
Is the allocation and deallocation kept within each class? (reset
pointer to NULL and throw if access tried, and have a test)
Never keep more than one permanent pointer to the memory, (get the
pointer only within the function requiring it)
Try adding index checking via class operator[] overloading, and throw
exceptions if outside the range)
Always pass the classes that make up the matrix by reference, avoids a
copy constructor and subsequent destructor which may free memory
inappropriately
Preferably use an STL class like vector which handles memory well
already, (why reinvent the wheel)
Good luck.

JB

 

[Ivan Vecerina]

:* Alf P. Steinbach:
: > The std::vector code is more likely to be correct.
:
: Care to elaborate on this hollow claim?
:
: stl::vector is an array with all kinds of nasty strings attached -
much
: like the rest of this troublesome library. It's an array with an
identity
: crisis, it's an array with an attitude, but in the end it is still an
array.
: If you can't new that array yourself, then STL surely can't either.
: STL *will* end up doing the new itself.
:
: There's an enormous amount of uncalled for STL zealotry going on, with
people
: "suggesting" to use STL without even the *slightest* bit of validation
to
: their claims. The STL is *not* The One Answer. In fact, often, it's
exactly
: the WRONG answer.

One should start by using well-designed and well-tested libraries
before trying to reinvent the wheel. Especially when one is unable
to correctly manages memory.

This is just sound advice to first "get things working".
Where do you see zealotrly in there?

Until proven wrong ("show us the code..."), incorrect memory management
in the OP's code remains the most likely hypothesis.

Ivan

 

[Ivan Vecerina]

: Thank you for replying.
:
: Ivan Vecerina wrote:
: > The first thing to check is that you have provided correct
: > implementations of the destructor, the copy-constructor, and the
: > copy-assignment operators.
:
: Everything is ok.
How do we know?
[ your final comment does nothing to make me trust a bare claim ]

: The second?
The second is: another source of Undefined Behavior.
Third: you are actually allocating more memory than is available.

: > But to make your life easier in the first place, you should consider
: > using std::vector instead of a raw pointer to store the matrix data.
:
: I needed raw pointers to use the BLAS library.

Sure: &vec.front() or &vec


And of course, there are C++ equivalents to BLAS, as others pointed out.

So there are several options to solve the problem.
But if you want a diagnosis, please post the source code.


Kind regards,
Ivan