total array size limitations?

[glennklockwood]

I'm in the process of porting some FORTRAN code to C and am running
into a problem where the program crashes if I set the size of arrays
too large. In the below code, my program runs fine if I set MAXN to
50000 or less, but at 60000 or more, the program immediately crashes
and GDB indicates the error is at the printf line.

What's strange is that I've tried this by separating my 2-dimensional
arrays into a bunch of one-dimensional arrays (eg, xc[60000][6] into
xc0[60000], xc1[60000], xc2[60000], etc) and the same sort of crashing
stopped if I commented out a few of those one-dimensional
declarations. Because of this, I know that the problem isn't with one
giant array, but the total size of the arrays I'm declaring. I assume
this all has to do with some sort of memory limit and a lot of spotty
advice posted over the internet in similar cases have suggested it has
something to do with a 64K stack size, but this doesn't make much
sense to me.

Can anyone shine some light on what exactly is going wrong and how I
can avoid it? By my calculation the size of these declared variables
is a little over 9MB which isn't a huge amount, and the FORTRAN
version of this code works for MAXN in excess of 100000.

If it matters, I am compiling on linux amd64, and this problem has
happened with both GCC 4.1 and Sun's C compiler (SS12).

Thanks much.

glenn

-- Begin code --

#define MAXN 60000
#define MAXLT 10

int main()
{
double ener[100][MAXLT], ihite[100][MAXLT];
double xc[MAXN][6], yc[MAXN][6], zc[MAXN][6];
double aij[MAXLT][MAXLT], bij[MAXLT][MAXLT], amass[MAXLT],
twopi[MAXLT][MAXLT], eta[MAXLT][MAXLT], ze[MAXLT][MAXLT],
rhop[MAXLT][MAXLT], za[MAXLT];
int ltype[MAXN], ibx[MAXN], iby[MAXN], ibz[MAXN];
char k9[16];

double etot, xl, yl, zl, zreft, deltim, temp;
int nmol, istill, nsur, jread, icalc, nbin;

int i,j;
int dblesize, intsize;
FORTRAN_OFFSET_TYPE offset;
FORTRAN_OFFSET_TYPE recsize;
FILE *fp;

int inum[MAXLT], iff;

recsize = 0;
dblesize = sizeof(xl);
intsize = sizeof(i);

printf( " What file?\n" );

.... (it crashes before the printf)

 

[Kenny McCormack]

I'm in the process of porting some FORTRAN code to C and am running
into a problem where the program crashes if I set the size of arrays
too large. In the below code, my program runs fine if I set MAXN to
50000 or less, but at 60000 or more, the program immediately crashes
and GDB indicates the error is at the printf line.

Any question that starts out like this, has only one answer:

The (total) size of automatic objects in C is strictly limited (*).
Use globals and/or dynamic allocation (malloc and friends) if at all
possible.

(*) And don't even think of using the 's word' in this group.

 

[Bartc]

I'm in the process of porting some FORTRAN code to C and am running
into a problem where the program crashes if I set the size of arrays
too large. In the below code, my program runs fine if I set MAXN to
50000 or less, but at 60000 or more, the program immediately crashes
and GDB indicates the error is at the printf line.

double ener[100][MAXLT], ihite[100][MAXLT];
double xc[MAXN][6], yc[MAXN][6], zc[MAXN][6];
double aij[MAXLT][MAXLT], bij[MAXLT][MAXLT], amass[MAXLT],
twopi[MAXLT][MAXLT], eta[MAXLT][MAXLT], ze[MAXLT][MAXLT],
rhop[MAXLT][MAXLT], za[MAXLT];
int ltype[MAXN], ibx[MAXN], iby[MAXN], ibz[MAXN];

Try putting static in front of these big arrays. Or moving the whole lot to
just before the main function, if the names will not clash.

 

[glennklockwood]

The (total) size of automatic objects in C is strictly limited (*).
Use globals and/or dynamic allocation (malloc and friends) if at all
possible.

(*) And don't even think of using the 's word' in this group.

Making everything global certainly did the trick; thank you for the
very quick reply. I suppose I should have checked the books a little
more thoroughly.

 

[Nate Eldredge]

I'm in the process of porting some FORTRAN code to C and am running
into a problem where the program crashes if I set the size of arrays
too large. In the below code, my program runs fine if I set MAXN to
50000 or less, but at 60000 or more, the program immediately crashes
and GDB indicates the error is at the printf line.

What's strange is that I've tried this by separating my 2-dimensional
arrays into a bunch of one-dimensional arrays (eg, xc[60000][6] into
xc0[60000], xc1[60000], xc2[60000], etc) and the same sort of crashing
stopped if I commented out a few of those one-dimensional
declarations. Because of this, I know that the problem isn't with one
giant array, but the total size of the arrays I'm declaring. I assume
this all has to do with some sort of memory limit and a lot of spotty
advice posted over the internet in similar cases have suggested it has
something to do with a 64K stack size, but this doesn't make much
sense to me.

On a typical system, a C program has a fixed-size region of memory set
aside for a stack. This is a very simple memory allocation scheme;
there is a single pointer (normally in a CPU register) marking the
"top". When a function gets called, it grabs some space
from the top of the stack, moving the pointer to indicate the new top,
and uses this space to store its local variables. When it returns, it
restores the pointer to its original value, thus allowing the space to
be reused later.

In your code, since the arrays you declare are local to the function
main(), they are allocated in this manner. But it appears that the
total size of your arrays is bigger than the total size of the stack
(typically a few megabytes). So trying to access those arrays runs off
the end of the stack and causes a crash.

The simplest way to fix it, in your case, would be to declare those
arrays with the `static' keyword. This causes them to be placed in a
different area of memory, which doesn't need to be reused like the
stack. It also means that multiple calls to the function will all
access the same copy of the arrays, rather than each one getting its own
copy, but that is irrelevant here since I presume you don't intend to
have main() recursively call itself. There is also the possibly
convenient side effect that it will cause all the arrays to be
initialized with zeros.

On Unix, you can query and set the default stack size in kbytes with
"ulimit -s" (from a Bourne shell) or "limit stacksize" (from a C shell)
or the rlimit() function (from within the program), so that would be
another possibility. A third option would be to allocate those arrays
dynamically using malloc(), but that is a little awkward (though
certainly possible) to do with multidimensional arrays.

Can anyone shine some light on what exactly is going wrong and how I
can avoid it? By my calculation the size of these declared variables
is a little over 9MB which isn't a huge amount, and the FORTRAN
version of this code works for MAXN in excess of 100000.

I make it more like 36 MB. The C `double' type occupies 8 bytes on
x86/amd64.

If it matters, I am compiling on linux amd64, and this problem has
happened with both GCC 4.1 and Sun's C compiler (SS12).

FWIW, on the Linux amd64 box I have access to, the stack is 10 MB by
default. Yours could be configured differently, of course.

Thanks much.

glenn

-- Begin code --

#define MAXN 60000
#define MAXLT 10

int main()

I'll just mention that modern C standards call for you to use "void" as
the argument list of a function that doesn't take any arguments. So
this should read

int main(void)

(Technically speaking, there seems to be some dispute as to whether
"int main()" is actually illegal or merely obsolescent.)

 

[CBFalconer]

I'm in the process of porting some FORTRAN code to C and am running
into a problem where the program crashes if I set the size of arrays
too large. In the below code, my program runs fine if I set MAXN to
50000 or less, but at 60000 or more, the program immediately crashes
and GDB indicates the error is at the printf line.

Automatic storage is generally assigned on a stack (if the system
uses a stack). Stack size is normally limited. I suggest using
pointers and mallocing the associated storage.

 

[Antoninus Twink]

And, of course, one should check out hashlib and nmalloc.

*snigger*

You forgot ggets(), the swiss army knife of C programming that solves
every problem!

 

[CBFalconer]

Fortran 77 is still alive and kicking. I have to use it myself,
sometimes.

All arguments are passed by reference, and all variables are static.
So code can be very optimal - no hard to detect pointer aliasing
problems.

And no routines are re-entrant, nor is recurrance allowed or
feasible. It is impossible to isolate variables for use in a
single function.

 

[CBFalconer]

And, of course, one should check out hashlib and nmalloc.

Why? I failed to see any application for them in the OPs request.
Please teach us your magnificent method of detecting requirements
in verbiage such as the above.

 

[CBFalconer]

Surely you must be able to find something. You have so often been
able to find reason for recommending them in the most unpromising
of circumstances.

I am not responsible for your inability to detect suitable
applications. I, on the other hand, having designed those modules,
am aware of their abilities and can notice useful applications.

 

[Guest]

<snip>



Why?  I failed to see any application for them in the OPs request.
Please teach us your magnificent method of detecting requirements
in verbiage such as the above.

they are pulling your leg. These are not serious posts.