Failing Malloc

J

James Kuyper

On 09/12/2012 08:07 PM, Ian Collins wrote:
....
I use the OS provided stack trace functions to build a stack trace to
include in the exception, so there's no need to pick anything use up
along the way.

My code is allowed to assume support for any feature mandated by POSIX
1003.2-1992 (except for a long list of prohibited functions - but none
of those seem relevant), but is not otherwise allowed to depend upon
OS-specific features. Do you know if any such functions were included in
that version of POSIX?

If not, can you at least identify the OS you're using, and the specific
stack trace functions you're talking about for that OS? I'd like to know
how they work - off-hand, functionality that would be an adequate
substitute for what I'm currently doing seems implausible.
 
I

Ian Collins

On 09/12/2012 08:07 PM, Ian Collins wrote:
....

My code is allowed to assume support for any feature mandated by POSIX
1003.2-1992 (except for a long list of prohibited functions - but none
of those seem relevant), but is not otherwise allowed to depend upon
OS-specific features. Do you know if any such functions were included in
that version of POSIX?

If not, can you at least identify the OS you're using, and the specific
stack trace functions you're talking about for that OS? I'd like to know
how they work - off-hand, functionality that would be an adequate
substitute for what I'm currently doing seems implausible.

I think most of this stuff is OS specific and there's a big whole in
POSIX where it should be.

I mainly use Solaris which has a family of stack reading functions:

http://docs.oracle.com/cd/E23824_01/html/821-1465/walkcontext-3c.html#scrolltoc

Googling for "solaris walkcontext" will find the Linux near equivalents.
 
J

James Kuyper

On 09/13/2012 03:37 PM, Ian Collins wrote:
....
I think most of this stuff is OS specific and there's a big whole in
"hole"?

POSIX where it should be.

I mainly use Solaris which has a family of stack reading functions:

http://docs.oracle.com/cd/E23824_01/html/821-1465/walkcontext-3c.html#scrolltoc

Googling for "solaris walkcontext" will find the Linux near equivalents.

Those functions seem to be missing something, that I expected to be
missing, that would be necessary in order to produce output containing
information comparable to that produced by my current error messages. I
compared that information to an "annotated stack trace". The "annotated"
is important: it refers to the fact that for every function in the call
stack, the error messages include the values of all scalar non-pointer
parameters, and the text of any strings pointed at by character pointer
input parameters.

Less consistently, if an input pointer parameter points at a struct
containing a single member that identifies what the struct contents
represent, the message will sometimes include the value of that member
(or of the string it points at). Example: one of the packages I use
provides a file handle structure that allows me to print out
file_handle->filename.

gdb can provide similar information when debugging a program compiled
with -g, but when I followed that URL, it was not clear to me that the
walkcontext routines have any features that would enable such annotation.

Another aspect of the "annotation" is that the error messages also
contain the value of the loop index for any loop containing the code
that generates the error message (or, more usefully, a string associated
with that index value, if one is available), but that's functionality I
could not reasonably expect a stack-walking routine to provide.
 
E

Eric

If your program has an alternative way of working which uses less memory
(say, for example, a slower method) you could put in a new malloc asking
for less memory and see if that works. Or you could allocate more memory
than you need early in the program, so you've got a bit already to hand
when you run a crucial bit of the program. Or you could tell the user
that unfortunately the program can't do what was requested.

Getting more memory from the stack seems a little risky, as there are
normally less checks whether it worked. Swapping your own program's data
out to disk might be a possibility if you keep track of what you have
done. Swapping other programs out to disk is system-specific, if you
want to try this then other newsgroups will be able to give better help.

What I mean is, imagine a very large allocation say 4MB. Maybe that's not
available in one chunk, however 4 seperate 1MB allocations could succeed.
That sort of thing.

By stack I mean essentially a local array, i.e.
func()
{
char buf[1000]; /* <--- example */
/* buf can never be Null */
}

Note, that in this case, Null can never be returned for the array.

Thanks
 
E

Eric Sosman

[...]
By stack I mean essentially a local array, i.e.
func()
{
char buf[1000]; /* <--- example */
/* buf can never be Null */
}

Note, that in this case, Null can never be returned for the array.

Probably works for 1000 chars, probably works for 10000,
but what if you need 10MB? Lots of systems devote only a
limited amount of memory for `auto' variables, and if your
program exceeds the limit it's likely to die even more quickly
than if malloc() returned `NULL' (not `Null'). That's what
Paul N meant by "there are normally less checks whether it
worked:" malloc() tells you when it fails and gives you a
choice about how to proceed, but stack explosion doesn't.

Also, I'm a little worried about that word "returned." If
you intend to have func() return a pointer to buf[] or any
part of it, you're in deep trouble: buf[] ceases to exist as
soon as func() returns, and if a caller tries to use a pointer
to the non-existent buf[] there will be fertilizer on the fan.
 
J

James Kuyper

On 09/13/2012 05:21 PM, William Ahern wrote:
....
... most of the time the
execution paths to any particular point in application code can usually be
counted on a single hand missing 4 fingers).

Not if it's a utility routine that's called from many different
locations in the code. In the program I'm working on right now, most of
the code seems to be in routines like that. Only routines from the top
two levels of the program are called from unique locations.
 
J

James Kuyper

What I mean is, imagine a very large allocation say 4MB. Maybe that's not
available in one chunk, however 4 seperate 1MB allocations could succeed.
That sort of thing.


Here's code that shows how to do something like that:
#define BLOCK_SIZE 0x40000000
#define NUM_BLOCKS 4
#define BUFFER_SIZE (NUM_BLOCKS * BLOCK_SIZE)

// Note: on many systems the calculation of BUFFERSIZE will overflow

int func(void)
{
void *blocks[NUM_BLOCKS] = {0};
int numb;
blocks[0] = malloc(BUFFER_SIZE);
// Even if the calculation doesn't overflow, the conversion to
// size_t may cause unsigned wrap-around. I'll leave it as an
// exercise for the reader to figure out how to deal with those
// possibilities.

if(blocks[0])
{
for(numb=0; numb<NUM_BLOCKS-1; numb++)
blocks[numb+1] = blocks[numb] + BLOCK_SIZE;
}
else
{
for(numb=0; numb<NUM_BLOCKS && blocks[numb]=malloc(BLOCK_SIZE);
numb++);
}
if(numb == NUM_BLOCKS)
{
// code that works with blocks[];
// the key thing is that this code doesn't care whether or not
// the block of memory pointed at by block[1] immediately
// follows the block pointed at by block[2].
}
else
{
// error handling for insufficient memory
}
while(numb>0)
free(blocks[--numb]);
}

However, if you can write your code to work with 4 smaller blocks, it's
probably best to always allocate 4 blocks, rather than one large one.
By stack I mean essentially a local array, i.e.
func()
{
char buf[1000]; /* <--- example */
/* buf can never be Null */
}

Note, that in this case, Null can never be returned for the array.

OK, that's a different approach. Here's some code that shows how
something like that could be done:

int called(void*); // This function does the actual work.

int caller(void)
{
void *p = malloc(BUFFER_SIZE);
if(p)
return called(p);
else
{
char buf[BUFFER_SIZE];
return called(buf);
}
}

Note: if p is non-null, it's guaranteed to be correctly aligned for any
object type. No such guarantee applies to buf. If called() makes use of
the pointer in any way that requires a special alignment, you need to do
something to make sure that 'buf' is suitably aligned. In C99, you
should declare the parameter of called() as being a pointer to an object
with suitable alignment requirements, and declare buf to be an array of
that same object type. C 2011 adds some new alignment-oriented features
that allow for more exotic ways of achieving that same purpose, but this
is still the most appropriate method to use.

On many systems, the memory pointed at by 'p' comes out of the heap,
while 'buf' resides on the stack, but the C standard doesn't use those
terms, and does not require that such things even exist. The stack and
the heap could be two separate memory pools, or they could be different
ends of the same block of memory; more complicated alternatives exist as
well. If they are from a single block of memory, not having enough heap
for malloc() to succeed could correspond to not having enough stack for
the allocation of buf to occur. If that's the case, the behavior of that
program is undefined, and there's nothing you can do to prevent that,
anticipate that, or deal with it.
 
E

Edward A. Falk

If malloc is returning null, there isn't a lot you can do.

Pretty much this.
You can wait a short interval and pray that more memory becomes available
(perhaps some other program just exited, freeing up a lot of memory.)

You can free some memory that your program is using elsewhere.

This is your best bet. Does your app have an "undo" buffer? If so,
flush it. Is it caching anything? Flush the caches. Does the
application allow the user to have multiple files or windows opened?
If so, abort the operation and suggest to the user that they close any
unused windows or files.

If those options aren't available to you, you're pretty much boned.

Or as other posters have suggested, have an alternative algorithm
available that uses less memory, or just fail that particular operation
and hope that the user can live with it.
 
P

Philip Lantz

James said:
On 09/12/2012 06:28 PM, Ian Collins wrote:
...

On my new task, I'm actually using C++, and I've been trying to figure
out how to make use of exceptions to produce the same kind of error
output I get from the my C programs. It's a fantasy, since the new group
I'm working with has it's own well-entrenched error handling conventions
that would not mix well with my preferred style.

However, when I did indulge that fantasy, I found that one key problem
was that exceptions discard information about where they came from, as
the call stack unwinds. Unless I catch and rethrow around every function
call that could throw an exception (including the implicit function
calls due to ctors, dtors, and operator overloads), it can't duplicate
the information generated by my error messages.

If I did catch and rethrow everywhere, it would end up looking very
similar to my current C code, which constantly checks for error codes.
More than one person has looked at my code and come away with the
impression that its primary purpose was error handling. One person even
submitted a test plan for a module that triggered every single error
message that could be generated by that module, but failed to cover
normal processing.

I'd hoped to find a way to use C++ exceptions to reduce the error
message clutter, but I can't figure out how to do so without a
corresponding reduction in the amount of information conveyed by the
error messages.


Perhaps you could define a class something like this:

class backtrace_handler
{
// members that hold the information that would be
// useful in a backtrace
unsigned uns[4];
char *ptr[4];
double dbl[4];

// a member that indicates which fields were initialized
unsigned valid;

public:
backtrace_handler(int a)
: valid(0x0001)
{ uns[0] = a; }
backtrace_handler(char *a)
: valid(0x0002)
{ ptr[0] = a; }
backtrace_handler(double a)
: valid(0x0003)
{ dbl[0] = a; }
backtrace_handler(char *a, double b)
: valid(0x0032)
{ ptr[0] = a; dbl[1] = b; }
backtrace_handler(double a, int b, char *c)
: valid(0x0213)
{ dbl[0] = a; uns[1] = b; ptr[2] = c; }
<additional constructors for different number/types of useful info>

// Here's the key part:
~backtrace_handler()
{
//dump the members that were initialized
int i = 0;
while (valid != 0) {
switch (valid & 0xf)
{
case 1: printf(..., uns); break;
case 2: printf(..., ptr); break;
...
}
valid >>= 4;
i++;
}
}
};

At the beginning of each function, declare an instance:

void f(int a, char *b)
{
const backtrace_handler bth(a, b);

...
}

Obviously you would want to define the members and valid bits more
cleanly, but I hope you get the idea. The destructor could be as complex
as you need, to get the pretty output you want, but the constructors
should be as simple as possible.

The obvious downside is that there is some cost to initialize the
structure in each function. However, from your description, it seems
that that may be minimal; it will essentially be simply spilling some of
the function parameters to the stack, plus a single constant indicating
which members were initialized and the order you want them displayed in
the backtrace. (If the compiler's really smart, it might even be able to
reuse the parameters that were spilled to the backtrace object instead
of spilling them separately if there is register pressure. I'm not sure
I've ever seen a compiler that was that smart, but I have seen some do
some pleasantly surprising things.)

Oh, I guess you need to add the function name, too.

Unfortunately, I can't think of a way to get the line number within the
function where the failed call occurred. But it's almost 3am; maybe I'll
think of something tomorrow. :)
 
I

Ian Collins

On Linux there are two easy ways to get a stack trace:

1) glibc's backtrace() from<execinfo.h>.

That's good to know, Solaris has the same header/interface.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads


Staff online

Members online

Forum statistics

Threads
474,078
Messages
2,570,570
Members
47,204
Latest member
MalorieSte

Latest Threads

Top