malloc and realloc

[ravi.cs.2001]

Hi all,

I m relatively new to C. I have few queries related to malloc():

#1. When we perform malloc(), the memory allocated dynamically comes
from the heap area of the process in concern. Well, we then say that
the heap has shrinked. my query is: Is it that the heap physically
does not shrink but the perticular nodes are marked 'ALLOCATED' and
for subsequent calls to malloc() the memory manager remembers them and
does not reference them?

#2. With realloc(), if some pointer 'ptr' is pointing initially to a
perticular position in a buffer (char *buffer) then on performing a
realloc() on this buffer, what will be 'ptr' pointing to?

#3. whats the maximum memory size that we can allocate dynamically by
calling malloc() ?

thanx in advance

Ravs

 

[Richard Heathfield]

(e-mail address removed) said:

Hi all,

I m relatively new to C. I have few queries related to malloc():

#1. When we perform malloc(), the memory allocated dynamically comes
from the heap area of the process in concern.

The Standard doesn't guarantee that. In C terms, the memory is allocated
from the free store, wherever that may be.

Well, we then say that
the heap has shrinked. my query is: Is it that the heap physically
does not shrink but the perticular nodes are marked 'ALLOCATED' and
for subsequent calls to malloc() the memory manager remembers them and
does not reference them?

This is entirely up to the implementation.

#2. With realloc(), if some pointer 'ptr' is pointing initially to a
perticular position in a buffer (char *buffer) then on performing a
realloc() on this buffer, what will be 'ptr' pointing to?

If realloc succeeds, the value should be treated as indeterminate. The
proper way to use realloc is:

tmp = realloc(ptr, newsize);
if(tmp != NULL)
{
ptr = tmp; /* old pointer value should be considered dead, as your
buffer may have moved. */
}
else
{
allocation failed, but at least you still have your old buffer intact
}

#3. whats the maximum memory size that we can allocate dynamically by
calling malloc() ?

The absolute limit for any one call is (size_t)-1 bytes - i.e. the largest
value that can be stored in a size_t, because that's the largest value you
can pass to malloc. But you should not be surprised if implementations do
not allow you to allocate this much. ALWAYS check the result of malloc to
determine whether the call succeeded before trying to use the space you
requested.

 

[CBFalconer]

I m relatively new to C. I have few queries related to malloc():

#1. When we perform malloc(), the memory allocated dynamically comes
from the heap area of the process in concern. Well, we then say that
the heap has shrinked. my query is: Is it that the heap physically
does not shrink but the perticular nodes are marked 'ALLOCATED' and
for subsequent calls to malloc() the memory manager remembers them and
does not reference them?

More or less. The actual mechanism is up to the system involved.
There is no reason a 'heap' has to exist. The system could be
implemented with small boys writing data on a slate. It might be
slow, but there is no speed specification.

#2. With realloc(), if some pointer 'ptr' is pointing initially to a
perticular position in a buffer (char *buffer) then on performing a
realloc() on this buffer, what will be 'ptr' pointing to?

If the pointer is anything other than the value originally returned
by malloc, calloc, or realloc the action is undefined. You could
set off WWIII.

#3. whats the maximum memory size that we can allocate dynamically by
calling malloc() ?

No such. The only indication you have is the failure of
malloc/calloc/realloc by returning NULL. After which you decide
what to do.

--
<http://www.cs.auckland.ac.nz/~pgut001/pubs/vista_cost.txt>

"A man who is right every time is not likely to do very much."
-- Francis Crick, co-discover of DNA
"There is nothing more amazing than stupidity in action."
-- Thomas Matthews

 

[santosh]

Hi all,

I m relatively new to C. I have few queries related to malloc():

#1. When we perform malloc(), the memory allocated dynamically comes
from the heap area of the process in concern. Well, we then say that
the heap has shrinked. my query is: Is it that the heap physically
does not shrink but the perticular nodes are marked 'ALLOCATED' and
for subsequent calls to malloc() the memory manager remembers them and
does not reference them?

C doesn't talk about heaps. As far as a portable C program is
concerned, the *alloc() functions are to be treated as a black box.
They either succeed or fail. The details of their internal operation
are not specified by the C standard. Different systems may use
different strategies, even varying from call to call. In a portable
program, the calling code need not bother about such things.

#2. With realloc(), if some pointer 'ptr' is pointing initially to a
perticular position in a buffer (char *buffer) then on performing a
realloc() on this buffer, what will be 'ptr' pointing to?

realloc() may physically move the block during resizing, so any
pointers to the block become invalidated after a successful call to
malloc(). They'll need to be updated. Note though that if realloc()
fails, the original memory is left untouched and remains valid to use.

#3. whats the maximum memory size that we can allocate dynamically by
calling malloc() ?

Since malloc() takes a parameter of type size_t, the absolute largest
value you can pass to it, without causing wrap-around, is SIZE_MAX
bytes, defined in stddef.h.

 

[Serve Laurijssen]

More or less. The actual mechanism is up to the system involved.
There is no reason a 'heap' has to exist. The system could be
implemented with small boys writing data on a slate. It might be
slow, but there is no speed specification.

I really really wanna see a platform like that. And when the boys have
weekend malloc returns NULL I take it?

 

[santosh]

santosh wrote:

realloc() may physically move the block during resizing, so any
pointers to the block become invalidated after a successful call to
malloc().

That should read:
[ ... ] after a successful call to realloc().

 

[CBFalconer]

I really really wanna see a platform like that. And when the boys
have weekend malloc returns NULL I take it?

No, the malloc call just doesn't return until Monday NULL is
for when all the boys slates are in use. BTW, a pointer here
resolves to a boys name, and cannot meaningfully be converted
to/from an int.

--
<http://www.cs.auckland.ac.nz/~pgut001/pubs/vista_cost.txt>

"A man who is right every time is not likely to do very much."
-- Francis Crick, co-discover of DNA
"There is nothing more amazing than stupidity in action."
-- Thomas Matthews

 

[David T. Ashley]

I really really wanna see a platform like that. And when the boys have
weekend malloc returns NULL I take it?

No. Such systems also maintain a large collection of trained rats. When
the boys have weekend, the trained rats do the same job. As part of the
weekend activities, the boys have to bring back food and treats for the
trained rats.

In fact, this is the way my *nix box actually works internally. The system
documentation makes reference to CPU, silicon wafer, etc., but I know
better.

 

[Malcolm McLean]

The absolute limit for any one call is (size_t)-1 bytes - i.e. the largest
value that can be stored in a size_t, because that's the largest value you
can pass to malloc. But you should not be surprised if implementations do
not allow you to allocate this much. ALWAYS check the result of malloc to
determine whether the call succeeded before trying to use the space you
requested.

Actually Windows guarantees that allocations of under 8K or something like
that will succeed, unless the program is responding to a request to shut
down due to lack of memory, in which case 64K of allocation or so is
guaranteed, but not an infinite supply of 64K chunks.

So on a modern operating system you don't need to check small mallocs. You
do if code is to be completley portable, of course.

 

[matevzb]

Actually Windows guarantees that allocations of under 8K or something like
that will succeed, unless the program is responding to a request to shut
down due to lack of memory, in which case 64K of allocation or so is
guaranteed, but not an infinite supply of 64K chunks.

I'll have to ask for chapter&verse on this one. If you're referring to
Windows malloc(), the manual explicitly states "Always check the
return from malloc, even if the amount of memory requested is small".
Furthermore, malloc() uses HeapAlloc() to get the memory: "Memory
allocated by HeapAlloc is not movable. Because the memory is not
movable, the heap can become fragmented". In theory, it could become
so fragmented, that you wouldn't be able to get 8K.

So on a modern operating system you don't need to check small mallocs. You
do if code is to be completley portable, of course.

The conclusion is drawn from Windows, can you be sure this should
apply to all other systems as well?

 

[Flash Gordon]

Malcolm McLean wrote, On 27/01/07 20:16:

Actually Windows guarantees that allocations of under 8K or something like
that will succeed, unless the program is responding to a request to shut
down due to lack of memory, in which case 64K of allocation or so is
guaranteed, but not an infinite supply of 64K chunks.

<OT>
So on Windows you are guaranteed to be able to cause programs to be shut
down by repeatedly allocating blocks just under 8K...

Hmm. Another reason to avoid Windows I think. At least on Linux you can
(I believe) turn off lazy allocation.

So on a modern operating system you don't need to check small mallocs. You
do if code is to be completley portable, of course.

And you also know that small allocations will never fail on modern
versions of Linux (including any patched versions of libraries/kernel
some distributions may have), AIX, FreeBSD, NetBSD...

You also know that they will never change this behaviour? Microsoft have
a habit of changing things, and they *have* changed things such that
they break badly behaved programs in the past, and a program that does
not check the value returned by malloc *is* badly behaved.

Far better to actually right correct code then you do not have to worry
about such things.

 

[Richard Heathfield]

Malcolm McLean said:

Actually Windows

I thought we were discussing C, not Windows.

guarantees that allocations of under 8K or something like
that will succeed, unless the program is responding to a request to shut
down due to lack of memory, in which case 64K of allocation or so is
guaranteed, but not an infinite supply of 64K chunks.

So what you seem to be saying is that there'll always be memory available
except when there isn't. Well, gosh!

So on a modern operating system you don't need to check small mallocs.

Yes, you do. As you said yourself, the system might be trying to shut
processes down due to lack of memory. I know of no standard C way to detect
such a state, but one can at least detect the lack of memory by checking
the return value of malloc.

You do if code is to be completley portable, of course.

I see all these people saying "you don't need to defend against foo, because
that situation never arises in practice", and I see all these programs
falling over right, left, and centre because, in the Real World, foo
happens. You do as you will, of course, but I will continue to check that
I've succeeded in acquiring a resource before I try to use it.

 

[Keith Thompson]

Actually Windows guarantees that allocations of under 8K or something like
that will succeed, unless the program is responding to a request to shut
down due to lack of memory, in which case 64K of allocation or so is
guaranteed, but not an infinite supply of 64K chunks.

I think what you're actually saying is not that allocations will
always succeed, but that they fail by shutting down your program (or
the OS?) rather than by properly returning a null pointer. Ick.

 

[Malcolm McLean]

I think what you're actually saying is not that allocations will
always succeed, but that they fail by shutting down your program (or
the OS?) rather than by properly returning a null pointer. Ick.

That's about the size of it.
I've outsourced all my computing to China, so when they run out of those
little wooden blocks they can always resort to grains of rice. Some of my
Chinamen can even write the Lord's Prayer on a rice grain.
However even they will ultimately run out of resources if, say, my program
is allocating in an infinite loop. The question is whether they say "sorry
no more rice" or send a messenger a few hours before saying "Sir, our rice
situation is getting desperate".

 

[tphipps]

realloc() on this buffer, what will be 'ptr' pointing to?If realloc succeeds, the value should be treated as indeterminate. The

proper way to use realloc is:

tmp = realloc(ptr, newsize);
if(tmp != NULL)
{
ptr = tmp; /* old pointer value should be considered dead, as your
buffer may have moved. */}else
{
allocation failed, but at least you still have your old buffer intact

}

If you are always going to be that cautious, why not save typing by
wrapping it in a macro, e.g.

#include <stdlib.h>
#define realloc(x,y) ( (x=realloc(x,y)) ? x : ( abort(), NULL ) )

 

[santosh]

If you are always going to be that cautious, why not save typing by
wrapping it in a macro, e.g.

#include <stdlib.h>
#define realloc(x,y) ( (x=realloc(x,y)) ? x : ( abort(), NULL ) )

Macros are not as flexible as native language constructs.
Additionally, hardwiring the logic into a macro is not sensible. For
example, one may not always want to abort() on memory allocation
failure.

 

[matevzb]

If you are always going to be that cautious, why not save typing by
wrapping it in a macro, e.g.

#include <stdlib.h>
#define realloc(x,y) ( (x=realloc(x,y)) ? x : ( abort(), NULL ) )

Because:
a) If realloc() fails, you still lose the original pointer.
b) It may be okay to abort() for smaller programs, but surely not in
other cases.
c) Wrapping multiple function calls in one macro makes the code less
readable and/or more difficult to debug.

 

[CBFalconer]

.... snip ...

If you are always going to be that cautious, why not save typing
by wrapping it in a macro, e.g.

#include <stdlib.h>
#define realloc(x,y) ( (x=realloc(x,y)) ? x : ( abort(), NULL ) )

Please don't remove attribution lines from material you quote.
Those are the lines that say "Joe wrote:" at the beginning of the
article.

Terrible idea. Illegal to start with (#defining system routines)
although it will usually work. However normally you don't want to
abandon everything when realloc fails, since all your data is still
intact. A better idea is:

void *tmp;
...
if (tmp = realloc(x, y)) x = tmp;
else {
/* whatever you need to recover */
}

Any time you see "x = realloc(x, size);" you have a potential
memory leak, and you know the coder is probably inexperienced.
Exceptions exist.

 

[tphipps]

If you are always going to be that cautious, why not save typing by

a) Ifrealloc() fails, you still lose the original pointer.

So there's a memory leak for some tiny number of clock cycles until
abort()
is called... big deal.

b) It may be okay to abort() for smaller programs, but surely not in
other cases.

"E.g." means "for example". As you admit, this example already has
uses -
in many small single-purpose programs it simply doesn't make sense to
carry on if you can't allocate memory.

With a little initiative you can modify the macro... e.g. have a
global variable
to store the original pointer, and replace abort() by some cleanup
code.
If you do it right, you'll be able to transparently handle allocation
failures
in-place too, on the few occasions when that's really beneficial.

c) Wrapping multiple function calls in onemacromakes the code less
readable and/or more difficult to debug.

That's personal opinion. Consistency and clear documentation are what
make code easier to debug. I personally find source files with
hundreds
of if-clauses for (m|re)alloc failure scattered everywhere far harder
to read.

 

[santosh]

(e-mail address removed) wrote:

Please don't snip attributions. It makes it more difficult to find out
who said what.

So there's a memory leak for some tiny number of clock cycles until
abort() is called... big deal.

In many programs, when a call to *alloc() fails, it may be possible to
continue, or gracefully terminate, by trying to allocate a smaller
amount, or by free()'ing non-critical memory and attempting allocation
again. The program might want to print debug messages or flush one or
more buffers, close files etc. Simply calling abort() is inappropriate
except for very small or specialised programs.

"E.g." means "for example". As you admit, this example already has uses - in many small
single-purpose programs it simply doesn't make sense to carry on if you can't allocate memory.

Fair enough, but I doubt that Richard can afford to be so cavalier as
to call abort().

With a little initiative you can modify the macro... e.g. have a
global variable to store the original pointer, and replace abort() by some cleanup
code. If you do it right, you'll be able to transparently handle allocation
failures in-place too, on the few occasions when that's really beneficial.

Trying to do too much within a macro is inadvisable. If your logic for
allocation and dealing with associated failures is more than a line or
two, it makes sense to wrap the code inside a function, which is more
robust than a macro.

That's personal opinion. Consistency and clear documentation are what
make code easier to debug. I personally find source files with hundreds
of if-clauses for (m|re)alloc failure scattered everywhere far harder to read.

Indeed, that's why they should be encapsulated in one or more
functions. Increases readability and reduces code redundancy.