size_t or int for malloc-type functions?

[CBFalconer]

CBFalconer wrote:
.... snip ...

Then why don't you make it

if (!n || ((size_t)-1) / n >= s)

instead?

I haven't thought it through, but I think there are cases involving
the truncation effects of the division where the == condition may
or may not be an error. The actual needs of malloc in terms of
alignment and tracking overhead are not generally known outside of
the malloc code proper. This is one more reason for putting calloc
in the malloc module. We can afford the extra overhead of a
thorough check there because there is all that overhead of doing
the memset anyhow, and the detailed needs of malloc are known.

When I thrash this out I will update my published nmalloc issue.
Meanwhile this constitutes a warning to users of the above code. I
think it is fine for the actual limits used in nmalloc on DJGPP,
but it affects the portability of that code. It is another hidden
assumption, to be either avoided or documented.

 

[CBFalconer]

You can find many of them six feet under the ground.

That last error was obviously fatal.

 

[kuyper]

pete wrote:
....

If calloc can't return a pointer to
"space for an array of nmemb objects,
each of whose size is size"
then it should return a null pointer,
rather than a pointer to
((size_t)nmemb * (size_t)size) bytes of memory.

In the description you quote, nmemb and size represent the parameters
of calloc(), not it's arguments (without implying that the parameters
are required to have those names). As a result, conversion to size_t is
a no-op, and both mounts of memory are the same. If calloc() can't
allocate the first amount, it can't allocate the second one, either,
and must therefore return a null pointer.

 

[kuyper]

pete wrote:
....

If calloc can't return a pointer to
"space for an array of nmemb objects,
each of whose size is size"
then it should return a null pointer,
rather than a pointer to
((size_t)nmemb * (size_t)size) bytes of memory.

In the description you quote, nmemb and size represent the parameters
of calloc(), not it's arguments (without implying that the parameters
are required to have those names). As a result, conversion to size_t is
a no-op, and both mounts of memory are the same. If calloc() can't
allocate the first amount, it can't allocate the second one, either,
and must therefore return a null pointer.

 

[pete]

pete wrote:
...

In the description you quote, nmemb and size represent the parameters
of calloc(), not it's arguments (without implying that the parameters
are required to have those names).
As a result, conversion to size_t is
a no-op, and both mounts of memory are the same. If calloc() can't
allocate the first amount, it can't allocate the second one, either,
and must therefore return a null pointer.

I put the cast in there to emphasize that the result
of the multiplication would be according to the rules
governing arithmetic operations on size_t operands.

 

[kuyper]

I put the cast in there to emphasize that the result
of the multiplication would be according to the rules
governing arithmetic operations on size_t operands.

My apologies: for some reason I was thinking that you were talking
about passing signed arguments to calloc(). I should have paid more
attention to what you actually wrote.

Yes, the current spec for calloc() already requires that it either
allocate space that many objects of that size, or return null, whether
or not nmemb*size correctly calculates the amount of space required. An
implementation that simply multiplied those numbers and passed the
product to malloc() would therefore not be conforming. This has been a
long and hotly contested thread, so I can't be sure, but I don't think
there's anyone who's expressed an opinion in conflict with that
statement.

Richard Heathfield has correctly pointed out that calloc() can't test
for overflow in the calculation of nmemb*size, because that product
can't overflow. But that's just a quibble about terminology. That
product can fail to correctly calculate the amount of space needed to
store nmemb object of the specified size, and calloc() not only can,
but must, either check for whether or not that would be the case, or
perform the multiplication using an integer type sufficiently wide to
ensure that the product is does give the amount of space required.

 

[Richard Heathfield]

(e-mail address removed) said:

Yes, the current spec for calloc() already requires that it either
allocate space that many objects of that size, or return null, whether
or not nmemb*size correctly calculates the amount of space required. An
implementation that simply multiplied those numbers and passed the
product to malloc() would therefore not be conforming. This has been a
long and hotly contested thread, so I can't be sure, but I don't think
there's anyone who's expressed an opinion in conflict with that
statement.

I believe you're correct that no such opinion has been expressed.

Richard Heathfield has correctly pointed out that calloc() can't test
for overflow in the calculation of nmemb*size, because that product
can't overflow. But that's just a quibble about terminology.

I must, however, disagree here. It's not *just* a quibble about terminology.
It's a quibble about terminology. If we aren't all using the same meanings
of words, the potential for misunderstanding increases vastly. Terminology
*matters*, because only by using the same language can we hope to
understand each other, and it is only when we use the same meanings of
words that we can intelligently disagree (or even agree!) with each other.

 

[Joe Wright]

pete said:



Certainly true.

If I can sneak in here, it seems to me that the allocator will attempt
its task with one argument of size_t. It doesn't know or care the type
of the size argument passed to it. The argument is automagically
converted to size_t as the functions are called.

There is no checking of the magnitude of 'size_t size'. The allocator
takes what it gets and tries to do it. Success yields a pointer to
allocated memory. Failure yields NULL.

As we programmers call our *alloc functions, it is our job to ensure
size is reasonable. The functions will succeed or fail by their own
lights and even on success will not tell us the actual size of the
allocation.

The Standard library does not wipe our noses for us. Upthread it is
noted that 65521 * 65552 is 65296 in 32 bits. The 'normal' product of
4295032952 requires 33 bits. If we give calloc(65521, 65552) and get a
pointer to 65296 bytes while expecting a pointer to 4295032952 bytes,
then we are wrong. The allocator did just what it should.

 

[av]

C Standard is wrong on that
you're hardly wrong

in the fact size_t overflow when mod is so right

 

[jacob navia]

Joe Wright a écrit :

The Standard library does not wipe our noses for us. Upthread it is
noted that 65521 * 65552 is 65296 in 32 bits. The 'normal' product of
4295032952 requires 33 bits. If we give calloc(65521, 65552) and get a
pointer to 65296 bytes while expecting a pointer to 4295032952 bytes,
then we are wrong. The allocator did just what it should.

No. This is wrong. the allocator should return NULL,
not a pointer to an object that is undersized.

 

[santosh]

Joe Wright a écrit :

No. This is wrong. the allocator should return NULL,
not a pointer to an object that is undersized.

I don't think that the standard library should try to interpret the
intent of the programmer. Rather, the programmer should know what he's
doing, (in this case the modulo 2^n nature of unsigned integers in C).

If hand-holding is needed we have a pleothra of other languages.

 

[Richard Heathfield]

santosh said:

I don't think that the standard library should try to interpret the
intent of the programmer. Rather, the programmer should know what he's
doing, (in this case the modulo 2^n nature of unsigned integers in C).

No, you've missed Mr Navia's point, which is that calloc is *required* by
the Standard to return either a pointer to at least the memory actually
requested - i.e. enough storage for at least 65521 objects each 65552 bytes
in size, OR if it can't do that, to return NULL.

The Standard does not give calloc licence to do a quick multiply, malloc,
and memset, if so doing means that the request is not properly met.

 

[Cesar Rabak]

Joe Wright escreveu:

If I can sneak in here, it seems to me that the allocator will attempt
its task with one argument of size_t. It doesn't know or care the type
of the size argument passed to it. The argument is automagically
converted to size_t as the functions are called.

This is correct, but...

There is no checking of the magnitude of 'size_t size'. The allocator
takes what it gets and tries to do it. Success yields a pointer to
allocated memory. Failure yields NULL.

OK, we agree on a such definition of failure, which is the Standard
gives us, now:

As we programmers call our *alloc functions, it is our job to ensure
size is reasonable. The functions will succeed or fail by their own
lights and even on success will not tell us the actual size of the
allocation.

the theory seems correct until...

The Standard library does not wipe our noses for us. Upthread it is
noted that 65521 * 65552 is 65296 in 32 bits. The 'normal' product of
4295032952 requires 33 bits. If we give calloc(65521, 65552) and get a
pointer to 65296 bytes while expecting a pointer to 4295032952 bytes,
then we are wrong. The allocator did just what it should.

Here, I think you mistakenly considering the use of malloc which takes
only a 'size_t size' argument.

calloc takes _two_ each one of size_t type. So getting a pointer to an
allocated chunk of memory to 65296 which is far lees than 4295032952 is
plainly wrong.

The issue that certain combinations of arguments lead to non available
memory is akin the EDOM errors with certain mathematical functions.

calloc implementations that not return a NULL should be considered
faulty and repaired asap.

 

[Cesar Rabak]

santosh escreveu:

I don't think that the standard library should try to interpret the
intent of the programmer. Rather, the programmer should know what he's
doing, (in this case the modulo 2^n nature of unsigned integers in C).

If hand-holding is needed we have a pleothra of other languages.

This is not hand-holding is hiding a bug. Consider the following:

A program uses calloc and the programmer used the idiomatic, and all the
available to him, technique of checking the returned pointer to non
nullnes.

During the operation of the program memory beyond the actually allocated
space may be (de)referenced, depending on the platform it may be noticed
as an error or may become a fault, perhaps a security threat.

jacob when started this thead used the funny subject name exactly
because of this kind of (expected to him) reaction.

Is it so hard to accept that a buggy implementation is there at the
corner, even more when we can see it is not Standard compliant?

 

[santosh]

santosh said:

No, you've missed Mr Navia's point, which is that calloc is *required* by
the Standard to return either a pointer to at least the memory actually
requested - i.e. enough storage for at least 65521 objects each 65552 bytes
in size, OR if it can't do that, to return NULL.

The Standard does not give calloc licence to do a quick multiply, malloc,
and memset, if so doing means that the request is not properly met.

You're right. The standard does require calloc() to return a pointer to
the amount of memory specified by the combination of it's two
parameters or to return a null pointer.

In which case I suppose calloc() should check whether the total memory
requested can be represented by the size_t type without causing
wrap-around. I suppose it's also free to use a larger internal type,
like say uintmax_t, to calculate the product of it's arguments, but the
standard does specify that the largest single dynamic object shall be
SIZE_MAX or less bytes. So even if it can calculate the memory
requested correctly, it'll still be forced to return NULL for amounts
greater than SIZE_MAX.

 

[Keith Thompson]

There is no checking of the magnitude of 'size_t size'. The allocator
takes what it gets and tries to do it. Success yields a pointer to
allocated memory. Failure yields NULL.
Agreed.

As we programmers call our *alloc functions, it is our job to ensure
size is reasonable. The functions will succeed or fail by their own
lights and even on success will not tell us the actual size of the
allocation.
Agreed.

The Standard library does not wipe our noses for us. Upthread it is
noted that 65521 * 65552 is 65296 in 32 bits. The 'normal' product of
4295032952 requires 33 bits. If we give calloc(65521, 65552) and get a
pointer to 65296 bytes while expecting a pointer to 4295032952 bytes,
then we are wrong. The allocator did just what it should.

No, if calloc(65521, 65552) allocates only 65296 and returns a
non-null pointer, then the implementation of calloc() is broken.

The standard says:

The calloc function allocates space for an array of nmemb objects,
each of whose size is size. The space is initialized to all bits
zero.

It *doesn't* say that calloc() multiplies nmemb by size, yielding a
size_t result, to determine how many bytes to allocate; it simply says
that it allocates space for an array of objects with the specified
length and element size. If it can't do that, by whatever means,, it
needs to return a null pointer.

(In most, perhaps all, implementations, calloc() isn't going to be
able to allocate more than SIZE_MAX bytes.)

malloc() is different. If I call malloc(65521 * 65552) on a system
with a 32-bit size_t (and no padding bits), malloc() must attempt to
allocate 65296 bytes. That's because *I* explicitly did the
multiplication, and 65296 is the correct result of that
multiplication; malloc() can't know that I intended 4295032952.

 

[Keith Thompson]

You're right. The standard does require calloc() to return a pointer to
the amount of memory specified by the combination of it's two
parameters or to return a null pointer.
Right.

In which case I suppose calloc() should check whether the total memory
requested can be represented by the size_t type without causing
wrap-around.

Right. In fact, it *must* do so (though there are implementations
that don't, and are therefore buggy).

I suppose it's also free to use a larger internal type,
like say uintmax_t, to calculate the product of it's arguments, but the
standard does specify that the largest single dynamic object shall be
SIZE_MAX or less bytes. So even if it can calculate the memory
requested correctly, it'll still be forced to return NULL for amounts
greater than SIZE_MAX.

Where does the standard say that "the largest single dynamic object
shall be SIZE_MAX or less bytes"? Chapter and verse, please. The
standard says that size_t is the type of the result of the sizeof
operator, but you can't apply sizeof to a dynamically allocatd array
object.

A declared object can't be bigger than SiZE_MAX byges, because if it
were it would break the sizeof operator. An object allocated by
malloc() or realloc() can't be bigger than SIZE_MAX bytes, because the
size parameter to both functions is of type size_t. But I assert that
calloc() *can* theoretically create an object bigger than SIZE_MAX
bytes, if the mathematical product of its two arguments exceeds
SIZE_MAX. (I'd be surprised if any actual implementation did this.)
Can you demonstrate how such an implementation would violate the
standard? (I'd actually be glad if you could demonstrate this; it
would make things simpler and more consistent.)

 

[Keith Thompson]

Joe Wright escreveu: [...]

The Standard library does not wipe our noses for us. Upthread it is
noted that 65521 * 65552 is 65296 in 32 bits. The 'normal' product
of 4295032952 requires 33 bits. If we give calloc(65521, 65552) and
get a pointer to 65296 bytes while expecting a pointer to 4295032952
bytes, then we are wrong. The allocator did just what it should.

Here, I think you mistakenly considering the use of malloc which takes
only a 'size_t size' argument.

calloc takes _two_ each one of size_t type. So getting a pointer to an
allocated chunk of memory to 65296 which is far lees than 4295032952
is plainly wrong.

The issue that certain combinations of arguments lead to non available
memory is akin the EDOM errors with certain mathematical functions.

calloc implementations that not return a NULL should be considered
faulty and repaired asap.

Not quite. A calloc implementation that returns a non-NULL pointer
without allocating the requested space is faulty. A calloc
implementation that actually allocates the requested space (more than
SIZE_MAX bytes) and returns a pointer to it would be unusual but, as
far as I can tell, conforming.

 

[Cesar Rabak]

Keith Thompson escreveu:

Not quite. A calloc implementation that returns a non-NULL pointer
without allocating the requested space is faulty. A calloc
implementation that actually allocates the requested space (more than
SIZE_MAX bytes) and returns a pointer to it would be unusual but, as
far as I can tell, conforming.

I concur that in theory the wording of the Standard allows this
interpretation, OTOH, can you devise a scheme where a conforming
implementation could have this behaviour?

It is possible to have a conforming implementation returning some kind
of 'super size_t' pointer (that could address in principle SIZE_MAX²)?

 

[Joe Wright]

No, if calloc(65521, 65552) allocates only 65296 and returns a
non-null pointer, then the implementation of calloc() is broken.

The standard says:

The calloc function allocates space for an array of nmemb objects,
each of whose size is size. The space is initialized to all bits
zero.

It *doesn't* say that calloc() multiplies nmemb by size, yielding a
size_t result, to determine how many bytes to allocate; it simply says
that it allocates space for an array of objects with the specified
length and element size. If it can't do that, by whatever means,, it
needs to return a null pointer.

(In most, perhaps all, implementations, calloc() isn't going to be
able to allocate more than SIZE_MAX bytes.)

malloc() is different. If I call malloc(65521 * 65552) on a system
with a 32-bit size_t (and no padding bits), malloc() must attempt to
allocate 65296 bytes. That's because *I* explicitly did the
multiplication, and 65296 is the correct result of that
multiplication; malloc() can't know that I intended 4295032952.

While I agree that it ought to be as you describe, it often isn't. My
own DJGPP implementation defines..

void *
calloc(size_t size, size_t nelem)
{
void *rv = malloc(size*nelem);
if (rv)
memset(rv, 0, size*nelem);
return rv;
}

Does any C implementation that you know of get it right?