free() Causing Indeterminately-Valued Pointer

[Shao Miller]

The "Re: How to solve problem about "segmentation fault; core dump" in
GCC?" thread had a few posters post about the value of a pointer being
indeterminate[1] after 'free'. While there's clearly a reference for
that in a C Standard, isn't it a bit impossible?

One could copy a pointer as an array of 'char', compress it, encrypt it,
OR it with something the implementation cannot possibly prove in
advance will be all zero bits, but is all zero bits nonetheless, send it
to a friend, then 'free' the pointed-to-object, receive the package back
from the friend, then decrypt the pointer, decompress it, then copy it
back, or to another such pointer?

Does it seem reasonable that 'free' should have this kind of influence?

[1] "C99" C Standard draft with filename 'n1256.pdf': 6.2.4p2

 

[Jens Thoms Toerring]

The "Re: How to solve problem about "segmentation fault; core dump" in
GCC?" thread had a few posters post about the value of a pointer being
indeterminate[1] after 'free'. While there's clearly a reference for
that in a C Standard, isn't it a bit impossible?

One could copy a pointer as an array of 'char', compress it, encrypt it,
OR it with something the implementation cannot possibly prove in
advance will be all zero bits, but is all zero bits nonetheless, send it
to a friend, then 'free' the pointed-to-object, receive the package back
from the friend, then decrypt the pointer, decompress it, then copy it
back, or to another such pointer?

i don't understand what you mean by "copy a pointer as an array of
'char'", sorry. But I don't think that's the issue here - the
address stored in a pointer after that pointer has been used as
an argument to free() is a "hot potato". According to the stan-
dard you may not even "look" at that value anymore (e.g. print
it out, compare it to another pointer etc.). It was also quite
surprising to me when I first heard about that (since free()
doesn't can change anything about the value it receives - the
value of the pointer, of course, remains unchanged) and had a
rather long session reading several parts of the standard until
I had convinced myself that it's actually the true. But by the
mere act of free()-ing that pointer the system is allowed to treat
that value as what seems to be called a "trap representation", i.e.
a value that whenever used allows it to do all kinds of unpleasant
things, not in any way specified by the C standard. E.g. a system
with dedicated address registers would be allowed to check each
value stored in such a register to check it for validity (i.e. is
it an address "belonging" to the process that is using it) and do
whatever it wants in case it's not valid. I haven't experienced a
system yet that would do something like that but then I haven't
had the pleasure of surviving programming a DS9K;-)

Regards, Jens

 

[Gene]

The "Re: How to solve problem about "segmentation fault; core dump" in
GCC?" thread had a few posters post about the value of a pointer being
indeterminate[1] after 'free'.  While there's clearly a reference for
that in a C Standard, isn't it a bit impossible?

One could copy a pointer as an array of 'char', compress it, encrypt it,
  OR it with something the implementation cannot possibly prove in
advance will be all zero bits, but is all zero bits nonetheless, send it
to a friend, then 'free' the pointed-to-object, receive the package back
from the friend, then decrypt the pointer, decompress it, then copy it
back, or to another such pointer?

Does it seem reasonable that 'free' should have this kind of influence?

[1] "C99" C Standard draft with filename 'n1256.pdf': 6.2.4p2

You need to read the Standard. Indeterminate is defined as
"unspecified or a trap representation." A trap representation is a
bit pattern stored in a location where an object could potentially be
stored, but that isn't a member of that object's type.

So all the Standard is saying is that a pointer referencing freed
memory is no longer a valid pointer.

 

[Keith Thompson]

The "Re: How to solve problem about "segmentation fault; core dump" in
GCC?" thread had a few posters post about the value of a pointer being
indeterminate[1] after 'free'. While there's clearly a reference for
that in a C Standard, isn't it a bit impossible?

One could copy a pointer as an array of 'char', compress it, encrypt it,
OR it with something the implementation cannot possibly prove in
advance will be all zero bits, but is all zero bits nonetheless, send it
to a friend, then 'free' the pointed-to-object, receive the package back
from the friend, then decrypt the pointer, decompress it, then copy it
back, or to another such pointer?

Does it seem reasonable that 'free' should have this kind of influence?

[1] "C99" C Standard draft with filename 'n1256.pdf': 6.2.4p2

I would argue that free(ptr) cannot change the representation stored in
ptr, since it's passed by value, as all function arguments are. (It's
been argued that it can, but let's leave that aside for now.)

Some pointer representations may be trap representations; others
correspond to valid pointer values (that includes the null pointer).

Though free(ptr) cannot change the representation stored in ptr, it
*can* alter the subset of pointer representations that are trap
representations, in such a way that the same representation that was a
valid value before free(ptr) is a trap representation after free(ptr).

Consider, for example, a system that checks the validity of any value
loaded into an address register, and traps (crashes the program)
if it's not valid. On such a system, this:

if (ptr == NULL) ...;
free(ptr);
if (ptr == NULL) ...;

could cause such a trap on the second comparison but not on the first.

Of course "trap representation" doesn't mean that accessing it must
cause a trap; the behavior is undefined, which can include behaving
just as you'd expect. If an implementation says that free(ptr)
causes ptr to contain a trap representation, that merely means that
the implemntation chooses not to document the behavior of accessing it.

Another interesting case:

ptr1 = malloc(42);
assert(ptr1 != NULL);
// ptr1 now contains a valid pointer value
free(ptr1);
// ptr1 now contains a trap representation
ptr2 = malloc(42);
// malloc re-uses the same chunk of memory; now ptr1,
// by coincidence, contains a valid pointer value again.

 

[Marcin Grzegorczyk]

The "Re: How to solve problem about "segmentation fault; core dump" in
GCC?" thread had a few posters post about the value of a pointer being
indeterminate[1] after 'free'. While there's clearly a reference for
that in a C Standard, isn't it a bit impossible?

Definitely possible.

For example, on a segmented architecture, part of the pointer could be a
segment selector; and if a segment is deallocated and removed from the
segment descriptor table, any attempt to load its (now invalid) selector
into a segment register could cause an exception. (This is nothing
exotic -- that's exactly how the x86's protected mode works, although
most 32-bit systems arrange things such that user code does not need to
fiddle with segment selectors, so pointers do not include them.)

 

[Seebs]

Does it seem reasonable that 'free' should have this kind of influence?

Yes.

[1] "C99" C Standard draft with filename 'n1256.pdf': 6.2.4p2

So all the Standard is saying is that a pointer referencing freed
memory is no longer a valid pointer.

For a real-world case where this matters:

* There exist machines on which merely storing an unmapped address
in an address register is *in and of itself* enough to trigger a
hardware trap, *EVEN IF YOU DO NOT ATTEMPT TO USE THE ADDRESS*.
* There exist many C libraries where at least some malloc operations
are satisfied by requesting a new mapping of a block of address
space.
* On such implementations, freeing such a block could be implemented
by unmapping it.

So you could end up with a case where simply loading a pointer to
freed memory into an address register, without any kind of use of it
or operation on it, could result in a crash.

Since this does not affect any code which makes sense, there's no reason
to prohibit the implementation from doing what is natural and efficient.

-s

 

[Keith Thompson]

The "Re: How to solve problem about "segmentation fault; core dump" in
GCC?" thread had a few posters post about the value of a pointer being
indeterminate[1] after 'free'. While there's clearly a reference for
that in a C Standard, isn't it a bit impossible?

There are only four kinds of object pointer values:
1 the address of an object
2 one past the address of an object
3 NULL
4 indeterminate

Note that categories 1 and 2 are not disjoint. The address of one
object can be one past the address of a different object. If the two
objects are adjacent elements of an array, this must be true; in other
cases, it may or may not be true. The standard's description
of pointer equality explicitly acknowledges this (C99 6.5.9p6).

 

[Shao Miller]

The "Re: How to solve problem about "segmentation fault; core dump" in
GCC?" thread had a few posters post about the value of a pointer being
indeterminate[1] after 'free'. While there's clearly a reference for
that in a C Standard, isn't it a bit impossible?

One could copy a pointer as an array of 'char', compress it, encrypt it,
OR it with something the implementation cannot possibly prove in
advance will be all zero bits, but is all zero bits nonetheless, send it
to a friend, then 'free' the pointed-to-object, receive the package back
from the friend, then decrypt the pointer, decompress it, then copy it
back, or to another such pointer?

Does it seem reasonable that 'free' should have this kind of influence?

[1] "C99" C Standard draft with filename 'n1256.pdf': 6.2.4p2

Thanks, Jens, Gene, Keith, Marcin, Richard, pete, and Seebs.

Some of you have offered that the very act of loading a pointer value
into an address register could be the cause of a fault. Under which
circumstances might we expect an implementation to perform this
operation? I ask because:

In order to copy a pointer value in the first place, we need to treat it
as a 'char[]' (or such). That means addressable storage (right?). I
might expect that in order for an implementation to be conforming, the
programmer is allowed to copy pointers, like any other object, as
'char[]' (or such), if they so desire. Otherwise, copying 'struct's
with pointer members could be a bit of a problem, couldn't it?

So how about indirection ("dereferencing")? That seems like a spot
where a pointer value might be loaded into an address register. Or
pointer arithmetic, perhaps?

Do we have any guarantees that comparing (via '==') two pointer members
of two 'struct's will not load such address registers for, let's say, a
comparison? Or might the comparison do that?

So does a C Standard allow copying a pointer as 'char[]' or not? If so,
is it only the act of evaluating (as a pointer) that can lead to
non-standard behaviour? What is the trigger, exactly?

If we have five pointers pointing to the same object and use one of them
with 'free', could we potentially experience non-standard behaviour
because the other pointers might be address registers and suddenly
contain a trap representation? That is, must we definitely set four of
them to 'NULL' before calling 'free', even if we would never use the
others in an evaluation, but perhaps only for re-assignment? In
multiple threads with this situation, do we need to halt everything and
set these four to 'NULL'?

Just trying to think through the consequences, here. Thanks a lot.

 

[Shao Miller]

In the allocator that I wrote and use, there is an entrypoint
called queryspace. It checks whether an address is valid. If it
is, it returns the usable space in the block. If not, it returns
0. It at least lets you check, but it is pretty much a pious
gesture.

Interesting. So you pass a pointer or do you pass an integer or
something opqaue, which 'queryspace' knows how to interpret and return a
pointer for? If you'd rather not share, I accept and respect that, too.

 

[Keith Thompson]

Some of you have offered that the very act of loading a pointer value
into an address register could be the cause of a fault. Under which
circumstances might we expect an implementation to perform this
operation? I ask because:

In order to copy a pointer value in the first place, we need to treat it
as a 'char[]' (or such). That means addressable storage (right?). I
might expect that in order for an implementation to be conforming, the
programmer is allowed to copy pointers, like any other object, as
'char[]' (or such), if they so desire. Otherwise, copying 'struct's
with pointer members could be a bit of a problem, couldn't it?

A C program can treat any object as an array of char; that doesn't imply
that the code generated by a C compiler must do so. For example, the
code to access a pointer object might very well use a single instruction
to load it into a register. (Whether it's an address register or a
general-purpose register, or indeed whether there are any registers at
all, is outside the scope of the standard.)

Certainly implementations *can* deal with pointers in a way that never
cause a fault, but they're not required to.

So how about indirection ("dereferencing")? That seems like a spot
where a pointer value might be loaded into an address register. Or
pointer arithmetic, perhaps?

Anything. If a pointer object contains a trap representation, the
standard does not define the behavior of any code that accesses the
value of that object in any way.

ptr = malloc(1);
free(ptr);
ptr; /* undefined behavior */

Certainly that includes indirection and pointer arithmetic.

N1256 6.2.6.1p6:

The value of a structure or union object is never a trap
representation, even though the value of a member of the structure
or union object may be a trap representation.

Do we have any guarantees that comparing (via '==') two pointer members
of two 'struct's will not load such address registers for, let's say, a
comparison? Or might the comparison do that?

If you access the pointer members as pointers, there is of course
no such guarantee. If you access a struct as a whole (say, via an
assignment), the behavior is well defined. The implementation must
do whatever it needs to do to implement that behavior; if it has to
worry about CPU registers to do that, then that's what it has to do.

So does a C Standard allow copying a pointer as 'char[]' or not? If so,
is it only the act of evaluating (as a pointer) that can lead to
non-standard behaviour? What is the trigger, exactly?

Evaluating the value of a pointer object invokes UB if the pointer's
value is indeterminate. Accessing the representation of that object
as a character array does not.

If we have five pointers pointing to the same object and use one of them
with 'free', could we potentially experience non-standard behaviour
because the other pointers might be address registers and suddenly
contain a trap representation? That is, must we definitely set four of
them to 'NULL' before calling 'free', even if we would never use the
others in an evaluation, but perhaps only for re-assignment? In
multiple threads with this situation, do we need to halt everything and
set these four to 'NULL'?

If five pointers point to the same object, the value of each of the
pointers becomes indeterminate. Accessing that value in any way invokes
undefined behavior.

 

[Shao Miller]

Some of you have offered that the very act of loading a pointer value
into an address register could be the cause of a fault. Under which
circumstances might we expect an implementation to perform this
operation? I ask because:

In order to copy a pointer value in the first place, we need to treat it
as a 'char[]' (or such). That means addressable storage (right?). I
might expect that in order for an implementation to be conforming, the
programmer is allowed to copy pointers, like any other object, as
'char[]' (or such), if they so desire. Otherwise, copying 'struct's
with pointer members could be a bit of a problem, couldn't it?

A C program can treat any object as an array of char;

The object representation, right.

that doesn't imply
that the code generated by a C compiler must do so. For example, the
code to access a pointer object might very well use a single instruction
to load it into a register.

That would meet my expectation for efficiency.

(Whether it's an address register or a
general-purpose register, or indeed whether there are any registers at
all, is outside the scope of the standard.)

Agreed. They are also useful discussion points, as they are often used
to demonstrate why the Standard might not standardize certain details.

Certainly implementations *can* deal with pointers in a way that never
cause a fault, but they're not required to.


Anything. If a pointer object contains a trap representation, the
standard does not define the behavior of any code that accesses the
value of that object in any way.

Where we distinguish "value" from "object representation" by pairing
value with type, agreed.

ptr = malloc(1);
free(ptr);
ptr; /* undefined behavior */

Certainly that includes indirection and pointer arithmetic.

N1256 6.2.6.1p6:

The value of a structure or union object is never a trap
representation, even though the value of a member of the structure
or union object may be a trap representation.

Do we have any guarantees that comparing (via '==') two pointer members
of two 'struct's will not load such address registers for, let's say, a
comparison? Or might the comparison do that?

If you access the pointer members as pointers, there is of course
no such guarantee. If you access a struct as a whole (say, via an
assignment), the behavior is well defined. The implementation must
do whatever it needs to do to implement that behavior; if it has to
worry about CPU registers to do that, then that's what it has to do. Agreed.

So does a C Standard allow copying a pointer as 'char[]' or not? If so,
is it only the act of evaluating (as a pointer) that can lead to
non-standard behaviour? What is the trigger, exactly?

Evaluating the value of a pointer object invokes UB if the pointer's
value is indeterminate. Accessing the representation of that object
as a character array does not. Agreed.

If we have five pointers pointing to the same object and use one of them
with 'free', could we potentially experience non-standard behaviour
because the other pointers might be address registers and suddenly
contain a trap representation? That is, must we definitely set four of
them to 'NULL' before calling 'free', even if we would never use the
others in an evaluation, but perhaps only for re-assignment? In
multiple threads with this situation, do we need to halt everything and
set these four to 'NULL'?

If five pointers point to the same object, the value of each of the
pointers becomes indeterminate. Accessing that value in any way invokes
undefined behavior.

What I take as your suggestion is that a _read_ of the value (requiring
a pointer type) is where we obtain non-standard behaviour. I would
include even operations which produce a new value derived from the old
to imply a read, too. Is that right by you?

What I'm gathering is that if one _needs_ to work with a pointer after
the pointed-to object expires, one might be best off with a:

union expiring_ptr {
/* Don't read if expired! */
type *ptr;
/* The bytes are still here, if you like. */
char obj_rep[sizeof (type *)];
};

Thanks, Keith.

 

[Keith Thompson]

What I'm gathering is that if one _needs_ to work with a pointer after
the pointed-to object expires, one might be best off with a:

union expiring_ptr {
/* Don't read if expired! */
type *ptr;
/* The bytes are still here, if you like. */
char obj_rep[sizeof (type *)];
};

If one thinks one needs to work with a pointer after the pointed-to
object expires, one is very probably wrong about what one needs
to do.

But yes, if you really do need to (or just want to) examine a pointer
object after its value has become indeterminate, treating it as an
array of char (or, better, unsigned char) is the way to do it.

 

[Jens Thoms Toerring]

If we have five pointers pointing to the same object and use one of them
with 'free', could we potentially experience non-standard behaviour
because the other pointers might be address registers and suddenly
contain a trap representation?

I don't think that's possible - you as the C programmer have no
influence if and when the pointers are stored in address registers.
So it would be impossible to make sure that this doesn't happen.
If such a machine should exist I think the compiler would have to
make sure it produces code that insures that this situation never
is encountered.

That is, must we definitely set four of
them to 'NULL' before calling 'free',

Even that would not make 100% sure such a pointers value is never
in an address register - it might still linger in a register that's
currently not used. Only some extra code generated by the compiler
could "clean out" such registers, but the compiler would have to
know by itself that this is needed as it's the instance that put
the addresses there, outside of the control of the C programmer.

Regards, Jens

 

[Keith Thompson]

I don't think that's possible - you as the C programmer have no
influence if and when the pointers are stored in address registers.
So it would be impossible to make sure that this doesn't happen.
If such a machine should exist I think the compiler would have to
make sure it produces code that insures that this situation never
is encountered.

Ah, I think I misunderstood the question. Existing pointer objects can
take on indeterminate values when (a copy of) that value is passed to
free() -- but that can't result in undefined behavior unless one of
those objects is accessed. If the value is just sitting somewhere, it
doesn't cause any problems. As you say, it's up to the implementation
to ensure this works properly.

You don't need multiple objects to demonstrate this. A call to
free(ptr) leaves an indeterminate value in ptr.

In practice, I doubt that any real-world systems misbehave when a stored
pointer value becomes indeterminate, so real-world implementations don't
have to do anything special to deal with this.

[...]