[union] Pointers to inherited structs are valid ?

[Ben Bacarisse]

No, it wouldn't. The C semantics do not include undefined behaviour.

This is such an odd statement that a recap is probably needed. What do
you mean by "the C semantics"? And are we still talking about a program
that accesses a pointer whose bit-pattern looks like 0xC?

I use the phrase to mean what the C standard say about the meaning of a
C program. The standard says that such a pointer representation *might*
be a trap representation. No one is saying that *is* one (well, I'm
not) but I have disagreed with the arguments you've put forward for
asserting that it is *not* one.

"[...what a constraint violation is...] Undefined behavior is
otherwise indicated in this International Standard by the words
‘‘undefined behavior’’ or by the omission of any explicit definition
of behavior. There is no difference in emphasis among these three;
they all describe ‘‘behavior that is undefined’’."

I used the word "subtle." There is _no_chance_ that _anything_other_
than what _is_ described by the C Standard will happen. No undefined
behaviour. No trap representation.

Too subtle for me. What, then, does the C standard say will happen when
the representation in question is used as a pointer?

You appear to be agreeing with me. Did you think I meant something
else?

Yes, of course I did.

The difference is that the behaviour is _defined_, instead of
_undefined_. Yes, the behaviour for both can match. No, the
expectation is different between them; you don't know what to expect
from undefined behaviour.

I can't unpick this. If it's central to your point, then I don't
understand what your point is. If it isn't, maybe we can just put it to
one side.

I say why right below. I didn't realize that you couldn't easily
accept this and that you required evidence.

I don't object to the conclusion, I object to the argument. If, below,
you say why 0xC can't be a trap representation in this particular
implementation, then I'll happily accept it. What I did not accept is
what looked to me like your spurious arguments in support of this
claim.

I'm not suggesting that at all. This is a case of Standard behaviour
plus implementation-defined behaviour. Since Keith asked for it, I
dug it up from Microsoft:

"an integral type can be converted to a pointer type according to
the following rules:

- If the integral type is the same size as the pointer type, the
conversion simply causes the integral value to be treated as a pointer
(an unsigned integer)."

Does that help in any way? Can the subject of the parentheses have a
trap representation if all 32 bits are value bits?

This does not say that a pointer object can not hold a trap
representation. The phrase "treated as a pointer" is just too vague to
be sure. Let's say that some bit pattern like 0xC is indeed a void *
trap representation. The conversion (void *)0xC "treats it as a
pointer" but does not involve a representation, so we can't say from
that phrase alone whether that bit-pattern would be a trap
representation if it were in a pointer object.

It seems likely that this text intends to say that all bit patterns
represent valid or invalid (i.e. non-trap) representations for all
pointer types, but it does not get round to actually saying it. And
since the behaviour of a trap representation is a free-for-all, there
is very little way to tell, other than an actual statement of that
fact.

 

[Shao Miller]

Yes, you did quite say that. You said the pointer type "has no trap
representations".

I did _not_ say "that 32-bit pointers on MS Windows have no trap
representations". I said that a "VA32", an "unsafe pointer" and a "lull
pointer" do not have trap representations. Yes, I meant to imply that
there was some relevance to Windows. The relationship I would suggest
between a VA32 and Windows is described immediately below.

That looks a lot like the C Standard's requirements for pointer
conversions, with some extra information about how Microsoft's compiler
performs such conversions. Unlike the standard, it doesn't mention trap
representations.

There is a good reason for this: It conforms to C90. There were no trap
representations in C90. Fortunately, developing for Windows doesn't
require any. What we do have are what's been available since the
dinosaurs roamed the data-centre: Bit patterns that carry useful
debugging information.

Wait, are you saying that any pointer that does not match one of {null
pointer, points to an object} must necessarily be a trap representation?

I believe so, yes.

Given a pointer value that is neither a null pointer nor a pointer to an
object [or function], what criteria would cause you to claim that it's *not* a trap
representation?

I think it has to work the other way around. I think the implementation
has to define it as a trap representation. I don't think we can infer
this property from anything that any Standard says. This could be where
our agreement diverges.

/* Let struct foo have a member m at offset 12 (0xc) */
struct foo *ptr = NULL;
do_something_with(foo->m);

The evaluation of `foo->m` has undefined behavior because the value of
`foo` is a null pointer.
Agreed.

It's likely that the generated code will take
the value stored in `foo` (0x00000000), add the offset 12 to it
(yielding 0x0000000c), and then attempt to dereference the resulting
pointer value. If the program is being executed under the debugger,
this is likely to cause a trap. The debugger sees an attempt to
dereference address 0x0000000c and, quite reasonably, infers that it was
probably the result of accessing a member of a structure or class, at an
offset of 12 bytes, via a null pointer. The debugger may well have
other information available that makes that inference stronger.

(Conceivably a compiler could generate code to test the value of ptr
against 0x00000000 before attempting to add the offset to it; that could
catch the error slightly sooner and more directly, but at a considerable
performance cost.)

I believe you've been implying that this means that 0x0000000c is
a null pointer. It isn't. Seeing Philip's remark, quoted above,
it's a little clearer why you might have thought so.

I did not mean to imply that. Glen Herrmannsfeldt suggested a scenario
where that could be true, but I responded by saying that his case was
"along the lines that might be" the case I was talking about, which is
exactly true: The Microsoft debugger distinguishes between an invalid
pointer and an invalid pointer in the "null class". This implies that
the bits are significant and Glen's masking comes into play. It does
_not_ imply that that I'm saying 0x0000000C is a null pointer. If it
was confusing before, it should have been clear once I answered you
about it comparing as unequal with a null pointer. Sorry about the
confusion.

[snip]

 

[Shao Miller]

This is such an odd statement that a recap is probably needed. What do
you mean by "the C semantics"? And are we still talking about a program
that accesses a pointer whose bit-pattern looks like 0xC?

The C semantics are defined by the Standard. A strictly conforming
program's behaviour is predictable, given these.

Where we see a mention of implementation-defined subject matter, a C
program's behaviour is defined by both the C semantics as well as
implementation-specific definitions. The program's behaviour is
predictable, because the implementation documents their definitions.

Where we see a mention of undefined behaviour, a C program's behaviour
is not guaranteed to be defined by any known means. The program's
behaviour is unpredictable, _unless_ we happen to know it by means which
aren't referred to by the Standard. An implementation or other Standard
(such as POSIX) can certainly define the behaviour.

I'm trying to argue that case for undefined behaviour does not apply.

I use the phrase to mean what the C standard say about the meaning of a
C program. The standard says that such a pointer representation *might*
be a trap representation. No one is saying that *is* one (well, I'm
not) but I have disagreed with the arguments you've put forward for
asserting that it is *not* one.

Ok. I understand that you didn't like my argument. My recent argument
about IRPs used the English "extremely valuable." This is supposed to
suggest that the English "value" might have some relevance to C's "valid
value": The implementation wishes to allow for something that is
valuable to be represented and worked-with in a well-defined manner. It
is roughly a thought experiment. Maybe that's not the argument that you
didn't enjoy.

"[...what a constraint violation is...] Undefined behavior is
otherwise indicated in this International Standard by the words
‘‘undefined behavior’’ or by the omission of any explicit definition
of behavior. There is no difference in emphasis among these three;
they all describe ‘‘behavior that is undefined’’."

I used the word "subtle." There is _no_chance_ that _anything_other_
than what _is_ described by the C Standard will happen. No undefined
behaviour. No trap representation.

[...] What, then, does the C standard say will happen when
the representation in question is used as a pointer?

The C Standard _plus_ the implementation say: It can be stored, read,
passed, discarded, converted, compared, its size determined, etc.
Pretty well anything that doesn't involve using the pointer for indirect
access.

I can't unpick this. If it's central to your point, then I don't
understand what your point is. If it isn't, maybe we can just put it to
one side.

It is central. Please see my notes about predictability, above.

I don't object to the conclusion, I object to the argument. If, below,
you say why 0xC can't be a trap representation in this particular
implementation, then I'll happily accept it. What I did not accept is
what looked to me like your spurious arguments in support of this
claim.

Ok. I understand your objection. I didn't think it'd be objectionable.

This does not say that a pointer object can not hold a trap
representation. The phrase "treated as a pointer" is just too vague to
be sure. Let's say that some bit pattern like 0xC is indeed a void *
trap representation. The conversion (void *)0xC "treats it as a
pointer" but does not involve a representation, so we can't say from
that phrase alone whether that bit-pattern would be a trap
representation if it were in a pointer object.

There was no claim that this Microsoft quote states that a pointer
cannot hold a trap representation. Such a claim would be impossible,
since there are no trap representations in C90 (which Microsoft conforms
to). The quote is certainly a hint that Microsoft represents pointers
the same way as unsigned integers with the same size. Since we happen
to know (I hope "we" applies) that all bits are value bits for the
integer representation, it suggests to me that the number of pointer
values == the number of the corresponding unsigned integer type's values.

However, although C90 has no trap representations, I would still say
that a pointer-to-object whose representation addresses a misaligned
object is a trap representation. I couldn't prove such a statement, but
it makes sense, to me.

It seems likely that this text intends to say that all bit patterns
represent valid or invalid (i.e. non-trap) representations for all
pointer types, but it does not get round to actually saying it. And
since the behaviour of a trap representation is a free-for-all, there
is very little way to tell, other than an actual statement of that
fact.

The reason why it doesn't get around to saying it is discussed, above.
Else-thread:

Fortunately, developing for Windows doesn't require any. What we do
have are what's been available since the dinosaurs roamed the
data-centre: Bit patterns that carry useful debugging information.

(Which is what Geoff was talking about, I surely hope.)

 

[Philip Lantz]

Actually there are four kinds of pointer values (with the
understanding that "value" here includes some that cannot
be used definedly):

1. unusable (any use is undefined behavior)
2. null pointers (can be compared for equality/inequality)
3. equality, pointer arithmetic, relational (eg <) comparison
4. like 3 but also can be dereferenced

Type 3 values are, eg, pointers one past the end of an array, or
non-null values returned from doing a malloc(0). Type 4 values
are regular pointers to objects.

The footnote makes it clear that the phrase 'invalid value' used
in 6.5.3.2 p4 means categories 1-3 above. However this meaning
of the phrase is meant to apply only to this section.

If a pointer value is stored, the resulting object representation
can be a trap representation only for values of type 1. (Of
course any attempt to store a value of type 1 could store anything
at all, including some representation of any of the above
categories.)

If an object is read as a pointer type, and the resulting value
is of type 1 (and assuming there wasn't anything else causing
undefined behavior), then the object representation of that
object must be (or have been) a trap representation (when
considered as the type used to do the access). This follows from
the definition of trap representation.

I think there may be one more kind of pointer value, which is the kind
that caused this thread. It is a value that may appear in a pointer
object as a result of some prior undefined behavior. This is of course
outside the scope of the C standard--which is why it correctly doesn't
appear in your list--but it does occur in actual implementations.

It behaves much like your type 1, but I'm not sure it's identical; in
particular, its behavior can't be described by referring to the
standard. Before reading this thread, I would not have ever thought to
call it a trap representation, but it is pretty similar.

 

[Ben Bacarisse]

<all snipped>

I don't have anything more to add to this exchange. Commenting on your
comments will just add to the noise, I fear, without adding anything
new.

 

[Keith Thompson]

I don't understand this as an answer for "it is imprecise to say that
such a value has a trap representation, because the behaviour is still
well-defined." I don't understand it as an answer for "Otherwise, the
last sentence of N1570's 6.5.3.2p4 is redundant." Did you think that I
didn't think that accessing a trap representation was undefined behaviour?

I *think* the phrase "such a value" refers to a pointer value such as
(void*)0x0000000C, in a Microsoft Windows 32-bit environment. If you're
talking about something else, the following probably won't make much
sense.

Let's try a concrete example. Given this code snippet:

int *ptr;
uintptr_t x = 0x0000000C;
memcpy(&ptr, &x, sizeof ptr);
int deref = *ptr;

I assert that evaluating `*ptr` has undefined behavior. I also assert,
though with slightly less confidence, that after the memcpy() call ptr
contains a trap representation.

Do you disagree? If so, can you cite where the C standard defines the
behavior of the dereference?

[snip]

"I see a Microsoft debugger catch these things and call them [non-C]
null pointers." -- I don't believe you have seen that.

"These things" == the subject that Geoff had most recently discussed:
"[non-C] trap values".

What you've seen, as I recall, is the Microsoft debugger, on an
empt to dereference (on the machine code level) a pointer with
the representation 0x0000000C, printing an error message that
includes the identifier "NULL_CLASS_PTR_DEREFERENCE". That is not
the debugger calling 0x0000000C a null pointer. It's the debugger
inferring, from the attempt to dereference 0x0000000C, that there
was an attempt to dereference a C null pointer (0x00000000). Or am
I missing something?

 

[Keith Thompson]

On 1/21/2013 18:50, Keith Thompson wrote: [...]

I believe you've been implying that this means that 0x0000000c is
a null pointer. It isn't. Seeing Philip's remark, quoted above,
it's a little clearer why you might have thought so.

I did not mean to imply that. Glen Herrmannsfeldt suggested a scenario
where that could be true, but I responded by saying that his case was
"along the lines that might be" the case I was talking about, which is
exactly true: The Microsoft debugger distinguishes between an invalid
pointer and an invalid pointer in the "null class". This implies that
the bits are significant and Glen's masking comes into play. It does
_not_ imply that that I'm saying 0x0000000C is a null pointer. If it
was confusing before, it should have been clear once I answered you
about it comparing as unequal with a null pointer. Sorry about the
confusion.

You're inferring that the debugger message "NULL_CLASS_PTR_DEREFERENCE"
refers to pointers in the "null class". I don't think that's what it
means.

Microsoft's development environment places a greater emphasis on C++
than on C. An attempted dereference of address 0x0000000C is likely
to result from an attempt to dereference a pointer to a class type
(in C++, structs are classes), and the debugger may have more
information that strengthens this inference.

class foo {
public:
char x[12];
int y;
};

foo* ptr = 0;
int kaboom = ptr->y;

(I just tried running a C++ program with the above code under MS Visual
C++ 2010 Express, and didn't get that message; instead, I got "Unhandled
exception at 0x009013a8 in null_class_ptr.exe: 0xC0000005: Access
violation reading location 0x0000000c.".)

 

[Keith Thompson]

Actually there are four kinds of pointer values (with the
understanding that "value" here includes some that cannot
be used definedly):

1. unusable (any use is undefined behavior)
2. null pointers (can be compared for equality/inequality)
3. equality, pointer arithmetic, relational (eg <) comparison
4. like 3 but also can be dereferenced

Type 3 values are, eg, pointers one past the end of an array, or
non-null values returned from doing a malloc(0). Type 4 values
are regular pointers to objects.

[...]

And I believe that type 1 is exactly the set of trap representations,
but I haven't been able to prove it.

The definition of "trap representation" in 3.19.4 is a bit vague:

trap representation

an object representation that need not represent a value of the
object type

I wonder why it doesn't say "does not" rather than "need not".

 

[Keith Thompson]

The C semantics are defined by the Standard. A strictly conforming
program's behaviour is predictable, given these.

Where we see a mention of implementation-defined subject matter, a C
program's behaviour is defined by both the C semantics as well as
implementation-specific definitions. The program's behaviour is
predictable, because the implementation documents their definitions.

Where we see a mention of undefined behaviour, a C program's behaviour
is not guaranteed to be defined by any known means. The program's
behaviour is unpredictable, _unless_ we happen to know it by means
which aren't referred to by the Standard. An implementation or other
Standard (such as POSIX) can certainly define the behaviour.

I agree with the above.

I'm trying to argue that case for undefined behaviour does not apply.

Does not apply *to what*?

Can you provide a small self-contained program that's relevant to
the point you're making?

If you're saying that there's some construct whose behavior you
say is implementation-defined, and I say is undefined, then please
point out the construct in question in your program.

 

[Keith Thompson]

I think there may be one more kind of pointer value, which is the kind
that caused this thread. It is a value that may appear in a pointer
object as a result of some prior undefined behavior. This is of course
outside the scope of the C standard--which is why it correctly doesn't
appear in your list--but it does occur in actual implementations.

It behaves much like your type 1, but I'm not sure it's identical; in
particular, its behavior can't be described by referring to the
standard. Before reading this thread, I would not have ever thought to
call it a trap representation, but it is pretty similar.

How does "its behavior can't be described by referring to the standard"
differ from "any use is undefined behavior"? (Apart from the quibble
that pointer values don't have behavior; operations on them do.)

 

[Shao Miller]

I *think* the phrase "such a value" refers to a pointer value such as
(void*)0x0000000C, in a Microsoft Windows 32-bit environment. If you're
talking about something else, the following probably won't make much
sense.

Let's try a concrete example. Given this code snippet:

int *ptr;
uintptr_t x = 0x0000000C;
memcpy(&ptr, &x, sizeof ptr);
int deref = *ptr;

I assert that evaluating `*ptr` has undefined behavior.

I'd say that it is implementation-defined behaviour; the implementation
knows

All I've been trying to point out is that, for Windows NT, 'ptr' does
not contain a trap representation, so any use _other_ than indirection
(via '*', '->', or function-call '()', for example) is well-defined.
For indirection, it depends on if 'ptr' points to an 'int' object, as
always.

Any confusion could be a matter of my not picking the right choice of
words at the right times.

I also assert,
though with slightly less confidence, that after the memcpy() call ptr
contains a trap representation.
Do you disagree? If so, can you cite where the C standard defines the
behavior of the dereference?

Yes, I disagree with the second part about 'ptr' containing a trap
representation, for Windows NT.

" 6.2.6.1 General

1 The representations of all types are unspecified except as stated
in this subclause.

2 Except for bit-fields, objects are composed of contiguous
sequences of one or more bytes, the number, order, and encoding of which
are either explicitly specified or implementation-defined."

we don't see 'int *' explicitly specified, thereafter. For 32-bit
Windows NT, it so happens that this type is represented the same way as
an unsigned integer with all value bits. It so happens that 0xC is a
multiple of 'sizeof (int)' and represents a suitable alignment for an 'int'.

[snip]

"I see a Microsoft debugger catch these things and call them [non-C]
null pointers." -- I don't believe you have seen that.

"These things" == the subject that Geoff had most recently discussed:
"[non-C] trap values".

What you've seen, as I recall, is the Microsoft debugger, on an
empt to dereference (on the machine code level) a pointer with
the representation 0x0000000C, printing an error message that
includes the identifier "NULL_CLASS_PTR_DEREFERENCE". That is not
the debugger calling 0x0000000C a null pointer. It's the debugger
inferring, from the attempt to dereference 0x0000000C, that there
was an attempt to dereference a C null pointer (0x00000000). Or am
I missing something?

I don't think there needs to be an inference that were was a dereference
to a null pointer, but other than that, I think you're right. I really
wish I'd said "NULL_CLASS_PTR_DEREFERENCE" in the beginning, so it
wouldn't have been an issue of discussion. The trap values that Geoff
had discussed were not C trap representations, so I was likewise being
loose with "null pointers." I am filled with regret about that. As
we've already discussed, such a pointer value does not compare equal to
a C null pointer, so the debugger could hardly claim that it's a C null
pointer.

In order to understand what I mean by "loose" and why I might've done
so, for what it's worth, there are some people who don't use the C
definition for "null pointer" in non-pedantic discussion:

Scott Noone (well known at OSR Online):
http://social.msdn.microsoft.com/Forums/en-AU/windbg/thread/669a4137-f328-495f-a002-120fb6841542

Chad Bramwell:
http://www.altdevblogaday.com/2011/09/26/how-did-i-crash-in-that-func/

On 1/21/2013 18:50, Keith Thompson wrote: [...]

I believe you've been implying that this means that 0x0000000c is
a null pointer. It isn't. Seeing Philip's remark, quoted above,
it's a little clearer why you might have thought so.

I did not mean to imply that. Glen Herrmannsfeldt suggested a scenario
where that could be true, but I responded by saying that his case was
"along the lines that might be" the case I was talking about, which is
exactly true: The Microsoft debugger distinguishes between an invalid
pointer and an invalid pointer in the "null class". This implies that
the bits are significant and Glen's masking comes into play. It does
_not_ imply that that I'm saying 0x0000000C is a null pointer. If it
was confusing before, it should have been clear once I answered you
about it comparing as unequal with a null pointer. Sorry about the
confusion.

You're inferring that the debugger message "NULL_CLASS_PTR_DEREFERENCE"
refers to pointers in the "null class". I don't think that's what it
means.

Microsoft's development environment places a greater emphasis on C++
than on C. An attempted dereference of address 0x0000000C is likely
to result from an attempt to dereference a pointer to a class type
(in C++, structs are classes), and the debugger may have more
information that strengthens this inference.

class foo {
public:
char x[12];
int y;
};

foo* ptr = 0;
int kaboom = ptr->y;

(I just tried running a C++ program with the above code under MS Visual
C++ 2010 Express, and didn't get that message; instead, I got "Unhandled
exception at 0x009013a8 in null_class_ptr.exe: 0xC0000005: Access
violation reading location 0x0000000c.".)

I get your point. I think another example would be a class method (or
whatever they're called) doing something with 'this->y'.

Off-topically, I'd suggest installing Debugging Tools for Windows for
messing around with Windows debugging; it's quite nifty! After an
error, '!analyze -v'.

I agree with the above.

I'm glad to read that.

Does not apply *to what*?

Considering a Windows pointer with representation 0xC, I meant with
regards to what Ben had asked about and what I had typed later in the
same post: "The C Standard _plus_ the implementation say: It can be
stored, read, passed, discarded, converted, compared, its size
determined, etc. Pretty well anything that doesn't involve using the
pointer for indirect access."

I would've added pointer arithmetic as allowed, but then I'd've had to
add a note about a complete object type. I would've added function
calls as not being allowed, but I hoped that would be understood by
"indirect access". An exhaustive list would be tedious.

Can you provide a small self-contained program that's relevant to
the point you're making?

If you're saying that there's some construct whose behavior you
say is implementation-defined, and I say is undefined, then please
point out the construct in question in your program.

int main(void) {
void * vp1 = (void *) 0xC;
void * vp2 = vp1;
return 0;
}

This is implementation-defined, not undefined. If the implementation
does not define the result of the cast to be a trap representation, or
better yet, defines that pointers are implemented as unsigned integers
with all value bits, then there is no undefined behaviour. This is the
case for Windows NT, as far as I'm aware.

If someone wants to suggest that 'vp1' and 'vp2' have trap
representations, I'd like to know why. If we can agree that their
representations can provide hints to a debugger, then I'll be glad to
read that.

 

[Keith Thompson]

I'd say that it is implementation-defined behaviour; the implementation
knows

I disagree, and I'll elaborate below.

All I've been trying to point out is that, for Windows NT, 'ptr' does
not contain a trap representation, so any use _other_ than indirection
(via '*', '->', or function-call '()', for example) is well-defined.
For indirection, it depends on if 'ptr' points to an 'int' object, as
always.

Assume that there is no int object at address 0x0000000C. Do you
believe that the behavior of an access to ptr is well-defined *by
the C standard*?

A trap representation is a representation such that accessing an object
holding it has undefined behavior. Undefined behavior is behavior that
is not defined by the C standard, regardless of whether the
implementation chooses to define it.

[...]

I don't think there needs to be an inference that were was a dereference
to a null pointer, but other than that, I think you're right. I really
wish I'd said "NULL_CLASS_PTR_DEREFERENCE" in the beginning, so it
wouldn't have been an issue of discussion. The trap values that Geoff
had discussed were not C trap representations, so I was likewise being
loose with "null pointers." I am filled with regret about that. As
we've already discussed, such a pointer value does not compare equal to
a C null pointer, so the debugger could hardly claim that it's a C null
pointer.

And in fact it doesn't claim either that 0x000000C a C null pointer, or
that it's any kind of null pointer.

In order to understand what I mean by "loose" and why I might've done
so, for what it's worth, there are some people who don't use the C
definition for "null pointer" in non-pedantic discussion:

Scott Noone (well known at OSR Online):
http://social.msdn.microsoft.com/Forums/en-AU/windbg/thread/669a4137-f328-495f-a002-120fb6841542

Chad Bramwell:
http://www.altdevblogaday.com/2011/09/26/how-did-i-crash-in-that-func/

I disagree with your interpretation of both linked discussions. In both
cases, there was a null pointer dereference in C (or C++) code, which
resulted in an attempted dereference of a non-null but invalid pointer
(similar to 0x0000000C) in the generated machine code.

Machine code *implements* C semantics; it needn't precisely mirror them.

[...]

Considering a Windows pointer with representation 0xC, I meant with
regards to what Ben had asked about and what I had typed later in the
same post: "The C Standard _plus_ the implementation say: It can be
stored, read, passed, discarded, converted, compared, its size
determined, etc. Pretty well anything that doesn't involve using the
pointer for indirect access."

Sure, but the C standard doesn't permit any of those things. It's well
known that an implementation can define behavior that isn't defined by
the C standard. Such behavior is still "undefined behavior" as defined
by 3.4.3.

[...]

int main(void) {
void * vp1 = (void *) 0xC;
void * vp2 = vp1;
return 0;
}

This is implementation-defined, not undefined. If the implementation
does not define the result of the cast to be a trap representation, or
better yet, defines that pointers are implemented as unsigned integers
with all value bits, then there is no undefined behaviour. This is the
case for Windows NT, as far as I'm aware.

No, it's undefined. It may additionally be defined by the
implementation, but it's not implementation-defined.

The phrase "implementation-defined behavior", as defined by the C
standard, refers *only* to behavior that is explicitly referred to
by the standard as "implementation-defined". It doesn't just mean
"behavior that is defined by the implementation".

In your program, the behavior of the initialization:

void * vp2 = vp1;

is not defined by the C standard. For example, it could cause the
program to crash in a conforming implementation. It is therefore,
by definition, undefined behavior. If an implementation chooses
to define its behavior, that doesn't change any of the above --
and the implementation is not obligated to document its choice.

If someone wants to suggest that 'vp1' and 'vp2' have trap
representations, I'd like to know why. If we can agree that their
representations can provide hints to a debugger, then I'll be glad to
read that.

They are pointer values such that accessing them has undefined
behavior. They are neither null pointers, nor pointers to any
object, nor pointers just past the end of any object. It's not 100%
clear to me, from the standard's definition of "trap representation",
that they *must* be trap representations, but I believe that
they are.

Speculation: Perhaps what bothers you is the idea that "undefined
behavior" and "trap representation" imply "This is evil, don't
touch!!!". They don't. It's perfectly valid for an implementation
to define the behavior of something that has "undefined behavior" in
the standard, or to use a C trap representation for its own purposes.

Certainly their representation can provide hints to a debugger; we've
seen that demonstrated. That doesn't cause them not to be trap
representations.

 

[Keith Thompson]

Sure, but the C standard doesn't permit any of those things. It's well
known that an implementation can define behavior that isn't defined by
the C standard. Such behavior is still "undefined behavior" as defined
by 3.4.3.

[...]

Sorry, that was imprecise. What I should have said is that the C
standard doesn't define the behavior of any of those things.

 

[Geoff]

What you've seen, as I recall, is the Microsoft debugger, on an
empt to dereference (on the machine code level) a pointer with
the representation 0x0000000C, printing an error message that
includes the identifier "NULL_CLASS_PTR_DEREFERENCE". That is not
the debugger calling 0x0000000C a null pointer. It's the debugger
inferring, from the attempt to dereference 0x0000000C, that there
was an attempt to dereference a C null pointer (0x00000000). Or am
I missing something?

You're not missing anything. That is precisely what is happening.
The actual message is something like: "Unhandled exception at 0x01091000 in
trap.exe: 0xC0000005: Access violation writing location 0x0000000c."

Error 0xC0000005 is STATUS_ACCESS_VIOLATION, the memory could not be written or
read. The process attempted to access memory outside its memory pool.

 

[Shao Miller]

I disagree, and I'll elaborate below.


Assume that there is no int object at address 0x0000000C. Do you
believe that the behavior of an access to ptr is well-defined *by
the C standard*?

Assuming you really meant "access to ptr" and not "access to *ptr", my
answer is: No, then no.

The first access is during the 'memcpy'. The implementation defines the
representations of both 'x' and 'ptr'. If they do not have the same
size, there could be an out-of-bounds condition. If they do have the
same size, then this access is fine and we move on.

The second access is during the lvalue conversion that determines the
operand of the unary '*' operator. This second access has
implementation-defined behaviour, since the implementation defines the
mapping from object representations to values. Anything not defined (by
omission) or explicitly defined to be a trap representation would be a
trap representation. In that case, the lvalue conversion would yield
undefined behaviour. Otherwise, the value is a valid value and we get
past that lvalue conversion.

After that point, we can fret about the '*' operator. Whether or not
there's an object at address 0xC is not a consideration until this
point, so my understanding goes.

What I would say is an extremely relevant piece of Standard has
accidentally been snipped:

" 6.2.6.1 General

1 The representations of all types are unspecified except as stated
in this subclause.

2 Except for bit-fields, objects are composed of contiguous
sequences of one or more bytes, the number, order, and encoding of which
are either explicitly specified or implementation-defined."

Until the indirection in the last line, your example above should have
no different expectation for undefined/defined behaviour when compared to:

#include <string.h>
#include <stdio.h>

int main(void) {
float f;
unsigned int x = 0x4048F5C3;
memcpy(&f, &x, sizeof x);
printf("%f\n", f);
return 0;
}

If you'd say that this is undefined behaviour rather than
implementation-defined behaviour, please do explain why.

A trap representation is a representation such that accessing an object
holding it has undefined behavior. Undefined behavior is behavior that
is not defined by the C standard, regardless of whether the
implementation chooses to define it.

Did you just say that implementation-defined behaviour is a subset of
undefined behaviour? I don't think you did, given that you discuss
"implementation-defined," down below.

I think you think that I'm arguing that there is undefined behaviour and
that it so happens to be defined by Microsoft, so I'm claiming it to be
"well-defined". I'm not! I'm saying that there are parts where the
Standard calls something "implementation-defined" instead of
"undefined", and this is one such instance.

I tried to explain this before with the three paragraphs in response to
Ben about "predictability." "Well-defined" to me means that either the
Standard defines it directly, or defines some of it and defines that an
implementation defines the rest of it. Undefined behaviour doesn't
match either of those, even if the implementation provides definitions.
Was that not clear?

[...]

I don't think there needs to be an inference that were was a dereference
to a null pointer, but other than that, I think you're right. I really
wish I'd said "NULL_CLASS_PTR_DEREFERENCE" in the beginning, so it
wouldn't have been an issue of discussion. The trap values that Geoff
had discussed were not C trap representations, so I was likewise being
loose with "null pointers." I am filled with regret about that. As
we've already discussed, such a pointer value does not compare equal to
a C null pointer, so the debugger could hardly claim that it's a C null
pointer.

And in fact it doesn't claim either that 0x000000C a C null pointer, or
that it's any kind of null pointer.

I don't think it's intentional, but I'm going to share a result of this:
I've been contacted by another regular who has the impression that we've
been arguing for a long time about whether or not 0xC is a null pointer.

while (1) {
You("It's not a null pointer.");
Me("Well I didn't mean a C null pointer. Sorry.");
}

What do you mean by "any kind of null pointer"? After this discussion,
I didn't think it was remotely possible to believe that anything other
than the C definition could be spoken of. Heh.

There are at least 4 IDs that WinDbg uses for an invalid memory access:

1. BAD_PTR_DEREFERENCE
2. NULL_CLASS_PTR_DEREFERENCE
3. NULL_DEREFERENCE
4. STRING_DEREFERENCE

An exception can be analyzed (with 'analyze -v') and a "default bucket
ID" will be chosen by WinDbg. I seldom debug programs where it chooses
#1 or #4. I usually see it choose #2 or #3. A garbage pointer usually
just yields a default bucket ID "progtype_FAULT" (such as "DRIVER_FAULT"
or "APPLICATION_FAULT"), with no mention of "NULL" anywhere.

I disagree with your interpretation of both linked discussions. In both
cases, there was a null pointer dereference in C (or C++) code, which
resulted in an attempted dereference of a non-null but invalid pointer
(similar to 0x0000000C) in the generated machine code.

Machine code *implements* C semantics; it needn't precisely mirror them.

I don't think you understood my interpretation.

Scott: "This is a NULL pointer dereference in NTFS."

Chad: "Once again we are dereferencing a NULL pointer and once again our
program is crashing..."

Chad: "...the real problem was that your pointer was NULL?"

Chad: "So there ya go. A whole lot of null dereferences..."

In none of the cases was 'NULL' dereferenced. In none of the cases was
"null pointer" typed. That is, people don't always say precisely what
they mean. I'll try to do better in the future.

[...]

Considering a Windows pointer with representation 0xC, I meant with
regards to what Ben had asked about and what I had typed later in the
same post: "The C Standard _plus_ the implementation say: It can be
stored, read, passed, discarded, converted, compared, its size
determined, etc. Pretty well anything that doesn't involve using the
pointer for indirect access."

Sure, but the C standard doesn't permit any of those things. It's well
known that an implementation can define behavior that isn't defined by
the C standard. Such behavior is still "undefined behavior" as defined
by 3.4.3.

Discussed above. I'm claiming it's defined by the Standard to be
implementation-defined behaviour, not that it's
undefined-behaviour-relative-to-the-Standard and which happens to be
defined elsewhere. I could be wrong, as always, but I've yet to
understand why that might be.

[...]

int main(void) {
void * vp1 = (void *) 0xC;
void * vp2 = vp1;
return 0;
}

This is implementation-defined, not undefined. If the implementation
does not define the result of the cast to be a trap representation, or
better yet, defines that pointers are implemented as unsigned integers
with all value bits, then there is no undefined behaviour. This is the
case for Windows NT, as far as I'm aware.

No, it's undefined. It may additionally be defined by the
implementation, but it's not implementation-defined.

I don't understand this perspective, given 6.3.2.3p5:

"An integer may be converted to any pointer type. Except as
previously specified, the result is implementation-defined, might not be
correctly aligned, might not point to an entity of the referenced type,
and might be a trap representation.67)"

The phrase "implementation-defined behavior", as defined by the C
standard, refers *only* to behavior that is explicitly referred to
by the standard as "implementation-defined". It doesn't just mean
"behavior that is defined by the implementation".

Yes, and that's the stuff I'm talking about. (See above.)

In your program, the behavior of the initialization:

void * vp2 = vp1;

is not defined by the C standard. For example, it could cause the
program to crash in a conforming implementation. It is therefore,
by definition, undefined behavior. If an implementation chooses
to define its behavior, that doesn't change any of the above --
and the implementation is not obligated to document its choice.

You're talking about undefined behaviour. I'm talking about
implementation-defined behaviour. This line cannot stand on its own...

We have to ask, "Was the lvalue conversion of 'vp1' defined?"

The answer is, "Yes, because it has a valid value."

Then we ask, "How do we know that?"

The answer is, "Because it was initialized with the valid value that was
the result of the cast."

Then we ask, "How do we know the result of the cast was a valid value?"

The answer is, "Because the result of the cast conversion is
implementation-defined, and Microsoft's implementation defined it to be
a valid value."

If we'd encountered any lack of definitions along the way, there'd be
undefined behaviour. Or, more naturally, we could work from the
beginning, too.

They are pointer values such that accessing them has undefined
behavior. They are neither null pointers, nor pointers to any
object, nor pointers just past the end of any object. It's not 100%
clear to me, from the standard's definition of "trap representation",
that they *must* be trap representations, but I believe that
they are.

Well I suppose it's hard to prove either way, given that it's C90. I
just figured that it is up to the implementation to define which
representations map to which of {value, non-value}. I'm about C99%
sure, so learning otherwise would be a valuable learning experience.

Speculation: Perhaps what bothers you is the idea that "undefined
behavior" and "trap representation" imply "This is evil, don't
touch!!!". They don't. It's perfectly valid for an implementation
to define the behavior of something that has "undefined behavior" in
the standard, or to use a C trap representation for its own purposes.

Good guess, but nope.

Certainly their representation can provide hints to a debugger; we've
seen that demonstrated. That doesn't cause them not to be trap
representations.

Agreed.

 

[Tim Rentsch]

I think there may be one more kind of pointer value, which is the
kind that caused this thread. It is a value that may appear in a
pointer object as a result of some prior undefined behavior. This
is of course outside the scope of the C standard--which is why it
correctly doesn't appear in your list--but it does occur in actual
implementations.

It behaves much like your type 1, but I'm not sure it's identical;
in particular, its behavior can't be described by referring to the
standard. Before reading this thread, I would not have ever
thought to call it a trap representation, but it is pretty
similar.

Let me see if I can help untangle the description you're giving.

There are two different spaces of interest: values, and object
representations. What is stored in an object is an object
representation, not a value. Sometimes we talk about the "value
stored in an object", but that's just a shorthand for the value
represented by the object representation that the object holds.

Similarly, the result of evaluating an expression is a value, not
an object representation. It might be useful to think of mapping
between the two as a kind of "conversion" -- reading an object
converts an object representation to a value, and storing into an
object converts a value into an object representation.

In value space, the above four classes completely partition the
set of values for pointer types. A kind of value along the lines
you describe -- we might call it "semi-defined" -- would be a
subset of the Type 1 class, not a separate class.

In object representation space, we can consider the set of object
representations whose values will behave exactly in all the ways
the Standard requires (for the value type used to access the
object). These values are what might be called "well behaved",
and actually this set of values is what the Standard normally
means by 'value' (eg, in the phrase 'value of the object type').
For pointer types, these values correspond to types 2-4 above.

Any other object representation is one which when read (ie, by an
lvalue conversion) yields a "value" that might not behave in ways
the Standard requires. Any such object representation meets the
defining condition for trap representations: it need not represent
a (well-behaved) value of the object type. This set of object
representations is exactly the set of trap representations.

Of course, it should go without saying that the above assumes there
has been no previous undefined behavior. Whenever there has been
any previous undefined behavior, any assertion about how subsequent
actions will proceed might not hold up. There isn't any point in
talking about what might be true under such circumstances, because
anything at all _might_ be true, depending on how an implementation
chooses to exercise the unlimited license granted by the Standard
upon encountering undefined behavior.

 

[Tim Rentsch]

Actually there are four kinds of pointer values (with the
understanding that "value" here includes some that cannot
be used definedly):

1. unusable (any use is undefined behavior)
2. null pointers (can be compared for equality/inequality)
3. equality, pointer arithmetic, relational (eg <) comparison
4. like 3 but also can be dereferenced

Type 3 values are, eg, pointers one past the end of an array, or
non-null values returned from doing a malloc(0). Type 4 values
are regular pointers to objects.

[...]

And I believe that type 1 is exactly the set of trap representations,
but I haven't been able to prove it.

My response to Philip Lantz gives a line of reasoning related
to this.

The definition of "trap representation" in 3.19.4 is a bit vague:

trap representation

an object representation that need not represent a value of the
object type

I wonder why it doesn't say "does not" rather than "need not".

Because implementations may define (either explicitly or
implicitly) what happens for such representations so that they
do represent well-behaved values _in some cases_. Saying that
a TR need not represent a value of the object type leaves open
the possibility that under some circumstances it might be
okay. Basically, it lets the implementation eat its cake and
have it too.

 

[Shao Miller]

Scott Noone suggested that I won't find any documentation for
"NULL_CLASS_PTR_DEREFERENCE". D'oh well. :S

 

[Shao Miller]

Scott Noone suggested that I won't find any documentation for
"NULL_CLASS_PTR_DEREFERENCE". D'oh well. :S

Experiment reveals that this ID is yielded from indirection on address 1
all the way through binary 10011111111/decimal 1279/hexadecimal 0x4FF;
binary 10100000000/decimal 1280/hexadecimal 0x500 yields a different ID
(for 32-bit Windows XP). These numbers don't appear to have anything to
do with any C limits that I'm aware of.

 

[Geoff]

Scott Noone suggested that I won't find any documentation for
"NULL_CLASS_PTR_DEREFERENCE". D'oh well. :S

You won't find documentation on it. It's internal to Windbg. The bucketing codes
are used by Windbg to determine the kind of debug report it generates. As far as
I know, Microsoft has never documented these codes.

The 0xC0000005 code, STATUS_ACCESS_VIOLATION, is enough to know you have a bad
pointer somewhere and it's time to start unwinding the stack in the debugger to
see how it happened.

But all this is not C. I think this horse is muerto.