Template technicality - What does the standard say?

[Stephen Horne]

I've been using Visual C++ 2003 for some time, and recently started
working on making my code compile in GCC and MinGW. I hit on lots of
unexpected problems which boil down to the same template issue. A
noddy mixin layer example should illustrate the issue...

class Base
{
protected:
int m_Field;

public:
Base () : m_Field (0) {}
};

template<class T> class Derived : public T
{
public:
void Inc ()
{
T::m_Field++; // GCC needs the "T::" here - VC++ doesn't
}
};

In the above, the idea is to use Derived<Base>, but GCC (3.4.5 and
4.1.2) gives error messages even if you never instantiate the
template.

There are of course examples of similar templates in Stroustrup, but
because they are incomplete examples, I can't find one that shows an
inherited member being referenced from within the template class.

In non-template classes, scope resolution is only needed to access
inherited members if there is an ambiguity. There is no ambiguity in
the above - the only place m_Field can come from is the inherited
class Base via the template parameter T. Following that general
principle, I don't see any reason why scope resolution should be
needed.

What does the standard say about this - should the scope resolution
operator be required? If it is required, what's the rationale? If
there's something in Stroustrup about this, can you point me to the
right section?

 

[Michael DOUBEZ]

Stephen Horne a écrit :

I've been using Visual C++ 2003 for some time, and recently started
working on making my code compile in GCC and MinGW. I hit on lots of
unexpected problems which boil down to the same template issue. A
noddy mixin layer example should illustrate the issue...

class Base
{
protected:
int m_Field;

public:
Base () : m_Field (0) {}
};

template<class T> class Derived : public T
{
public:
void Inc ()
{
T::m_Field++; // GCC needs the "T::" here - VC++ doesn't
}
};

In the above, the idea is to use Derived<Base>, but GCC (3.4.5 and
4.1.2) gives error messages even if you never instantiate the
template. [snip]
What does the standard say about this - should the scope resolution
operator be required? If it is required, what's the rationale? If
there's something in Stroustrup about this, can you point me to the
right section?

Look up template-dependant name:
http://www.parashift.com/c++-faq-lite/templates.html#faq-35.19

 

[Stephen Horne]

Look up template-dependant name:
http://www.parashift.com/c++-faq-lite/templates.html#faq-35.19

Thanks.

Just read 35.18, 35.19 and 35.20. Now I *know* that the lunatics are
running the asylum. Different scoping/lookup rules depending on
whether something is "dependent" or "non-dependent" - pathetic. Next
it'll be phases of the moon.

Anyway, now that's done, I just have to figure out why GCC won't let
me use offsetof in a constant expression - ie in exactly the kind of
place you're always going to use offsetof. Where the last lot was an
"oh well, a few hours hunting and fixing" thing, this is a real
killer.

I have huge amounts of data-driven code that *needs* to use offsetof
in constant expressions to initialise static data. I also have code
generators that generate code using offsetof in constant expressions.
It'd be easier to develop my own offshoot of C++ than to never use
offsetof in constant expressions. It's the kind of thing people have
been doing in C and C++ since dinosaurs roamed the Earth, and frankly,
if someone has decided to ban it, that someone is a moron.

Sigh.

 

[Gianni Mariani]

Thanks.

Just read 35.18, 35.19 and 35.20. Now I *know* that the lunatics are
running the asylum. Different scoping/lookup rules depending on
whether something is "dependent" or "non-dependent" - pathetic. Next
it'll be phases of the moon.

Anyway, now that's done, I just have to figure out why GCC won't let
me use offsetof in a constant expression - ie in exactly the kind of
place you're always going to use offsetof. Where the last lot was an
"oh well, a few hours hunting and fixing" thing, this is a real
killer.

GCC does not let you use it as a constant because the standard says so.
SUCCS, I know, just is. You may get some luck out of using pointer to
member types.

I have huge amounts of data-driven code that *needs* to use offsetof
in constant expressions to initialise static data. I also have code
generators that generate code using offsetof in constant expressions.
It'd be easier to develop my own offshoot of C++ than to never use
offsetof in constant expressions. It's the kind of thing people have
been doing in C and C++ since dinosaurs roamed the Earth, and frankly,
if someone has decided to ban it, that someone is a moron.

You can probably refactor to use pointer to member.

If you have a code example, post it in a new thread and lets see if we
can give you a few suggestions.

 

[Michael DOUBEZ]

Stephen Horne a écrit :
[snip]

Anyway, now that's done, I just have to figure out why GCC won't let
me use offsetof in a constant expression - ie in exactly the kind of
place you're always going to use offsetof.

You should be able to use it in a constant expression.
Which version of gcc are you using ?

 

[Erik Wikström]

GCC does not let you use it as a constant because the standard says so.
SUCCS, I know, just is. You may get some luck out of using pointer to
member types.

What is it that I'm missing, offsetof is a macro, it should be expanded
before the compiler can even begin worrying about whether an expression
is constant or not.

 

[James Kanze]

What is it that I'm missing, offsetof is a macro, it should be
expanded before the compiler can even begin worrying about
whether an expression is constant or not.

I'm just guessing, because of course, g++ lets you use a legal
offsetof as a constant, but I'll bet he's actually trying to use
it on something that isn't a PODs, which is undefined behavior.
In that case, g++ generates an error; a lot of compilers will
just give you some random (constant) value.

 

[Stephen Horne]

GCC does not let you use it as a constant because the standard says so.
SUCCS, I know, just is. You may get some luck out of using pointer to
member types.

No - GCC didn't let me use it because it didn't know that the
class/struct parameter was a class/struct, reason being I needed to
add 'typename'. GCC was entirely in the right, and I'd even count it
as a pretty sane aspect of the standard.

It's a bit strange, actually. The errors suggested that GCC does a
pointer-casting macro expansion for offsetof, but as far as I can
tell, what it's supposed to do is macro-expand to a built-in along the
lines of __offsetof__ (that's probably spelt wrong, but you get the
idea).

Anyway, turns out that's the easier thing to fix. Some of the
dependent stuff inheritance problems aren't so easy to fix - when they
work in GCC, they break in VC++ and visa versa. I feel superfluous
private inline access methods with #ifdef bodies approaching in the
near future.

Pointer-to-member types are a problem for several reasons, portability
being one of them since VC++ and probably others need you to drop
nonstandard hints about whether the class is single inheritance,
multiple inheritance or whatever. And when you do use them, the syntax
is a mess, and no-one can read it because no-one uses them enough to
get that used to them. Another one of those untested ideas the
standards people dreamed up, that don't really work in practice.

 

[Stephen Horne]

I'm just guessing, because of course, g++ lets you use a legal
offsetof as a constant, but I'll bet he's actually trying to use
it on something that isn't a PODs, which is undefined behavior.
In that case, g++ generates an error; a lot of compilers will
just give you some random (constant) value.

No.

Read for tone and you'll see I'm seriously pissed off and frustrated,
and that thing about offsetof was ranting rather than a properly
organised request for help. For example, no quoted error messages.

Anyway, I said that GCC wouldn't let me use offsetof in a constant
expression, which *was* perfectly true. I also said "now I have to
figure out why" which I have since done.

The GCC errors directed me to lines that contained nothing much more
than offsetof, and complained about pointer operators etc that weren't
there (hence macro expansion). I just recently read the part of the
GCC manual that tells me it defines offsetof to map to a builtin,
unless I dreamed that up in a stress-related psychotic episode (it's
pretty bad when you hallucinate about reading manuals), making the
macro-expansion-implying errors seem strange.

Some other code relating to alignment used hand-written code that
effectively does an offsetof. When GCC complained about that, I fixed
it by simply using offsetof instead. This seems to suggest GCC is
actually doing something different to macro expansion, since replacing
the code that the macro is traditional expanded to with the macro gave
different behaviour. I guess it doesn't really matter, so long as
offsetof works, which it does - my problem was due to being in a
template again, and the need for "typename", which is perfectly
reasonable for a change.


You've got my interest, though. Why on earth should offsetof for a
non-POD struct/field be undefined? POD vs. non-POD shouldn't change
the layout of a struct. A non-POD struct might act as if it holds
extra data, though that data is external to the struct, but that has
nothing to do with offsetof. Some fields may be private and
inaccessible to offsetof (a compile-time error), some fields may be
pointers or references (not an error - just means you're referring to
the pointer), but that applies to POD and non-POD equally.

Also, it's not as if the macro expansion ever instantiates the type.
It just "imagines" a hypothetical instance at address zero and takes
the address of the field.

About the only thing I can think of which could *almost* legitimately
screw up offsetof would be overriding the * or -> pointer dereference
operators, but even if a macro expansion of offsetof uses ->, it uses
it with a *pointer* left argument rather than the non-pointer object,
so the override is irrelevant. Same goes if the macro uses * and .
instead.

Can you give me a reference to look this up?

I mean, the idea that I can't take the offsetof a field in a data
structure node just because the application is using a non-POD type
for the contained data is beyond ridiculous.

In keeping with my generally pissed-off tone, I'll also ask if the
standards people came up with this one specifically to drive me nuts?
Or are you just baiting me for fun?

 

[James Kanze]

Read for tone and you'll see I'm seriously pissed off and
frustrated, and that thing about offsetof was ranting rather
than a properly organised request for help. For example, no
quoted error messages.

I understood that you were pissed off. Still, g++ is fully
conform with regards to offsetof, and does allow it's use in an
integral constant expression. So presumably, your real problem
is elsewhere. And since most compilers don't complain about use
of offsetof on a non-POD, but g++ does, I made a guess that that
might be the real problem.

Anyway, I said that GCC wouldn't let me use offsetof in a
constant expression, which *was* perfectly true.

No it's not. The following code compiles perfectly well with
g++:

#include <cstddef>
#include <iostream>

struct S
{
int i ;
int j ;
int k ;
} ;

int
main()
{
char a[ offsetof( S, k ) ] ;
std::cout << sizeof( a ) << std::endl ;
return 0 ;
}

You're doing something else wrong.

I also said "now I have to figure out why" which I have since
done.

The GCC errors directed me to lines that contained nothing
much more than offsetof, and complained about pointer
operators etc that weren't there (hence macro expansion). I
just recently read the part of the GCC manual that tells me it
defines offsetof to map to a builtin, unless I dreamed that up
in a stress-related psychotic episode (it's pretty bad when
you hallucinate about reading manuals), making the
macro-expansion-implying errors seem strange.

Some other code relating to alignment used hand-written code
that effectively does an offsetof. When GCC complained about
that, I fixed it by simply using offsetof instead. This seems
to suggest GCC is actually doing something different to macro
expansion, since replacing the code that the macro is
traditional expanded to with the macro gave different
behaviour. I guess it doesn't really matter, so long as
offsetof works, which it does - my problem was due to being in
a template again, and the need for "typename", which is
perfectly reasonable for a change.

Fine. The problem wasn't due to g++'s expansion of offsetof not
being a constant integral expression; it was due to a syntax
error elsewhere.

For the record, g++ defines offsetof as a macro (as required by
the standard), but that macro expands to something like
"__builtin_offsetof( t, e )" (which is, IMHO, the only
reasonable way of implementing it). So what you actually get is
a compiler built in, which is either a constant integral
expression, or causes an error, depending on whether the use of
offsetof is legal or not.

You've got my interest, though. Why on earth should offsetof
for a non-POD struct/field be undefined?

Because it can't be implemented in the general case, and no one
considered it worth the effort of specifying when it would be
legal, and when not, except for a PODS, which is all that is
necessary for C compatibility (and the only reason it's there is
for reasons of C compatibility).

POD vs. non-POD shouldn't change the layout of a struct.

In some cases of inheritance (particularly where virtual
inheritance is involved), the layout is dynamically defined; the
compiler doesn't know it. And the "classical" implementation of
offsetof in C is something like:

#define offsetof( S, e ) ((size_t)(&(((S*)0)->e)))

Which is undefined behavior if you write it, but which the
library implementor can do if he knows that it will get by the
compiler.

Many libraries (e.g. Sun CC, VC++) still use something like
this, and the standards committee didn't want to ban it. And it
fails as soon as the member is private.

A non-POD struct might act as if it holds extra data, though
that data is external to the struct, but that has nothing to
do with offsetof. Some fields may be private and inaccessible
to offsetof (a compile-time error), some fields may be
pointers or references (not an error - just means you're
referring to the pointer), but that applies to POD and non-POD
equally.

References could also cause problems. In general, it would
doubtlessly be possible to loosen the rules somewhat. Doing so
would require a fairly detailed analysis, however, to ensure
that the new rules didn't cause any problem for potential
implementations, and no one on the committee felt the effort was
worthwhile. (The case of POD was "established practice", from
C.)

Also, it's not as if the macro expansion ever instantiates the
type. It just "imagines" a hypothetical instance at address
zero and takes the address of the field.

About the only thing I can think of which could *almost*
legitimately screw up offsetof would be overriding the * or ->
pointer dereference operators, but even if a macro expansion
of offsetof uses ->, it uses it with a *pointer* left argument
rather than the non-pointer object, so the override is
irrelevant. Same goes if the macro uses * and . instead.

Can you give me a reference to look this up?

To look what up? The standard says quite clearly (§18.1/5):

The macro offsetof accepts a restricted set of type
arguments in this International Standard. type shall be
a POD structure or a a POD union. The result of
applying the offsetof macro to a field that is a static
data member or a function member is undefined.

I mean, the idea that I can't take the offsetof a field in a
data structure node just because the application is using a
non-POD type for the contained data is beyond ridiculous.

No. It's a perfectly reasonable constraint.

In keeping with my generally pissed-off tone, I'll also ask if
the standards people came up with this one specifically to
drive me nuts? Or are you just baiting me for fun?

Actually, they're just trying to strike a compromise between the
ideal solution (drop the macro entirely) and C compatibility.

 

[James Kanze]

@mariani.ws> wrote:

[...]

Pointer-to-member types are a problem for several reasons,
portability being one of them since VC++ and probably others
need you to drop nonstandard hints about whether the class is
single inheritance, multiple inheritance or whatever.

That's not quite true. VC++ does require special compiler
options to truly be a C++ compiler (but that's true of all
compilers I know), but if you use them (/vmg, in this case),
pointers to members cause no problems. (At least that I can
tell; most of my experience with pointers to members has been
with pointer to member functions, which definitly don't work
with VC++ unless you specify this option.)

And when you do use them, the syntax is a mess, and no-one can
read it because no-one uses them enough to get that used to
them. Another one of those untested ideas the standards people
dreamed up, that don't really work in practice.

Sort of. In practice, there just isn't that much use for
pointers to members, so there's really no need for a
particularly simple syntax. And the syntax for using pointers
to a data member isn't that bad, either, although I can see
people getting a bit bothered when pointers to member functions
are involved.

 

[Stephen Horne]

No it's not. The following code compiles perfectly well with
g++:

No, it was perfectly true. It wouldn't let me use offsetof in a
constant expression because I was using it wrong. When I use it right,
it lets me use it. Hence I had to figure out why it wouldn't let me
use it in order that I could use it right.

I'm autistic, and you're demanding far more pedantic language from me
than I ever demanded from anyone else!

Fine. The problem wasn't due to g++'s expansion of offsetof not
being a constant integral expression; it was due to a syntax
error elsewhere.

Yes. In my original post, I didn't claim to know why GCC was rejecting
my usage, and while implying that the standard (rather than GCC) might
be at fault, I also made it clear that I didn't know yet who was at
fault. Correct answer - I was at fault. I don't think that really
needs restating any more.

In some cases of inheritance (particularly where virtual
inheritance is involved), the layout is dynamically defined;

I can see that there is a potential problem with a macro expansion
implementation (rather than GCCs wrapper approach) to offsetof, since
doing the field reference would access the virtual table/whatever of
an object that doesn't exist.

Even so, the layout of the class isn't dynamically defined, it's just
dynamically accessed. The obvious solution is to not use the
traditional macro expansion for offsetof, and just provide the actual
offset of the field using a builtin. Every class/struct has one layout
defined at compile-time irrespective of what inheritance features it
uses or anything else.

Which is undefined behavior if you write it, but which the
library implementor can do if he knows that it will get by the
compiler.

Many libraries (e.g. Sun CC, VC++) still use something like
this, and the standards committee didn't want to ban it. And it
fails as soon as the member is private.

Basic principle of C++ as given by Stroustrup - the programmer can do
anything the language/libraries can do. No special cases just to get
standard features implemented that can't be replaced by others. Anyone
is allowed to develop a library.

If the libraries can do something, I should be well within my rights
to develop alternative libraries. Standard libraries in particular
should not need to be implemented using non-standard features.

And it
fails as soon as the member is private.

At compile time, which it should do. Why should I have the right to
access private fields using offsetof? Why would I define a data
structure node and hide some of its fields from myself?

References could also cause problems. In general, it would
doubtlessly be possible to loosen the rules somewhat. Doing so
would require a fairly detailed analysis, however, to ensure
that the new rules didn't cause any problem for potential
implementations, and no one on the committee felt the effort was
worthwhile. (The case of POD was "established practice", from
C.)

And therefore likely to be used in container template libraries,
obviously, where the applications contained data may end up as
non-pod, thus implying that the data structure nodes that end up
containing it become non-POD too.

Which is presumably why Sun CC, VC++ etc do it. But as you say, it's
still *special* behaviour that only compiler vendors are allowed to
use for their library, and therefore it violoates the stated
principles of the C++ language.

Actually, technically not the case in my code, since I don't use the
application type directly within the node - just a proxy that has the
same size and alignment, to be initialised later with placement new
etc, which is technically POD. But even so, that's just because of the
specific data structures I'm using, which need to leave some items
uninitialised a bit like the overhead items in a vector.

In other words, I got lucky - but you can bet that others didn't.

To look what up? The standard says quite clearly (§18.1/5):

Thankyou. That is a useful reference, yes, and my requesting it is
hardly worthy of the "To look what up?".

In my current circumstances, paying for the standard would mean not
eating for a week, and in any case, finding a reference for something
specific in what is presumably a rather large standard is bound to be
non-trivial, whereas you were likely to know the reference since you
were already telling me what the rules are.

No. It's a perfectly reasonable constraint.

A constraint that means C data structure code cannot be adapted and
wrapped in a C++ template? IIRC, one of the basic design principles of
C++ is that people should be able to adapt their C libraries without a
complete rewrite.

A constraint that even you admit that the standards guys had to allow
compiler writers to break because they couldn't make their libraries
work otherwise?

That's what you call reasonable!

Actually, they're just trying to strike a compromise between the
ideal solution (drop the macro entirely) and C compatibility.

Really. And I suppose the modern solution is the member pointer, which
simply doesn't work at all for low level work.

For high level work, it doesn't do the most obvious useful job
(delegates) properly. In itself perfectly reasonable since it can be
used to implement a library that does.

But we're talking about replacing offsetof for low-level data
structure work.

For that, it is overengineered in that it handles a bunch of special
cases that real applications simply don't need. Its mummy-knows-best
safety rules (which are entirely inappropriate for low level code)
mean that it can't be used for many obvious tasks without
reinterpret-casting, and you can't cast it safely because of all those
unwanted special cases and their unknown implementations. The whole
thing is such a disaster that some compilers need you to drop
nonstandard hints (single inheritance, multiple inheritance, etc) in
order to use a member pointer at all.

The only pretence that member pointers have to being suitable for
low-level work is the unnecessarily cryptic syntax.

In short, it's another seems-like-a-nice-idea invention put into the
standard by people who were feeling creative, and not subjected to
testing in the real world prior to announcing that it would be part of
the standard. Then people got overcommitted to it and determined to
keep it even though it doesn't work.

An experiment that we now can't get rid of, like exception
specifications on function declarations, which are just as broken
(your black boxes are no longer black, and even nested black-boxes
cannot be replaced, many layers down, without probably violating all
the exception specifications running through layers of yours and other
peoples code) but at least no-ones trying to force anyone to use them.

As I said in another post, whatever happened to simply formalizing
existing best practice? - the supposed remit of standards authorities.
Experimental features should be implemented as compiler-specific
extensions and only standardized *after* they have proved their worth.

You know - like the alignment features that no-one bothered to
standardise despite at least two opportunities, that have been proving
their worth as non-standard compiler extensions for decades.

Trying to bully people into using member-pointers won't fix the flaws.
offsetof is needed, like it or not. Without it, there is simply *NO*
sane way for library developers to develop production quality
containers. As the standards people are clearly aware, or else they
wouldn't be exempting compiler writers from the rules they impose on
others.

 

[Stephen Horne]

Sort of. In practice, there just isn't that much use for
pointers to members, so there's really no need for a
particularly simple syntax.

This from the guy who just declared that offsetof is only there for C
compatibility, and shouldn't be used.

OK - what's the alternative, then?

You give the impression of being someone who does serious work, and
has almost certainly worked on containers and other low-level
libraries. To hold these opinions, I can only guess that you're either
benefiting from the compiler-vendors I-can-but-you-can't clause, or
else you're one of the standards guys - quite possibly both.

Sigh


BTW - the syntax for pointers to data members is itself overly cryptic
since the scope resolution operator is effectively overloaded. As so
often in C++, the syntax you are reading seemingly deliberately avoids
telling you what it's doing - you have to refer back to declarations
that are potentially in another universe far far away in order to read
virtually anything. Is this a class or a namespace? Is that a normal
member or static? Is C++ a more secure encryption protocol than RSA? -
no, because authorised readers can't decrypt it either.

With the C heritage, that was unavoidable. With new C++ features such
as member pointers, it was a choice made at a time when the people
involved should have learned the lessons of the past. Why not simply
use a keyword. If nothing else was practical, there's certainly all
those reserved underscore-prefixed names.

With rarely used features, IMO, it's *especially* important to have a
keyword - something to look up in your reference guide when you
encounter it in someone elses code and don't know what it is. The
overloaded nature of operator symbols makes this difficult to
impossible unless you already know which variation of the symbol is
being used.

Operator symbols should be there for frequently used everyone-knows-it
features, not for obfuscation. If you test new ideas as
compiler-specific extensions before standardizing them, you should
have some idea which features should be handled which *before* you
commit to everyone living with them for the next 50 years.

 

[Ian Collins]

This from the guy who just declared that offsetof is only there for C
compatibility, and shouldn't be used.

OK - what's the alternative, then?

You give the impression of being someone who does serious work, and
has almost certainly worked on containers and other low-level
libraries. To hold these opinions, I can only guess that you're either
benefiting from the compiler-vendors I-can-but-you-can't clause, or
else you're one of the standards guys - quite possibly both.

Maybe James, like me, in many years of C++ development has never needed
pointers to members, or offsetof. Before you ask, most of my work has
been low level libraries.

 

[peter koch]

No, it was perfectly true. It wouldn't let me use offsetof in a
constant expression because I was using it wrong. When I use it right,
it lets me use it. Hence I had to figure out why it wouldn't let me
use it in order that I could use it right.

I'm autistic, and you're demanding far more pedantic language from me
than I ever demanded from anyone else!


Yes. In my original post, I didn't claim to know why GCC was rejecting
my usage, and while implying that the standard (rather than GCC) might
be at fault, I also made it clear that I didn't know yet who was at
fault. Correct answer - I was at fault. I don't think that really
needs restating any more.


I can see that there is a potential problem with a macro expansion
implementation (rather than GCCs wrapper approach) to offsetof, since
doing the field reference would access the virtual table/whatever of
an object that doesn't exist.

Even so, the layout of the class isn't dynamically defined, it's just
dynamically accessed. The obvious solution is to not use the
traditional macro expansion for offsetof, and just provide the actual
offset of the field using a builtin. Every class/struct has one layout
defined at compile-time irrespective of what inheritance features it
uses or anything else.



Basic principle of C++ as given by Stroustrup - the programmer can do
anything the language/libraries can do. No special cases just to get
standard features implemented that can't be replaced by others. Anyone
is allowed to develop a library.

If the libraries can do something, I should be well within my rights
to develop alternative libraries. Standard libraries in particular
should not need to be implemented using non-standard features.


At compile time, which it should do. Why should I have the right to
access private fields using offsetof? Why would I define a data
structure node and hide some of its fields from myself?


And therefore likely to be used in container template libraries,
obviously, where the applications contained data may end up as
non-pod, thus implying that the data structure nodes that end up
containing it become non-POD too.

Which is presumably why Sun CC, VC++ etc do it. But as you say, it's
still *special* behaviour that only compiler vendors are allowed to
use for their library, and therefore it violoates the stated
principles of the C++ language.

Actually, technically not the case in my code, since I don't use the
application type directly within the node - just a proxy that has the
same size and alignment, to be initialised later with placement new
etc, which is technically POD. But even so, that's just because of the
specific data structures I'm using, which need to leave some items
uninitialised a bit like the overhead items in a vector.

In other words, I got lucky - but you can bet that others didn't.



Thankyou. That is a useful reference, yes, and my requesting it is
hardly worthy of the "To look what up?".

In my current circumstances, paying for the standard would mean not
eating for a week, and in any case, finding a reference for something
specific in what is presumably a rather large standard is bound to be
non-trivial, whereas you were likely to know the reference since you
were already telling me what the rules are.



A constraint that means C data structure code cannot be adapted and
wrapped in a C++ template? IIRC, one of the basic design principles of
C++ is that people should be able to adapt their C libraries without a
complete rewrite.

A constraint that even you admit that the standards guys had to allow
compiler writers to break because they couldn't make their libraries
work otherwise?

The implementor is allowed to do more than the ordinary developer as
he controls the environment and can have special, supporting code.

That's what you call reasonable!



Really. And I suppose the modern solution is the member pointer, which
simply doesn't work at all for low level work.
Why?


For high level work, it doesn't do the most obvious useful job
(delegates) properly. In itself perfectly reasonable since it can be
used to implement a library that does.

But we're talking about replacing offsetof for low-level data
structure work.

For that, it is overengineered in that it handles a bunch of special
cases that real applications simply don't need. Its mummy-knows-best
safety rules (which are entirely inappropriate for low level code)
mean that it can't be used for many obvious tasks without
reinterpret-casting, and you can't cast it safely because of all those
unwanted special cases and their unknown implementations. The whole
thing is such a disaster that some compilers need you to drop
nonstandard hints (single inheritance, multiple inheritance, etc) in
order to use a member pointer at all.

The only pretence that member pointers have to being suitable for
low-level work is the unnecessarily cryptic syntax.

I do not believe it is so cryptic, as soon as you become familiar with
it. And this, I suppose, is the case for most constructs.

In short, it's another seems-like-a-nice-idea invention put into the
standard by people who were feeling creative, and not subjected to
testing in the real world prior to announcing that it would be part of
the standard. Then people got overcommitted to it and determined to
keep it even though it doesn't work.

No. Your idea about the standards committee is quite wrong. The
pointer-to-member is necessary, and offsetof can not be replace it.

An experiment that we now can't get rid of, like exception
specifications on function declarations, which are just as broken
(your black boxes are no longer black, and even nested black-boxes
cannot be replaced, many layers down, without probably violating all
the exception specifications running through layers of yours and other
peoples code) but at least no-ones trying to force anyone to use them.

You can't violate an exception specification, but I agree that the
general case is not so useful. The empty exception-specification comes
in handy now and then, however.

As I said in another post, whatever happened to simply formalizing
existing best practice? - the supposed remit of standards authorities.

What did you have in mind?

Experimental features should be implemented as compiler-specific
extensions and only standardized *after* they have proved their worth.

You know - like the alignment features that no-one bothered to
standardise despite at least two opportunities, that have been proving
their worth as non-standard compiler extensions for decades.

What do you need alignment for? I know there are special cases where
specific alignment can be an advantage, but in general there is very
little need for alignment, and for those,

Trying to bully people into using member-pointers won't fix the flaws.
offsetof is needed, like it or not.

offsetof is not needed - you can't use it in the general case.

Without it, there is simply *NO*
sane way for library developers to develop production quality
containers. As the standards people are clearly aware, or else they
wouldn't be exempting compiler writers from the rules they impose on
others.

What? Could yo please tell me for what containers you need offsetof?
If you look at the implementation of the standard library, you'll find
no offsetof used, so they certainly are not "absolutely" needed.

/Peter

 

[James Kanze]

No, it was perfectly true. It wouldn't let me use offsetof in
a constant expression because I was using it wrong.

But that had nothing to do with the fact that it was a constant
expression. That was pure coincidence.

When I use it right, it lets me use it. Hence I had to figure
out why it wouldn't let me use it in order that I could use it
right.

I'm autistic, and you're demanding far more pedantic language
from me than I ever demanded from anyone else!

The problem with C++ (and all other computer languages I
know) is that they are pedantic. Talking about them generally
requires a great deal more precision and pedantry than for other
things.

[...]

I can see that there is a potential problem with a macro
expansion implementation (rather than GCCs wrapper approach)
to offsetof, since doing the field reference would access the
virtual table/whatever of an object that doesn't exist.

Even so, the layout of the class isn't dynamically defined,
it's just dynamically accessed.

Sorry, but when virtual inheritance is involved, it's
dynamically defined. At least with all of the compilers I know,
and I don't think it's possible to implement it otherwise.

The obvious solution is to not use the traditional macro
expansion for offsetof, and just provide the actual offset of
the field using a builtin.

The problem is that you can't provide it as a constant
expression if there is virtual inheritance.

Every class/struct has one layout defined at compile-time
irrespective of what inheritance features it uses or anything
else.

That's false.

Basic principle of C++ as given by Stroustrup - the programmer
can do anything the language/libraries can do. No special
cases just to get standard features implemented that can't be
replaced by others. Anyone is allowed to develop a library.

Another basic principle: as close to C as possible (but no
closer). The only reason offsetof is in C++ is because it is in
C. C compatibility, however, doesn't apply if you're trying to
use it on something which can't be defined in C.

And the reason offsetof was introduced into C (invented by the C
committee) was precisely because there is no way to do this in
C. That's the justification for a lot of things in the library:
anything concerning I/O, for example.

If the libraries can do something, I should be well within my
rights to develop alternative libraries.

Actually, you could legitimately argue just the opposite. If
there's no way of implementing something in the standard
language, you must offer something in the library which does it.
Thus, I/O, memory management, and in the next version, threads.

Standard libraries in particular should not need to be
implemented using non-standard features.

So how do you propose implementing <iostream> (or <stdio.h>)?
Or the function "time()"? Or malloc() and free()? Things that
can't be implemented using standard features must be provided by
the library. That's a basic principle, and has been since C.

At compile time, which it should do. Why should I have the
right to access private fields using offsetof? Why would I
define a data structure node and hide some of its fields from
myself?

Why should you have the right to find the offset of a data
member, period?

And therefore likely to be used in container template
libraries, obviously, where the applications contained data
may end up as non-pod, thus implying that the data structure
nodes that end up containing it become non-POD too.

??? I don't understand what you're trying to say. There's
certainly no need of offsetof in a container.

Which is presumably why Sun CC, VC++ etc do it.

More likely, they do it that way because that's the way the
implemented it in C, some 25 or 30 years ago. And since it's
something that is almost never used, and has no real use in
cleanly written code, there's been no presure on them to improve
it.

But as you say, it's still *special* behaviour that only
compiler vendors are allowed to use for their library, and
therefore it violoates the stated principles of the C++
language.

Actually, technically not the case in my code, since I don't
use the application type directly within the node - just a
proxy that has the same size and alignment, to be initialised
later with placement new etc, which is technically POD. But
even so, that's just because of the specific data structures
I'm using, which need to leave some items uninitialised a bit
like the overhead items in a vector.

In other words, I got lucky - but you can bet that others
didn't.

I've yet to see any library code which uses it. I have needed
to force alignment at times in some of my code; that's why I
developed my max align types.

In my current circumstances, paying for the standard would
mean not eating for a week,

What is its current price? IIRC, it's well under $100.

and in any case, finding a reference for something specific in
what is presumably a rather large standard is bound to be
non-trivial, whereas you were likely to know the reference
since you were already telling me what the rules are.

Finding all of the relevant passages in the standard IS
distinctly non-trivial. And how. But I really wasn't too sure
whether you were asking about what the standard said, or some
references concerning the rationale---why it said it. There is
no printed or online rationale for the C++ standard; what I know
about why certain things are as they are is from speaking with
people who were involved, or in some cases, because I was
involved myself.

A constraint that means C data structure code cannot be
adapted and wrapped in a C++ template?

C data structure code will only use PODS. Asking it to support
more will require rewriting.

IIRC, one of the basic design principles of C++ is that people
should be able to adapt their C libraries without a complete
rewrite.

I don't believe that anyone considered the possibility that the
C libraries would use templates. (I wonder why?)

A constraint that even you admit that the standards guys had
to allow compiler writers to break because they couldn't make
their libraries work otherwise?

That's what you call reasonable!

You don't seem to understand. The macro is there because it is
present in the C version of the library, and there doesn't seem
to be any justifying necessity to drop it. No one I know has
ever found a use for it, however, so it didn't seem reasonable
to invest any added work to define what additional cases (not
present in C) could be made to work.

Really. And I suppose the modern solution is the member
pointer, which simply doesn't work at all for low level work.

In order to define what the modern solution would be, you first
have to define the problem. As I said, I've yet to find a
problem for which offsetof was a solution.

For high level work, it doesn't do the most obvious useful job
(delegates) properly. In itself perfectly reasonable since it
can be used to implement a library that does.

But we're talking about replacing offsetof for low-level data
structure work.

For that, it is overengineered in that it handles a bunch of
special cases that real applications simply don't need. Its
mummy-knows-best safety rules (which are entirely
inappropriate for low level code) mean that it can't be used
for many obvious tasks without reinterpret-casting, and you
can't cast it safely because of all those unwanted special
cases and their unknown implementations. The whole thing is
such a disaster that some compilers need you to drop
nonstandard hints (single inheritance, multiple inheritance,
etc) in order to use a member pointer at all.

Having written parts of OS kernel code in C++, a garbage
collector in C++, marshalling and demarshalling code in C++, and
who knows what all else, I can say from experience that you
don't know what you're talking about. You do need a few
casts---it's normal that more dangerous actions are readily
visible---but C++ is quite effective at this level.

The only pretence that member pointers have to being suitable
for low-level work is the unnecessarily cryptic syntax.

I've not used member pointers that much either, and not really
at a low level.

In short, it's another seems-like-a-nice-idea invention put
into the standard by people who were feeling creative, and not
subjected to testing in the real world prior to announcing
that it would be part of the standard. Then people got
overcommitted to it and determined to keep it even though it
doesn't work.

An experiment that we now can't get rid of, like exception
specifications on function declarations, which are just as
broken (your black boxes are no longer black, and even nested
black-boxes cannot be replaced, many layers down, without
probably violating all the exception specifications running
through layers of yours and other peoples code) but at least
no-ones trying to force anyone to use them.

Actually, empty exception specifications are very useful.
They're used in the standard at times, for a reason.

As I said in another post, whatever happened to simply
formalizing existing best practice? - the supposed remit of
standards authorities. Experimental features should be
implemented as compiler-specific extensions and only
standardized *after* they have proved their worth.

There is a fine line which needs to be tread. I personally
think that C++98 did go to far with "experimenting". But don't
forget that when the committee started work, there were no
templates, and no exceptions. The initial project from ISO
specified existing practice, plus exceptionally useful
extensions, naming templates, exceptions and garbage collection
as examples of the latter.

You know - like the alignment features that no-one bothered to
standardise despite at least two opportunities, that have been
proving their worth as non-standard compiler extensions for
decades.

Trying to bully people into using member-pointers won't fix
the flaws. offsetof is needed, like it or not.

Offsetof is not needed. The proof: no one is using it, even in
C.

Without it, there is simply *NO* sane way for library
developers to develop production quality containers.

Bullshit. I've used several different pre-standard libraries,
and several different implementations of the standard library,
and none of them use offsetof.

As the standards people are clearly aware, or else they
wouldn't be exempting compiler writers from the rules they
impose on others.

Bullshit. The C committee first defined it as a macro because
someone, somewhere, thought that it might be useful. Turns out
that it wasn't, but you can't remove it once it's there.

 

[James Kanze]

This from the guy who just declared that offsetof is only
there for C compatibility, and shouldn't be used.

OK - what's the alternative, then?

What problem do they solve? I've used pointer to member
functions a couple of times, but never a pointer to a data
member.

You give the impression of being someone who does serious
work, and has almost certainly worked on containers and other
low-level libraries. To hold these opinions, I can only guess
that you're either benefiting from the compiler-vendors
I-can-but-you-can't clause, or else you're one of the
standards guys - quite possibly both.

Neither. I've just never found any need for pointers to a data
member. Nor offsetof. They're solutions looking for a problem.

And FWIW, I've implemented a full C standard library, and
several pre-standard container libraries for C++. I've also
implemented marshalling/demarshalling code for low-level data
communications, persistency and the like. And memory management
(globale operator new()/operator delete()). I have needed some
non-standard features: open(),read(),write(),close(), etc. for
my implementation of <stdio.h>, sbrk() for my implementation of
malloc/free (this was before mmap() was an option), etc. Even
today, I need non-standard features for some of my work.

There are some important features which are missing in C++. The
possibility to specify alignment, for example. (I actually
wrote a proposal for this for the C90 standard, but I think I
submitted it a bit too late.) And some support for full data
synchronization in the standard library; I can't use fstream for
persistency because of this. More generally, the absence of
garbage collection means I have to write a lot of code which
would otherwise be unnecessary.

I can't see any need for offsetof, however.

 

[Stephen Horne]

Sorry, but when virtual inheritance is involved, it's
dynamically defined. At least with all of the compilers I know,
and I don't think it's possible to implement it otherwise.

I'm sorry, but it's basic textbook. The compiler determines what is
inherited and places each inherited class at a particualar offset
within the derived class, as if it were member data (which it is).

Every inherited class gets a fixed offset, thus every inherited member
gets a fixed offset (ignoring statics etc). Those offsets may not
apply for classes that inherit this derived class, due to the sharing
aspect for virtual inheritance, but that's besides the point - right
at the start I said it was necessary to name the right struct/class.

We are dealing with a class/struct and a member. No dynamic issues as
there is no dynamic object in the first place.

The problem is that you can't provide it as a constant
expression if there is virtual inheritance.

Yes you can, as mentioned above. The problem is that you're imagining
casting-through-the-class-hierarchy issues where none exist.

That's false.

Back at you.

Actually, you could legitimately argue just the opposite. If
there's no way of implementing something in the standard
language, you must offer something in the library which does it.
Thus, I/O, memory management, and in the next version, threads.


So how do you propose implementing <iostream> (or <stdio.h>)?
Or the function "time()"? Or malloc() and free()? Things that
can't be implemented using standard features must be provided by
the library. That's a basic principle, and has been since C.

You know damn well what the context is here, and you know damn well
that Stroustrup and others have made a big deal of the supposed fact
that the C++ standard library *provides* tools, but only as one
choice. That they are explicitly acknowledged to be not perfect for
everything.

There is nothing about a container library that inherently requires
you to look outside of the language for the necessary tools. If C++
cannot allow programmers to develop containers without requiring
non-standard features, it's a joke.

Why should you have the right to find the offset of a data
member, period?

Because without it, C++ is not fit for purpose, period.

??? I don't understand what you're trying to say. There's
certainly no need of offsetof in a container.

Rubbish, but beside the point.

If you explicitly encourage people to bring C code and skillsets into
C++, you cannot complain when they do so.

What is its current price? IIRC, it's well under $100.

I'm on benefits. After the rent topup fraud (local governments way of
avoiding paying full housing benefit - they simply define "market
rent" to be around 2/3 of the cheapest market rent available and
refuse to pay the rest) my budget for everything that is not a fixed
bill is 35 UKP per week at best. I guess I could save a bit by not
showering and not wearing deoderant. It's not my preferred option.

Finding all of the relevant passages in the standard IS
distinctly non-trivial. And how. But I really wasn't too sure
whether you were asking about what the standard said, or some
references concerning the rationale---why it said it.

I was asking for rationale as well, IIRC, but only if you know it. I'm
not holding you responsible except to the extent that you appear to be
complicit ;-)

I don't believe that anyone considered the possibility that the
C libraries would use templates. (I wonder why?)

A *STANDARD* technique for moving to C++ is to wrap old C code in
templates or other adaptors, with minimal rewriting - which you
clearly know. As you say...

More likely, they do it that way because that's the way the
implemented it in C, some 25 or 30 years ago.

So it is of course blatantly obvious that people will use templates
with old C libraries.

In order to define what the modern solution would be, you first
have to define the problem. As I said, I've yet to find a
problem for which offsetof was a solution.

This is trivial.

I need to reference a field in a struct (not a particular instance). I
control the definition of the struct, but some fields take types that
are template parameters (and can therefore be a source of non-PODness)
In some cases, these are the fields I need to reference - but I don't
need to reference (or even know about) members within those
template-parameter types.

My template is a thin wrapper around non-typesafe data-driven code.
The field-reference needs to be stored as an initialised value in a
static data block - a bit like a closure.

Doing this with member pointers seems to cause compilers to choke -
they don't want to treat the member pointer as a constant expression
for whatever reason each particular compiler comes up with, that
reason varying from compiler to compiler depending on the compromises
they made with member pointer implementation.

Hence offsetof, which is a simple, easy to use way around this issue
and - once you're confident of your access functions - perfectly safe
too.

At least until someone decides to restrict it.

*Maybe* compilers are better at handling member pointers now, but not
in my experience as of yet (bare in mind that I've just scratched the
surface with GCC - other than that my most recent compilers are VC++
2003 and Borland C++ 5.02) and in any case they certainly weren't
working when I was writing the initial code for these containers .

Which I tried to write according to the best pseudo-standards and
promises available to me at the time - an edition of Stroustrup that I
gave away when I bought the special edition. Only I couldn't, so I did
what worked instead.

Having written parts of OS kernel code in C++, a garbage
collector in C++, marshalling and demarshalling code in C++, and
who knows what all else, I can say from experience that you
don't know what you're talking about. You do need a few
casts---it's normal that more dangerous actions are readily
visible---but C++ is quite effective at this level.

Can you define the rules in which casting from one member pointer type
to another is safe, independent of any one compiler? I'll bet you
can't because these casts are explicitly undefined.

In other words, as soon as you declare that you need to cast member
pointers to get anything done, you equally declare that using member
pointers in any portable library is at best problematic and
nonportable.

I have nothing against more dangerous approaches when they are needed,
but I found that member pointers simply didn't work, so managing the
risk involved with offsetof was my only option. offsetof was the
*best* option at the time, chosen after trying and rejecting member
pointers because they didn't work.

If using member pointers requires casts that make it dangerous, just
what is the benefit relative to offsetof anyway? I say better to use a
tool where you know how it works (it's not implementation specific or
variable dependent on things you can't know), and which therefore
allows you to manage the risks.

Actually, empty exception specifications are very useful.
They're used in the standard at times, for a reason.

Empty exception specifications are a special case which has value, but
that's not a justification for the full exception specification part
of the language. Sometimes, extending and generalizing a good idea is
a bad idea.

Offsetof is not needed. The proof: no one is using it, even in
C.

By your own argument, people have used it in some pretty important
libraries in C, which have now become C++ libraries. Ergo it is used
in both C and C++. In standard library implementations, no less.

 

[Ian Collins]

On Tue, 14 Oct 2008 15:58:15 -0700 (PDT), James Kanze


Because without it, C++ is not fit for purpose, period.

Cutting though the waffle, why?

I've been using C++ for everything from drivers to libraries to GUIs as
a major part of my day job for many years and I've never had cause to
find the offset of a data member.

 

[Stephen Horne]

Cutting though the waffle, why?

I've been using C++ for everything from drivers to libraries to GUIs as
a major part of my day job for many years and I've never had cause to
find the offset of a data member.

Search for "thin wrapper" in the post.