Boost Workshop at OOPSLA 2004

[Paul Mensonides]

"Andrei Alexandrescu (See Website for Email)"

Cool. Before continuing the discussion, I have a simple question - how does
your implementation cope with commas in template types, for example:

TYPELIST(vector<int, my_allocator<int> >, vector<float>)

would correctly create a typelist of two elements? If not, what steps do I
need to take to creat such a typelists (aside from a typedef)?

You'd just have to parenthesize types that contain open commas:

TYPELIST((vector<int, my_allocator<int> >), vector<float>)

The DECODE macro in the example removes parentheses if they exist. E.g.

CHAOS_PP_DECODE(int) // int
CHAOS_PP_DECODE((int)) // int

Using parentheses is, of course, only necessary for types that contain open
commas. (There is also a ENCODE macro that completes the symmetry, but it is
unnecessary.)

There are also a several other alternatives. It is possible to pass a type
through a system of macros without the system of macros being intrusively
modified (with DECODE or similar). This, in Chaos terminology, is a "rail". It
is a macro invocation that effectively won't expand until some context is
introduced. E.g.

#include <chaos/preprocessor/punctuation/comma.h>

#define A(x) B(x)
#define B(x) C(x)
#define C(x) D(x)
#define D(x) x

A(CHAOS_PP_COMMA())

This will error with too many arguments to B. However, the following disables
evaluation of COMMA() until after the system "returns" from A:

#include <chaos/preprocessor/punctuation/comma.h>
#include <chaos/preprocessor/recursion/rail.h>

#define A(x) B(x)
#define B(x) C(x)
#define C(x) D(x)
#define D(x) x

CHAOS_PP_WALL(A(
CHAOS_PP_UNSAFE_RAIL(CHAOS_PP_COMMA)()
))

(There is also CHAOS_PP_RAIL that is similar, but getting into the difference
here is too complex a subject.)

In any case, the expansion of COMMA is inhibited until it reaches the context
established by WALL. The same thing can be achieved for types non-intrusively.
Chaos has two rail macros designed for this purpose, TYPE and TYPE_II. The
first, TYPE, is the most syntactically clean, but is only available with
variadics:

#include <chaos/preprocessor/facilities/type.h>
#include <chaos/preprocessor/recursion/rail.h>

#define A(x) B(x)
#define B(x) C(x)
#define C(x) D(x)
#define D(x) x

CHAOS_PP_WALL(A(
CHAOS_PP_TYPE(std:air<int, int>)
))
// std:air<int, int>

The second, TYPE_II, is more syntactically verbose, but it works even without
variadics without counting commas:

#include <chaos/preprocessor/facilities/type.h>
#include <chaos/preprocessor/recursion/rail.h>

#define A(x) B(x)
#define B(x) C(x)
#define C(x) D(x)
#define D(x) x

CHAOS_PP_WALL(A(
CHAOS_PP_TYPE_II(CHAOS_PP_BEGIN std:air<int, int> CHAOS_PP_END)
))
// std:air<int, int>

Thus, you *could* make a typelist using rails such as this to protect
open-comma'ed types, but for typelists (which inherently deal with types), it
would be pointless. Rails are more useful when some arbitrary data that you
need to pass around happens to be a type, but doesn't necessarily have to be.

Regards,
Paul Mensonides

 

[Andrei Alexandrescu \(See Website for Email\)]

(from another post)

The DECODE macro in the example removes parentheses if they exist. E.g.

CHAOS_PP_DECODE(int) // int
CHAOS_PP_DECODE((int)) // int

That's what I was hoping for; thanks.

(back to this other post)

Regarding actual code examples, here's a Chaos-based version of the old
TYPELIST_1, TYPELIST_2, etc., macros. Note that this example uses
variadics
which are likely to be added with C++0x. (It is also a case-in-point of
why
variadics are important.)

#include <chaos/preprocessor/control/iif.h>
#include <chaos/preprocessor/detection/is_empty.h>
#include <chaos/preprocessor/facilities/encode.h>
#include <chaos/preprocessor/facilities/split.h>
#include <chaos/preprocessor/limits.h>
#include <chaos/preprocessor/recursion/basic.h>
#include <chaos/preprocessor/recursion/expr.h>

#define TYPELIST(...) TYPELIST_BYPASS(CHAOS_PP_LIMIT_EXPR, __VA_ARGS__)
#define TYPELIST_BYPASS(s, ...) \
CHAOS_PP_EXPR_S(s)(TYPELIST_I( \
CHAOS_PP_OBSTRUCT(), CHAOS_PP_PREV(s), __VA_ARGS__, \
)) \
/**/
#define TYPELIST_INDIRECT() TYPELIST_I
#define TYPELIST_I(_, s, ...) \
CHAOS_PP_IIF _(CHAOS_PP_IS_EMPTY_NON_FUNCTION(__VA_ARGS__))( \
Loki::NilType, \
Loki::TypeList< \
CHAOS_PP_DECODE _(CHAOS_PP_SPLIT _(0, __VA_ARGS__)), \
CHAOS_PP_EXPR_S _(s)(TYPELIST_INDIRECT _()( \
CHAOS_PP_OBSTRUCT _(), CHAOS_PP_PREV(s), \
CHAOS_PP_SPLIT _(1, __VA_ARGS__) \
)) \
) \
/**/

I am sure I will develop a lot more appreciation for this solution once I
will fully understand all of the clever techniques and idioms used.

For now, I hope you will agree with me that the above fosters learning yet
another programming style, which is different than straight programming,
template programming, or MPL-based programming.

I would also like to compare the solution above with the "imaginary" one
that I have in mind as a reference. It uses LISP-macros-like artifacts and a
few syntactic accoutrements.

$define TYPELIST() { Loki::NullType }
$define TYPELIST(head $rest more) {
Loki::Typelist< head, TYPELIST(more) >
}

About all that needs to be explained is that "$rest name" binds name to
whatever other comma-separated arguments follow, if any, and that the
top-level { and } are removed when creating the macro.

If you would argue that your version above is more or as elegant as this
one, we have irreducible opinions. I consider your version drowning in
details that have nothing to do with the task at hand, but with handling the
ways in which the preprocessor is inadequate for the task at hand. Same
opinion goes for the other version below:

#include <chaos/preprocessor/facilities/encode.h>
#include <chaos/preprocessor/lambda/ops.h>
#include <chaos/preprocessor/punctuation/comma.h>
#include <chaos/preprocessor/recursion/expr.h>
#include <chaos/preprocessor/tuple/for_each.h>

#define TYPELIST(...) \
CHAOS_PP_EXPR( \
CHAOS_PP_TUPLE_FOR_EACH( \
CHAOS_PP_LAMBDA(Loki::TypeList<) \
CHAOS_PP_DECODE_(CHAOS_PP_ARG(1)) CHAOS_PP_COMMA_(), \
(__VA_ARGS__) \
) \
Loki::NilType \
CHAOS_PP_TUPLE_FOR_EACH( \
CHAOS_PP_LAMBDA(>), (__VA_ARGS__) \
) \
) \
/**/

This implementation can process up to ~5000 types and there is no list of
5000
macros anywhere in Chaos. (There are also other, more advanced methods
capable
of processing trillions upon trillions of types.)

I guess you have something that increases with the logarithm if that number,
is that correct?

The most fundamental thing would be the ability to separate the first
arbitrary
preprocessing token (or whitespace separation) from those that follow it
in a
sequence of tokens and be able to classify it in some way (i.e. determine
what
kind of token it is and what its value is). The second thing would be the
ability to take a single preprocessing token and deconstruct it into
characters.
I can do everything else, but can only do those things in a limited ways.

I understand the first desideratum, but not the second. What would be the
second thing beneficial for?


Andrei

 

[Paul Mensonides]

I am sure I will develop a lot more appreciation for this solution
once I will fully understand all of the clever techniques and idioms
used.
Yes.

For now, I hope you will agree with me that the above fosters
learning yet another programming style, which is different than
straight programming, template programming, or MPL-based programming.

Definitely. It is a wholly different language.

I would also like to compare the solution above with the "imaginary"
one that I have in mind as a reference. It uses LISP-macros-like
artifacts and a few syntactic accoutrements.

$define TYPELIST() { Loki::NullType }
$define TYPELIST(head $rest more) {
Loki::Typelist< head, TYPELIST(more) >
}

About all that needs to be explained is that "$rest name" binds name
to whatever other comma-separated arguments follow, if any, and that
the top-level { and } are removed when creating the macro.

So, for the above you need (basically) two things: overloading on number of
arguments and recursion. Both of those things are already indirectly possible
(with the qualification that overloading on number of arguments is only possible
with variadics). That isn't to say that those facilities wouldn't be useful
features of the preprocessor, because they would. I'm merely referring to those
things which can be done versus those things which cannot with the preprocessor
as it currently exists. I'm concerned more with functionality than I am with
syntactic cleanliness.

If you would argue that your version above is more or as elegant as
this one, we have irreducible opinions.

The direct "imaginary" version is obviously more elegant.

I consider your version
drowning in details that have nothing to do with the task at hand,
but with handling the ways in which the preprocessor is inadequate
for the task at hand. Same opinion goes for the other version below:

But the preprocessor *is* adequate for the task. It just isn't as syntactically
clean as you'd like it to be.

I guess you have something that increases with the logarithm if that
number, is that correct?

Exponential structure, yes. A super-reduction of the idea is this:

#define A(x) B(B(x))
#define B(x) C(C(x))
#define C(x) x

Here, the x argument gets scanned for expansion with a base-2 exponential. With
about 25 macros you're already into millions of scans. Each of those scans can
be an arbitrary computational step.

I understand the first desideratum, but not the second. What would be
the second thing beneficial for?

Identifier and number processing primarily, but also string and character
literals. Given those two things alone you could write a C++ interpreter with
the preprocessor--or, much more simply, you could trivially write your imaginary
example above. Speaking of which, you can already write interpreters that get
close. The one thing that you cannot do is get beyond of arbitrary
preprocessing tokens. They would have to be quoted in some way.

Regards,
Paul Mensonides

 

[Arkadiy Vertleyb]

"Andrei Alexandrescu (See Website for Email)"


In that case, I agree. Internally, Boost PP is a mess--but a mess caused by
lackluster conformance.

I think the actual value of a library can be determined as a
difference between the "mess" it gets as its input, and the "mess" (if
some still remains) its user gets on its output. In this regard, IMO,
it's difficult to overestimate the value of the Boost PP library, no
matter how messy its implementation might be.

(Just a thought from one of recently converted former PP-haters)

Regards,
Arkadiy

 

[Daveed Vandevoorde]

"Andrei Alexandrescu wrote:
[...]

Maybe "export" which is so broken and so useless and so abusive that its
implementers have developed Stockholm syndrome during the long years that
took them to implement it?

How is "export" useless and broken?

Have you used it for any project? I find it very pleasant
to work with in practice.

Daveed

 

[Andrei Alexandrescu \(See Website for Email\)]

Definitely. It is a wholly different language.

Not it only remains for me to convince you that that's a disadvantage ).

So, for the above you need (basically) two things: overloading on number
of
arguments and recursion. Both of those things are already indirectly
possible
(with the qualification that overloading on number of arguments is only
possible
with variadics). That isn't to say that those facilities wouldn't be
useful
features of the preprocessor, because they would. I'm merely referring to
those
things which can be done versus those things which cannot with the
preprocessor
as it currently exists. I'm concerned more with functionality than I am
with
syntactic cleanliness.

I disagree it's only syntactic cleanliness. Lack of syntactic cleanliness is
the CHAOS_PP_ that you need to prepend to most of your library's symbols.
But let me pull the code again:

#define REPEAT(count, macro, data) \
REPEAT_S(CHAOS_PP_STATE(), count, macro, data) \
/**/
#define REPEAT_S(s, count, macro, data) \
REPEAT_I( \
CHAOS_PP_OBSTRUCT(), CHAOS_PP_NEXT(s), \
count, macro, data \
) \
/**/
#define REPEAT_INDIRECT() REPEAT_I
#define REPEAT_I(_, s, count, macro, data) \
CHAOS_PP_WHEN _(count)( \
CHAOS_PP_EXPR_S _(s)(REPEAT_INDIRECT _()( \
CHAOS_PP_OBSTRUCT _(), CHAOS_PP_NEXT(s), \
CHAOS_PP_DEC(count), macro, data \
)) \
macro _(s, CHAOS_PP_DEC(count), data) \
) \
/**/

As far as I understand, REPEAT, REPEAT_S, REPEAT_INDIRECT, REPEAT_I, and the
out-of-sight CHAOS_PP_STATE, CHAOS_PP_OBSTRUCT, CHAOS_PP_EXPR_S are dealing
with the preprocessor alone and have zero relevance to the task. The others
implement an idiom for looping that I'm sure one can learn, but is far from
familiar to a C++ programmer. To say that that's just a syntactic
cleanliness thing is a bit of a stretch IMHO. By the same argument, any
Turing complete language will do at the cost of "some" syntactic
cleanliness.

But the preprocessor *is* adequate for the task. It just isn't as
syntactically
clean as you'd like it to be.

I maintain my opinion that we're talking about more than syntactic
cleanliness here. I didn't say the preprocessor is "incapable" for the task.
But I do believe (and your code strengthened my belief) that it is
"inadequate". Now I looked on www.m-w.com and I saw that inadequate means "
: not adequate : INSUFFICIENT; also : not capable " and that adequate means
"sufficient for a specific requirement" and "lawfully and reasonably
sufficient". I guess I meant it as a negation of the last meaning, and even
that is a bit too strong. Obviously the preprocessor is "capable", because
hey, there's the code, but it's not, let me rephrase - very "fit" for the
task.

Exponential structure, yes. A super-reduction of the idea is this:

#define A(x) B(B(x))
#define B(x) C(C(x))
#define C(x) x

Here, the x argument gets scanned for expansion with a base-2 exponential.
With
about 25 macros you're already into millions of scans. Each of those
scans can
be an arbitrary computational step.

Wouldn't it be nicer if you just had one mechanism (true recursion or
iteration) that does it all in one shot?


Andrei

 

[Walter]

"Andrei Alexandrescu wrote:
[...]

Maybe "export" which is so broken and so useless and so abusive that its
implementers have developed Stockholm syndrome during the long years that
took them to implement it?

How is "export" useless and broken?

Have you used it for any project? I find it very pleasant
to work with in practice.

From my readings on export, the benefits are supposed to be:

1) avoid pollution of the name space with the names involved in the
implementation details of the template, sometimes called "code hygene"

2) template source code hiding

3) faster compilation

Examining each of these in turn:

1) Isn't this what C++ namespaces are for?

2) Given the ease with which Java .class files are "decompiled" back into
source code, and the fact that precompiled templates will necessarilly
contain even more semantic info than .class files, it is hard to see how
exported templates offer secure hiding of template implementations. It is
not analagous to the problem of "turning hamburger back into a cow" that
object file decompilers have. While some "security through obscurity" may be
achieved by not documenting the file format, if the particular compiler
implementation is popular, there surely will appear some tool to do it.

3) The faster compilation is theoretically based on the idea that the
template implementation doesn't need to be rescanned and reparsed every time
it is #include'd. However, many modern C++ compilers already support
"precompiled headers", which already provide just that capability without
export at all. I'd be happy to accept a compilation speed benchmark
challenge of Digital Mars DMC++ with precompiled headers vs an export
implementation.

I look at export as a cost/benefit issue. What are the benefits, and what
are the costs? The benefits, as discussed above, are not demonstrated to be
significant. The cost, however, is enormous - 2 to 3 man years of
implementation effort, which means that other, more desirable, features
would necessarilly get deferred/delayed.

What export does is attempt to graft some import model semantics onto the
inclusion model semantics. The two are fundamentally at odds, hence all the
complicated rules and implementation effort. The D Programming Language
simply abandons the inclusion model semantics completely, and goes instead
with true imported modules. This means that exported templates in D are
there "for free", i.e. they involve no extra implementation effort and no
strange rules. And after using them for a while, yes it is very pleasant to
be able to do:

----- foo.d ----
template Foo(T) { T x; }
----- bar.d ----
import foo;

foo.Foo!(int); // instantiate template Foo with 'int' type

 

[Hyman Rosen]

From my readings on export, the benefits are supposed to be:

The benefits of export are the same as the benefits of
separate compilation of non-templated code. That is, for
normal code, I can write

In joe.h:
struct joe { void frob(); double gargle(double); };
In joe.c:
namespace { void fiddle() { } double grok() { return 3.7; } }
void joe::frob() { fiddle(); }
double joe::gargle(double d) { return d + grok(); }

And users of the joe class only include joe.h, and never
have to worry about joe.c. Without export, if joe were a
template class, then every compilation unit which uses a
method of joe would have to include the implementation of
those methods bodily. This constrains the implementation;
for example, that anonymous namespace wouldn't work. With
export, users include the header file and they are done.

 

[tom_usenet]

"Andrei Alexandrescu wrote:
[...]

Maybe "export" which is so broken and so useless and so abusive that its
implementers have developed Stockholm syndrome during the long years that
took them to implement it?

How is "export" useless and broken?

Have you used it for any project? I find it very pleasant
to work with in practice.

From my readings on export, the benefits are supposed to be:

1) avoid pollution of the name space with the names involved in the
implementation details of the template, sometimes called "code hygene"

2) template source code hiding

3) faster compilation

4) avoid pollution of the lookup context in which the template
definition exists with names from the instantiation context, except
where these are required (e.g. dependent names).

5) reduce dependencies

Examining each of these in turn:

Hmm, this is all rehashing I think.

1) Isn't this what C++ namespaces are for?

That ignores macros and argument dependent lookup, which transcend
namespaces (or rather operate in a slightly unpredictable set of
namespaces in the case of the latter).

2) Given the ease with which Java .class files are "decompiled" back into
source code, and the fact that precompiled templates will necessarilly
contain even more semantic info than .class files, it is hard to see how
exported templates offer secure hiding of template implementations. It is
not analagous to the problem of "turning hamburger back into a cow" that
object file decompilers have. While some "security through obscurity" may be
achieved by not documenting the file format, if the particular compiler
implementation is popular, there surely will appear some tool to do it.

Precompiled templates don't need more semantic information than class
files. In particular, all code involving non-dependent names can be
fully compiled, or at the very least, the names can be removed from
the precompiled template file. In other words, the file format might
consist of a combination of ordinary object code intermingled with
other more detailed stuff.

I believe that EDG may be working on something related to this, but
they're keeping fairly schtum about it so I don't know the details.

3) The faster compilation is theoretically based on the idea that the
template implementation doesn't need to be rescanned and reparsed every time
it is #include'd. However, many modern C++ compilers already support
"precompiled headers", which already provide just that capability without
export at all. I'd be happy to accept a compilation speed benchmark
challenge of Digital Mars DMC++ with precompiled headers vs an export
implementation.

The compilation speed advantages also come from dependency reductions.
If template definitions are modified, only the template
specializations need to be recompiled. If the instantiation context
(which I believe only consists of all extern names) is saved in each
case, then the instantiation can be recompiled without having to
recompile the whole TU containing the implicit template instantiation.

I look at export as a cost/benefit issue. What are the benefits, and what
are the costs? The benefits, as discussed above, are not demonstrated to be
significant. The cost, however, is enormous - 2 to 3 man years of
implementation effort, which means that other, more desirable, features
would necessarilly get deferred/delayed.

What export does is attempt to graft some import model semantics onto the
inclusion model semantics. The two are fundamentally at odds, hence all the
complicated rules and implementation effort. The D Programming Language
simply abandons the inclusion model semantics completely, and goes instead
with true imported modules. This means that exported templates in D are
there "for free", i.e. they involve no extra implementation effort and no
strange rules. And after using them for a while, yes it is very pleasant to
be able to do:

----- foo.d ----
template Foo(T) { T x; }
----- bar.d ----
import foo;

foo.Foo!(int); // instantiate template Foo with 'int' type
----------------

It seems to me that there were three alternatives in C++.

1. Don't support any kind of model except the inclusion one. If this
is done, two-phase name lookup should have been dropped as well, since
it is confusing at best, and only really necessary if export is to be
supported. It catches some errors earlier, but at some expense to
programmers. The "typename" and "template" disambiguators could also
blissfully be dropped.

2. Add module support to C++. This obviously is as large a proposal as
export, but clearly #include is unhelpful in a language as complex as
C++; really you just want to import extern names from particular
namespaces, not textually include a whole file.

3. Support separate compilation of templates in some form, without
modules. If you do this, I think you pretty much end up with two phase
name lookup and export (indicating that it isn't broken).

The committee rejected 1, I doubt anyone suggested 2, so 3 was the
remaining choice.

Personally, I might well have gone with 1; templates are complicated
enough, and two-phase name lookup and export have unnecessarily made
them much more complex. On the other hand, 1 doesn't provide the
benefits that export does provide.

So, ignoring implementation difficultly, I think export does just win
as a useful feature. With the implementation difficultly, it's not so
clear.

Tom

 

[Andrei Alexandrescu \(See Website for Email\)]

"Andrei Alexandrescu wrote:
[...]

Maybe "export" which is so broken and so useless and so abusive that its
implementers have developed Stockholm syndrome during the long years
that
took them to implement it?

How is "export" useless and broken?

Have you used it for any project? I find it very pleasant
to work with in practice.

Haven't used export, and not because I didn't wanna.

[Whom do you think I referred to when mentioning the Stockholm syndrome?
)]

I'd say, a good feature, like a good business idea, can be explained in a
few words. What is the few-words good explanation of export? (I *am*
interested.)

In addition, a good programming language feature does what it was intended
to do (plus some other neat things )), and can be reasonably implemented.

I think we all agree "export" fell the last test.

Does it do what it was intended to do? (Again, I *am* interested.)

A summary of what's the deal with export would be of great help to at least
myself, so I'd be indebted to anyone who'd give me one. For full disclosure,
my current perception is:

1. It's hard to give a good account of what export does in a few words, at
least an account that's impressive.

2. export failed horribly at doing what was initially supposed to do. I
believe what it was supposed to do was true (not "when"s and "cough"s and
"um"s) separate compilation of templates. Admittedly, gaffes in other areas
of language design are at fault for that failure. Correct me if I'm wrong.

3. export failed miserably at being reasonably easy to implement.

Combined with 1 and 2, I can only say: at the very best, export is a Pyrrhic
victory.


Andrei

 

[Thorsten Ottosen]

| Walter wrote:
| >From my readings on export, the benefits are supposed to be:
|
| The benefits of export are the same as the benefits of
| separate compilation of non-templated code.

| With
| export, users include the header file and they are done.

ok, but with separate compilation we get faster compilation. How much faster will/can the export version be?

Currently is also seriously tedious to implement class template member functions outside the class. I hope experience with
export can help promote class namespaces as described by Carl Daniel (see
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2003/n1420.pdf ).

br

Thorsten

 

[Walter]

The benefits of export are the same as the benefits of
separate compilation of non-templated code. That is, for
normal code, I can write

In joe.h:
struct joe { void frob(); double gargle(double); };
In joe.c:
namespace { void fiddle() { } double grok() { return 3.7; } }
void joe::frob() { fiddle(); }
double joe::gargle(double d) { return d + grok(); }

And users of the joe class only include joe.h, and never
have to worry about joe.c.

They would for templates, since the compiler will need to precompile it at
some point. You'd still have to put a dependency on joe.c in the makefile,
etc., since there's now an order to compiling the source files. Furthermore,
there's no indication to the compiler that the template implementation is in
joe.c, so some sort of cross reference would need building or it'd need to
be manually specified. That's all doable, of course, and is not that big an
issue, but I wished to point out that it isn't quite as simple as object
files are. A similar procedure is necessary for precompiled headers.

Without export, if joe were a
template class, then every compilation unit which uses a
method of joe would have to include the implementation of
those methods bodily.

Yes, that's right. But just what are the benefits of separate compilation?
They are the 3 I mentioned. There is no semantic benefit that namespaces
can't address. (The old problem of needing to separate things because of
insufficient memory for compilation has faded away.)

This constrains the implementation;
for example, that anonymous namespace wouldn't work.

I don't understand why namespaces wouldn't do the job. Isn't that kind of
problem exactly what namespaces were designed to solve?

With export, users include the header file and they are done.

So, we have, for the user:
export template foo ...
v.s.
#include "foo_implementation.h"

and they're done in either case. Sure, the former is slightly prettier, but
if we're going to overhaul C++ for aesthetic appeal, I'd do a lot of things
that are a LOT easier to implement before that one <g>.

 

[Walter]

4) avoid pollution of the lookup context in which the template
definition exists with names from the instantiation context, except
where these are required (e.g. dependent names).

Shouldn't namespaces should cover this? That's what they're for.

5) reduce dependencies

I think that is another facet of 1 and 4.

Hmm, this is all rehashing I think.


That ignores macros

I'll concede that it can help with macros, though I suggest that the
problems with macros remain and would be far better addressed with things
like scoped macros. Export is a particularly backwards way to solve problems
with the preprocessor, sort of like fixing rust by putting duct tape over it

and argument dependent lookup, which transcend
namespaces (or rather operate in a slightly unpredictable set of
namespaces in the case of the latter).

I don't see how this would be a problem.

Precompiled templates don't need more semantic information than class
files. In particular, all code involving non-dependent names can be
fully compiled,

There's very little of that in templates, otherwise, they wouldn't need to
be templates. Realistically, I just don't see compiler vendors trying to mix
object code with syntax trees - the implementation cost is very high and the
benefit for a typical template is nil.

or at the very least, the names can be removed from
the precompiled template file.

Removing a few names doesn't help much - .class files remove names from all
the locals, and that hasn't even slowed down making a decompiler for it.

In other words, the file format might
consist of a combination of ordinary object code intermingled with
other more detailed stuff.

Consider that within the precompiled template file, the compiler will need
to extract the names and offsets of all the members of the template classes,
coupled with the syntax trees of all the template functions, waiting to be
decorated with names and types. That is much more than what's in a .class
file. Another way to see this is that template files will necessarilly be
*before* the semantic stage of the compiler, whereas .class files are
generated *after* the semantic stage; more information can be thrown away in
the latter case.

The compilation speed advantages also come from dependency reductions.
If template definitions are modified, only the template
specializations need to be recompiled. If the instantiation context
(which I believe only consists of all extern names) is saved in each
case, then the instantiation can be recompiled without having to
recompile the whole TU containing the implicit template instantiation.

I'll still be happy to do the benchmark <g>. In my experience with projects
that attempted to maintain a complex dependency database, the time spent
maintaining it exceeded the time spent doing a global rebuild. Worse, the
dependency database was a rich source of bugs, so whenever there were
problems with the result, the first thing one tried was a global rebuild.
(For an example, consider "incremental linker" technology. You'd think that
would be a fairly easy problem, but incremental linkers suffered from so
many bugs that the first step in debugging was to do a clean build. Also,
Digital Mars' optlink does a full build faster than the incremental linkers
do an incremental one. Another example I know of is one where the dependency
database caused a year delay in the project and never did work right,
arguably causing the eventual failure of the entire project.)

So, ignoring implementation difficultly, I think export does just win
as a useful feature. With the implementation difficultly, it's not so
clear.

My problem is with the "ignoring implementation difficulty" bit. I attended
some of the C++ meetings early on, and a clear attitude articulated to me by
more than one member was that implementation difficulty was irrelevant, only
the user experience mattered. My opinion then, as now, is that
implementation difficulty strongly affects the users, since:

1) hard to implement features take a long time to implement, years in the
case of export, so users have to wait
2) hard to implement features usually result in buggy implementations that
take a long time to shake out
3) compiler vendors implement features in different orders, leading to
incompatible implementations
4) spending time on hard to implement features means that other features,
potentially more desirable to users, get backburnered

And we've all seen 1..4 impacting the users for years on end, and is still
ongoing.

There's another issue that may or may not matter depending on who you talk
to: hard to implement features wind up shrinking the number of
implementations. Back in the 80's, I once counted 30 different C compilers
available for just the IBM PC. How many different implementations of C++ are
there now? And it's still shrinking. I venture that a shrinking
implementation base is not good for the long term health of the language.

 

[Hyman Rosen]

They would for templates, since the compiler will need to precompile it at
some point. You'd still have to put a dependency on joe.c in the makefile,
etc., since there's now an order to compiling the source files.

If this comes to you from a library provider, they could ship
compiled versions of the implementation file, just as they now
ship object files (assuming the compiler vendor supplied such
support). If it's your own templates, you simply make your
overall project depend on the implementation files if your
vendor uses the freedom given by 14/9 to force you to compile
the implementations first.

Furthermore, there's no indication to the compiler that the
template implementation is in joe.c, so some sort of cross
reference would need building or it'd need to be manually
specified. That's all doable, of course, and is not that big
an issue, but I wished to point out that it isn't quite as
simple as object files are.

How is it different from object files? For normal source files
you must specify which object files are part of your program,
what their source files are, and what other files they depend
on. Some of this is done automatically by various development
platforms, but it is always done. How are exported templates
different?

But just what are the benefits of separate compilation?
They are the 3 I mentioned. There is no semantic benefit
that namespaces can't address.

I will believe this once you agree that C++ should require
definitions of all functions, template or not, to be included
in every compilation unit which calls them. If you do not
agree that this is a good idea for plain functions, I do not
see why it's a good idea for function templates.

I don't understand why namespaces wouldn't do the job.
Isn't that kind of problem exactly what namespaces were
designed to solve?

No. By requiring that implementations be bodily included
where instantiations are needed, the implementations are,
first, subject to the various pollutions of the instantiation
space, not least of which are macros, and secondly, are not
able to use anonymous namespaces since that will break the
ODR. As I said, unless you can convince me that the inclusion
model is good for ordinary methods, I have no reason to believe
that it's good for template methods.

So, we have, for the user:
export template foo ...
v.s.
#include "foo_implementation.h"
and they're done in either case.

You fail to see that the second case exposes the implementation
to the vagaries of the instantiation environment while the first
does not. Think of macros if nothing else.

 

[Hyman Rosen]

ok, but with separate compilation we get faster compilation.
How much faster will/can the export version be?

Faster by a factor of 3.7. The point is not to have faster
compilation, although that would be nice, but to have cleaner
compilation, so that implementations do not intertwine with
the usage environments any more than required by the lookup
rules.

Currently is also seriously tedious to implement class
template member functions outside the class.

Huh? Why? Just because you have to repeat the template
header part? That's not that much more onerous than
repeating ClassName:: in front of ordinary methods.
Make a macro if it's that bothersome. With export, you
won't have to worry about the macros colliding with
client code!

 

[Hyman Rosen]

What is the few-words good explanation of export?

Here's my attempt:
Because of historical reasons having to do with how templates are
implemented, template methods (and static data members) are
effectively considered inline, and so their definitions must be
included in any compilation unit which requires an instantiation.

The export keyword breaks this requirement. When a template method
is marked as export, its definition does not get included into
compilation units which use it. Instead, it goes into its own source
file(s), and is compiled separately.

The C++ standard permits an implementation to require that the
definition of an exported method or object must be compiled before a
reference to such an object is compiled.

2. export failed horribly at doing what was initially supposed to do. I
believe what it was supposed to do was true separate compilation of templates.

Compilation of templates is obviously not like compilation of ordinary
source, since template instantiation requires information from the
instantiation context and from the template parameters. But what do you
mean by "true" separate compilation? What kind of separate compilation
is not true? I understand that some people wish that export should be a
way to prevent people from examining template implementation code, but
that's hardly something for the standard to worry about. I understand
that some people either think or wish that export should affect how
temnplates are instantiated, but export has nothing to do with that.

To express it as simply as possible, imagine that C++ required that every
function be declared inline, and that therefore the implementation of every
function must be included in any compilation unit that used it. This is the
model that unexported templates labor under, and is what export is designed
to avoid.

 

[Walt Karas]

CALL FOR PAPERS/PARTICIPATION

C++, Boost, and the Future of C++ Libraries
Workshop at OOPSLA
October 24-28, 2004
Vancouver, British Columbia, Canada
http://tinyurl.com/4n5pf

I don't have the time to pursue it myself, but maybe someone else might.

I have written a prototype for an alternative approach to generic
ordered containers.

http://www.geocities.com/wkaras/gen_cpp/avl_tree.html

For someone who says: "I have some instances of a type that I want
to keep in order; I don't mind if the instances are copied in and
out of the heap as the means of storing them in order", STL map/set
are what they're looking for.

For someone who says, "I have some 'things' that I want to keep in
order; the 'things' are unique identified by 'handles'; I am willing
to 'palletize' my 'things' so that each one can store the link 'handles'
necessary to form the container; I don't (necessarily) want to copy
the 'things', and I don't want the container to rely on the heap; I'm
willing to provide more 'glue logic' than what's needed when using
map or set", this alternative type of ordered container is what
they're looking for.

This approach has similarities with the approach that relies on
the items to be collected having a base class with the links
needed to form the container. But it is significantly more
flexible.

On the other hand, given the existence of the WWW, I'm not sure
it's worth the effort to add more templates to the standard lib.
when they can easily be implemented in a portable way. It seems like
the standard lib. is becoming like the Academy Awards for good
code, rather than a way of making it easier to write portable
code.

 

[Jerry Coffin]

"Andrei Alexandrescu \(See Website for Email\)"

[ ... ]

I'd say, a good feature, like a good business idea, can be explained in a
few words. What is the few-words good explanation of export? (I *am*
interested.)

From what I've heard, the good explanation is that it prevented a

civil war in the C++ committee, so without it there might not be a C++
standard at all (or at least it might have been delayed considerably).

As to why people thought they wanted it: so it would be possible to
distribute template code as object files in libraries, much like
non-template code is often distributed. I don't believe that export,
as currently defined, actually supports this though.

Does it do what it was intended to do? (Again, I *am* interested.)

It's hard to say without certainty of the intent. Assuming my guess
above at the intent was correct, then I'm quite certain it does NOT do
what's intended.

If, OTOH, the idea is that template code is still distributed as
source code, and export merely allows that source code to be compiled
by itself, then it does what was intended, within a limited scope.

OTOH, if that was desired to speed up compilation, then I think it
basically fails -- at least with Comeau C++, compiling the templates
separately doesn't seem to gain much, at least for me (and since I
have no other compilers that support, or even plan to soon support
export, Comeau is about the only one that currently matters).

[ ... ]

2. export failed horribly at doing what was initially supposed to do. I
believe what it was supposed to do was true (not "when"s and "cough"s and
"um"s) separate compilation of templates. Admittedly, gaffes in other areas
of language design are at fault for that failure. Correct me if I'm wrong.

I don't know how much is true gaffes, and how much the simple fact
that templates are enough different from normal code that what was
expected was simply (at least extremely close to) impossible.

3. export failed miserably at being reasonably easy to implement.

I can hardly imagine how anybody could argue that one.

 

[Thorsten Ottosen]

| > Currently is also seriously tedious to implement class
| > template member functions outside the class.
|
| Huh? Why? Just because you have to repeat the template
| header part?

yes.

| That's not that much more onerous than
| repeating ClassName:: in front of ordinary methods.

many templates have several parameters; then they might have templated member functions.
This get *very* tedious to define outside a class.

br

Thorsten

 

[Daveed Vandevoorde]

"Andrei Alexandrescu wrote:
[...]

Maybe "export" which is so broken and so useless and so abusive that its
implementers have developed Stockholm syndrome during the long years
that
took them to implement it?

How is "export" useless and broken?

Have you used it for any project? I find it very pleasant
to work with in practice.

Haven't used export, and not because I didn't wanna.

What has prevented you from at least trying it? An affordable
implementation has been available for well over a year.

Without doing so, I fail to see how you can objectively make the
assertions you made.

[Whom do you think I referred to when mentioning the Stockholm syndrome?
)]

Adding a smiley to an innapropriate remark does not make it
any more appropriate.

I'd say, a good feature, like a good business idea, can be explained in a
few words. What is the few-words good explanation of export? (I *am*
interested.)

It allows you to separate a function, member function, or static data
member implementation ("definition") in a single translation unit.

In addition, a good programming language feature does what it was intended
to do (plus some other neat things )), and can be reasonably implemented.

To clarify: That is "good" in your personal point of view.

The intent of the feature was to protect template definitions from
"name leakage" (I think that's the term that was used at the time;
it refers to picking up unwanted declaration due to excessive
#inclusion). export certainly fulfills that.

export also allows code to be compiled faster. (I'm seeing gains
without even using an export-aware back end.)

export also allows the distribution of templates in compiled form
(as opposed to source form).

I think we all agree "export" fell the last test.

export was hard to implement for us, no doubt.

Does it do what it was intended to do? (Again, I *am* interested.)

Yes, and more. See above.

A summary of what's the deal with export would be of great help to at least
myself, so I'd be indebted to anyone who'd give me one. For full disclosure,
my current perception is:

1. It's hard to give a good account of what export does in a few words, at
least an account that's impressive.

Impressive is in the eye of the beholder. Whenever I use the feature,
I'm impressed that it works so smoothly.

2. export failed horribly at doing what was initially supposed to do. I
believe what it was supposed to do was true (not "when"s and "cough"s and
"um"s) separate compilation of templates. Admittedly, gaffes in other areas
of language design are at fault for that failure. Correct me if I'm wrong.

How does it fail at separate compilation?

3. export failed miserably at being reasonably easy to implement.

While it is true that it was hard to implement for EDG (I am not aware of
anyone else having even tried), it was never claimed by the proponents
that it would be easy to implement.

After EDG implemented export, Stroustrup once asked what change to
C++ might simplify its implementation without giving up on the separate
compilation aspect of it. I couldn't come up with anything other than the
very drastic notion of making the language 100% modular (i.e., every entity
can be declared in but one place). That doesn't mean that a template
separation model is not desirable.

Combined with 1 and 2, I can only say: at the very best, export is a Pyrrhic
victory.

The history C++ "export" feature may well be the very incarnation of irony.
However, I don't think there is a matter of "victory" here.

I contend that, all other things being equal, export templates are more
pleasant to work with than the equivalent inclusion templates. That by
itself is sufficient to cast doubt on your claim that the feature is "broken
and useless."

Daveed