Boost Workshop at OOPSLA 2004

[Gabriel Dos Reis]

| Andrei Alexandrescu (See Website for Email) wrote:
| > What is the few-words good explanation of export?

Do those few-words need to be technical or are they marketing purpose?
This is a genuine question, as I suspect that too much hype and
marketing have been pushed against export. That impression I got was
not cleared up after discussion of a well kown paper.

| Here's my attempt:
| Because of historical reasons having to do with how templates are
| implemented, template methods (and static data members) are
| effectively considered inline, and so their definitions must be
| included in any compilation unit which requires an instantiation.

I think that use of "inline" is unfortunate. I don't think that
description accurately covers what CFront did and other historical
repository-based instantiations (like in old Sun CC).

Export is the result of a compromise. A compromise between tenants of
inclusion model only and tenants of separate compilation of templates.

 

[Walter]

To express it as simply as possible, imagine that C++ required that every
function be declared inline, and that therefore the implementation of every
function must be included in any compilation unit that used it. This is the
model that unexported templates labor under, and is what export is designed
to avoid.

In the D programming language, all functions are potentially inline (at the
compiler's discretion), and so all the function bodies are available, even
though it follows the separate compilation model. All the programmer does is
use the statement:

import foo;

and the entire semantic content of foo.d is available to the compiler,
including whatever template and function bodies are in foo.d. So, is this a
burden the compiler labors under? Not that anyone has noticed, it compiles
code at a far faster rate than a C++ compiler can. I can go into the reasons
why if anyone is interested.

But back to what can be done with C++. Many compilers implement precompiled
headers, which do effectively address this problem reasonably well. Export
was simply not needed to speed up compilation, and for those who don't
believe me, I stand by my challenge to benchmark Digital Mars C++ with
precompiled headers against any export template implementation for project
build speed.

 

[Walter]

If this comes to you from a library provider, they could ship
compiled versions of the implementation file, just as they now
ship object files (assuming the compiler vendor supplied such
support). If it's your own templates, you simply make your
overall project depend on the implementation files if your
vendor uses the freedom given by 14/9 to force you to compile
the implementations first.

I agree it's not a big problem, it just isn't as simple as not having to

How is it different from object files? For normal source files
you must specify which object files are part of your program,
what their source files are, and what other files they depend
on. Some of this is done automatically by various development
platforms, but it is always done. How are exported templates
different?

The cross reference database is what the librarian does <g>. And again, I
agree that this is not a huge problem, but it is a problem and it does, for
example, impose an order on the compilations that was not required before.
But I'll also say that, as a compiler vendor, one of the most common tech
questions I get is "I am getting an undefined symbol message from the
linker, what do I do now?" I imagine this would be worse with template
interdependencies, but I could be wrong.

I will believe this once you agree that C++ should require
definitions of all functions, template or not, to be included
in every compilation unit which calls them. If you do not
agree that this is a good idea for plain functions, I do not
see why it's a good idea for function templates.

The D language does this, and it works fine. I've also heard of C++
compilers that do cross-module optimization that do this as well. Good idea
or not, it is doable with far less effort than export. But I emphasize that
the way the C++ language was designed makes it *easy* to implement separate
compilation for functions. That same design makes it *very hard* to do
separate compilation for templates, no matter how conceptually the same we
might wish them to be. If C++ had the concept of modules (rather than its
focus on source text), this would not be such a big problem. And it is not
enough that an idea be just a good idea, its advantages must outweigh the
costs. The costs of export are enormous, and the corresponding enormous gain
just isn't there.

No. By requiring that implementations be bodily included
where instantiations are needed, the implementations are,
first, subject to the various pollutions of the instantiation
space, not least of which are macros, and secondly, are not
able to use anonymous namespaces since that will break the
ODR.

Why not use a named namespace?

You fail to see that the second case exposes the implementation
to the vagaries of the instantiation environment while the first
does not. Think of macros if nothing else.

I'll agree on the macro issue, but nothing else <g>. And to repeat what I
said earlier about the macro issue, export seems to be an awfully expensive
solution to the macro scoping problem, especially since the macro pollution
issue still remains. Wouldn't it be better to come at the macro problem
head on?

 

[Andrei Alexandrescu \(See Website for Email\)]

Here's my attempt:
Because of historical reasons having to do with how templates are
implemented, template methods (and static data members) are
effectively considered inline, and so their definitions must be
included in any compilation unit which requires an instantiation.

The export keyword breaks this requirement. When a template method
is marked as export, its definition does not get included into
compilation units which use it. Instead, it goes into its own source
file(s), and is compiled separately.

The C++ standard permits an implementation to require that the
definition of an exported method or object must be compiled before a
reference to such an object is compiled.
Thanks.


Compilation of templates is obviously not like compilation of ordinary
source, since template instantiation requires information from the
instantiation context and from the template parameters. But what do you
mean by "true" separate compilation? What kind of separate compilation
is not true?

By "true" separate compilation I understand the dependency gains that
separate compilation achieves. That is crucial, and whole projects can be
organized around that idea. It translates into what needs be compiled when
something is touched, with the expected build speed and stability tradeoffs.

Templates cannot be meaningfully typechecked during their compilation. They
also cause complications when instantiated in different contexts. That makes
them unsuitable for "true" separate compilation. Slapping a keyword that
makes them appear as separately compilable while the true inside the
compilation system guts is different (in terms of dependency management and
compilation speed) didn't help.

I understand that some people wish that export should be a
way to prevent people from examining template implementation code, but
that's hardly something for the standard to worry about.

I consider that a secondary issue.

To express it as simply as possible, imagine that C++ required that every
function be declared inline, and that therefore the implementation of
every
function must be included in any compilation unit that used it. This is
the
model that unexported templates labor under, and is what export is
designed
to avoid.

I don't think that that's a good parallel. Because the non-inline functions
can be typechecked and compiled to executable code in separation. Templates
cannot.


Andrei

 

[Andrei Alexandrescu \(See Website for Email\)]

I don't know how much is true gaffes, and how much the simple fact
that templates are enough different from normal code that what was
expected was simply (at least extremely close to) impossible.

Fundamentally templates are sensitive to the point where they are
instantiated, and not on the parameters alone. That's a classic PL design
mistake because it undermines modularity. The effects of such a mistake are
easily visible ). They've been visible in a couple of early languages as
well.

Andrei

 

[Hyman Rosen]

That same design makes it *very hard* to do separate compilation for templates,
no matter how conceptually the same we might wish them to be.

Just because it's hard for compiler makers doesn't mean it's hard for
compiler users. Indeed, I find the inclusion model for templates to
be difficult and annoying and find export to be perfectly intuitive.

its advantages must outweigh the costs. The costs of export are enormous, and the
corresponding enormous gain just isn't there.

Export imposes no cost at all on programmers. It's a simple concept and simple to use.
Compiler vendors for the most part can't be bothered to implement it, because everyone
uses the inclusion model because they have to.

Why not use a named namespace?

Because then the inclusion model would violate the ODR, unless I prepare *another*
source file with the implementation of those gloabl functions that my template
methods want to call. Why should I jump through hoops when the standard give me a
perfectly simple way not to?

 

[Gabriel Dos Reis]

| | > "Andrei Alexandrescu wrote:
| > [...]
| > > Maybe "export" which is so broken and so useless and so abusive that its
| > > implementers have developed Stockholm syndrome during the long years
| > > that
| > > took them to implement it?
| >
| > How is "export" useless and broken?
| >
| > Have you used it for any project? I find it very pleasant
| > to work with in practice.
|
| Haven't used export, and not because I didn't wanna.

Very interesting.

 

[Andrei Alexandrescu \(See Website for Email\)]

"Andrei Alexandrescu wrote:
Without doing so, I fail to see how you can objectively make the
assertions you made.

[Whom do you think I referred to when mentioning the Stockholm syndrome?
)]

Adding a smiley to an innapropriate remark does not make it
any more appropriate.

Sorry you found it inappropriate. I thought of emailing a private excuse,
but excuses must be made in the front of those who witnessed the offense.
So - I am sorry; I had found the comparison funny and meant it without harm.

I think what I need to do is go use the feature, and then make noise when
I'm more based...


Andrei

 

[Hyman Rosen]

I don't believe that export, as currently defined, actually supports this though.

Why don't you believe it? What is it about export that you think would prevent
a compiler vendor from doing just that? Obviously compiled templates act as
further input to the "compiler" rather than to the "linker" (assuming those
distinctions exist), but what of that?

I can hardly imagine how anybody could argue that one.

Who said it was supposed to be easy? Templates aren't easy, exceptions
aren't easy, manipulating vtable pointers during construction isn't easy.

 

[Hyman Rosen]

many templates have several parameters; then they might have templated member functions.
This get *very* tedious to define outside a class.

Like I said, just use a macro. Combine this with export,
so that the macro is hidden in the implementation file.

 

[Hyman Rosen]

I think that use of "inline" is unfortunate. I don't think that
description accurately covers what CFront did and other historical
repository-based instantiations (like in old Sun CC).

That's not relevant to the language defined by the standard.
In standard C++, either a template method is marked by export,
or it must be included in every compilation unit that would
instantiate it. That is exactly the model of inline - an inline
method must be defined in every compilation unit that calls it.

Export is the result of a compromise. A compromise between tenants of
inclusion model only and tenants of separate compilation of templates.

Every aspect of the language has interesting historical background
that is largely irrelevant to undertanding how to use it. I think
export just got a bum rap because no one bothered to implement it
because you could get along without it and there were other fish to
fry.

 

[Hyman Rosen]

Export was simply not needed to speed up compilation

That's fine, because that's not what it's for.

 

[Andrei Alexandrescu \(See Website for Email\)]

"Andrei Alexandrescu \(See Website for Email\)"

| | > "Andrei Alexandrescu wrote:
| > [...]
| > > Maybe "export" which is so broken and so useless and so abusive that
its
| > > implementers have developed Stockholm syndrome during the long years
| > > that
| > > took them to implement it?
| >
| > How is "export" useless and broken?
| >
| > Have you used it for any project? I find it very pleasant
| > to work with in practice.
|
| Haven't used export, and not because I didn't wanna.

Very interesting.

Well, it's very banal really. I just only had the chance to use compilers
that don't implement export.

Andrei

 

[Gabriel Dos Reis]

| Gabriel Dos Reis wrote:
| > I think that use of "inline" is unfortunate. I don't think that
| > description accurately covers what CFront did and other historical
| > repository-based instantiations (like in old Sun CC).
|
| That's not relevant to the language defined by the standard.

Here is a piece that disappeared in your reply and that makes the
above statement the most relevant as a reply to your earlier message:

Because of historical reasons having to do with how templates are
implemented, template methods (and static data members) are
effectively considered inline, and so their definitions must be
included in any compilation unit which requires an instantiation.

If you believe "historical reasons" to be not relevant to the language
defined by the standard, why did you bring them in ther first place?

 

[Hyman Rosen]

Templates cannot be meaningfully typechecked during their compilation. They
also cause complications when instantiated in different contexts. That makes
them unsuitable for "true" separate compilation. Slapping a keyword that
makes them appear as separately compilable while the true inside the
compilation system guts is different (in terms of dependency management and
compilation speed) didn't help.

There have been environments in which the linker did whole-program
optimization, inlining routines out of object files into the call
sites. I think you are mistaken in concept when you try to peer under
the hood of the compiler to call some of its operations "true" and
some not. Step back and think of nothing but the point of view of
the user. Also consider that while extremely complex cases of
instantiation must be handled correctly by the compiler, most actual
templates that people write don't have weird name-capturing problems.

The export user sees method implementations compiled separately in an
implementation source file, and declarations included by header in
compilation units that need the methods. For the user, this appears
to work just like non-template code.

I don't think that that's a good parallel. Because the non-inline functions
can be typechecked and compiled to executable code in separation. Templates
cannot.

You are thinking of this from a compiler writer's perspective.
From a user's perspective, he submits the code to a compiler
and gets an executable out. Without export, the implementation
must be included in the places where it's called, subjecting it
to the vagaries of that environment. That's a pain.

 

[Jerry Coffin]

Why don't you believe it?

Because I see the same things you seem to want to ignore.

What is it about export that you think would prevent
a compiler vendor from doing just that? Obviously compiled templates act as
further input to the "compiler" rather than to the "linker" (assuming those
distinctions exist), but what of that?

In the end, such distinctions DO exist, and they exist for what many
people clearly consider very good reasons -- specifically that they
provide real benefits. I believe the expectation of most people was
that export would provide the same or similar benefits, but I don't
believe they do so.

Worse, your defense of export seems backward to me: you claim to be
recieving the benefits of export even with a backend that doesn't
support it. That sounds to me like the benefits don't really come from
export at all.

Who said it was supposed to be easy? Templates aren't easy, exceptions
aren't easy, manipulating vtable pointers during construction isn't easy.

The wording was "reasonably easy", not just easy.

Reasonable in this sort of usage normally implies one or both of two
things: 1) that it's roughly in line with what was expectd, and 2)
that the benefits are sufficient to justify the cost.

I'm not on the committee myself, but I've certainly conversed with a
number of committee members, and ALL of them I've talked to have
admitted that export has turned out to be substantially more difficult
to implement than was expected.

So far, few people who've really used export seem to have found
tremendous benefits to it either.

By contrast, nobody seems to have been terribly surprised by the
amount of work it has taken to implement exception handling, and most
people seem to believe that its benefits are justify the work.

The area where exception handling may have surprised some people
and/or had unjustified costs is not in its implementation, but the
burden for exception safety that's placed on the user. Though few
people seem to be aware of it (yet), export has a similar effect -- it
affects name lookup in ways most people don't seem to expect, and can
substantially increase the burden on the user.

So, if you'd prefer, I'd rephrase the question: as I recall, it's
claimed that EDG put something like three man-years of labor into
implementing export, and that's not all it takes to implement it
either. So far even if we take ALL the users into account, I've yet to
see an indication that anybody has saved three man-years of labor
because export was there. That seems to indicate that at this point,
export is a net loss.

The next obvious question would be at what point export breaks even.
At this point, the benefits are sufficiently tenuous that I'm not at
all sure anybody can really guess at the time, or even state with any
certainty that there will ever BE such a time.

 

[Walter]

Just because it's hard for compiler makers doesn't mean it's hard for
compiler users. Indeed, I find the inclusion model for templates to
be difficult and annoying and find export to be perfectly intuitive.

I find:
import foo;
to be far more natural and intuitive than:
#include "foo.h"
(and I don't even have to write a .h file), but the inclusion model is what
C++ is, and export doesn't fix that. If C++ had gone a step further than
export, and offered true modules, I suspect it would have been a lot easier
to implement.

Export imposes no cost at all on programmers. It's a simple concept and

simple to use.

Oh, the costs imposed on programmers are very high. The cost is in terms of
years of delay, deferring of the arrival of other improvements, shrinking
the number of independent C++ implementations, etc.

Compiler vendors for the most part can't be bothered to implement it, because everyone
uses the inclusion model because they have to.

The reason why is it takes 2 to 3 man years to implement. This wouldn't be
an issue at all if it were just a matter of bothering to implement it.

Because then the inclusion model would violate the ODR, unless I prepare *another*
source file with the implementation of those gloabl functions that my template
methods want to call.

I must apologize, because I'm just not getting why namespaces won't work or
what this has to do with the ODR.

Why should I jump through hoops when the standard give me a perfectly

simple way not to?

If export is the only improvement you want from existing C++ compilers,
that's fine. But I find it hard to believe there aren't other things you'd
be interested in having, like better templates, fewer bugs, better
optimization, better libraries, a lower price, better support, nicer IDE,
better debuggers, etc. What are you willing to give up to get export, and
what have you given up for it?

 

[Walter]

That's fine, because that's not what it's for.

I apologize, I assumed that's what you meant when you talked about
"laboring" under the inclusion model. It's repeatedly listed as one of the 3
justifications for exports, and not just by myself. Daveed Vandevoorde
listed them in his post in this thread:

---------------------------------------------------------------------
The intent of the feature was to protect template definitions from
"name leakage" (I think that's the term that was used at the time;
it refers to picking up unwanted declaration due to excessive
#inclusion). export certainly fulfills that.

export also allows code to be compiled faster. (I'm seeing gains
without even using an export-aware back end.)

export also allows the distribution of templates in compiled form
(as opposed to source form).

 

[Walter]

Who said it was supposed to be easy? Templates aren't easy, exceptions
aren't easy, manipulating vtable pointers during construction isn't easy.

2 to 3 man years to implement is quite another level of "easy". It's
extremely costly to implement, costly to users as well as vendors.

 

[tom_usenet]

export also allows the distribution of templates in compiled form
(as opposed to source form).

This I would love to hear about. What do compiled templates look like?
I hope commercial pressures don't prevent you from replying.

Are compiled templates easily decompiled (assuming the file format is
not obscure)? How much source information can be thrown away in
compiling them?

After EDG implemented export, Stroustrup once asked what change to
C++ might simplify its implementation without giving up on the separate
compilation aspect of it. I couldn't come up with anything other than the
very drastic notion of making the language 100% modular (i.e., every entity
can be declared in but one place).

That is exactly the same conclusion that I have reached (intuitively
rather than through experience or careful working through of the
problem); separate compilation of templates within the usual C++ TU
model pretty much leads you to two phase name lookup and export.

Tom