std::operator<<(basic_ostream&, type) vs. std::basic_ostream::operator(type)... ?

[Martin T.]

Hello.

I tried to overload the operator<< for implicit printing of wchar_t
string on a char stream.
Normally using it on a ostream will succeed as

>(std::basic_ostream<char,std::char_traits<char> > & _Ostr={...}, ___)

will be called.

However, I am using the boost::format library that internally makes use
of a string stream to store the formatted string. There:
std::basic_ostream<char,std::char_traits<char> >:perator<<(___)
will always be called.

On what basis will the compiler choose which kind of operator<< will be
called??

(see test code below)

thanks.
br,
Martin

test code:
###############
#include <iostream>
#include <fstream>
#include <sstream>
using namespace std;

// Output wchar_t* strings on char streams instead of formatting as
void* ...
ostream& operator<<(ostream& os, const wchar_t* c_ptr) {
return os << "(TYPE:wchar_t*)"; // placeholder
}

// Boost.Format library
#include <boost/format.hpp>

////////////////////////////////////////
int main(int argc, char* argv[])
{
ostringstream ostr;
ostream& o = ostr;

o << "\nTest of ostream << operator ..." << endl;
wchar_t* wstr = L"Hello world!";
o << "Output of wchar_t* -- " << wstr << endl;
// CALL STACK ...

>(std::basic_ostream<char,std::char_traits<char> > & _Ostr={...}, const

char * _Val=0x0042f860) Line 747 C++
// (1)
opov.exe!operator<<(std::basic_ostream<char,std::char_traits<char> > &
os={...}, const wchar_t * c_ptr=0x0042f8f0) Line 6 + 0xe bytes C++
// (0) opov.exe!main(int argc=1, char * * argv=0x00356828) Line 17 +
0x26 bytes C++

o << "\nTest of boost-library ostream << operator ..." << endl;
o << "Boost formatted wchar_t* -- " << boost::str(boost::format("%1%")
% wstr) << endl;
// CALL STACK ...

>:perator<<(const void * _Val=0x00433a54) Line 456 C++

// (5)
opov.exe!boost::io::detail:ut_last<char,std::char_traits<char>,wchar_t
*>(std::basic_ostream<char,std::char_traits<char> > & os={...}, wchar_t
* const & x=0x00433a54) Line 98 + 0x11 bytes C++
// (4)
opov.exe!boost::io::detail:ut<char,std::char_traits<char>,std::allocator<char>,wchar_t
* const &>(wchar_t * const & x=0x00433a54, const

> & specs={...},

std::basic_string<char,std::char_traits<char>,std::allocator<char> > &
res="",

> & buf={...}, std::locale * loc_p=0x00000000) Line 150 + 0x2c bytes C++

// (3)
opov.exe!boost::io::detail::distribute<char,std::char_traits<char>,std::allocator<char>,wchar_t
* const

> & self={...}, wchar_t * const & x=0x00433a54) Line 242 + 0x3f bytes C++

// (2)
opov.exe!boost::io::detail::feed<char,std::char_traits<char>,std::allocator<char>,wchar_t
* const

> & self={...}, wchar_t * const & x=0x00433a54) Line 251 + 0xd bytes C++

// (1)

>:perator%<wchar_t *>(wchar_t * const & x=0x00433a54) Line 64 +

0xd bytes C++
// (0) opov.exe!main(int argc=1, char * * argv=0x00356828) Line 53 C++

// As can be seen on (5) the parameter (now called x) is of type wchar_t*
// However, the basic_ostream operator<<(void*) will be called.

cout << ostr.str();
}
##################

 

[James Kanze]

I tried to overload the operator<< for implicit printing of wchar_t
string on a char stream.
Normally using it on a ostream will succeed as

will be called.

However, I am using the boost::format library that internally makes use
of a string stream to store the formatted string. There:
std::basic_ostream<char,std::char_traits<char> >:perator<<(___)
will always be called.

On what basis will the compiler choose which kind of
operator<< will be called??

Overload resolution on the set of visible functions.

You have to be very careful here with user defined << operators
on basic types. When << is invoked for a standard type from
within a template, only std:: will be searched at the point of
instantiation, so if the << wasn't defined at the point of
definition, it won't be found.

(see test code below)

test code:
###############
#include <iostream>
#include <fstream>
#include <sstream>
using namespace std;

// Output wchar_t* strings on char streams instead of formatting as
void* ...
ostream& operator<<(ostream& os, const wchar_t* c_ptr) {
return os << "(TYPE:wchar_t*)"; // placeholder
}

// Boost.Format library
#include <boost/format.hpp>

Except that here, you've declared (and defined, but that doesn't
matter) the operator before including the header. So even in a
template, the compiler should find your declaration, and use it.
(Unless, of course, boost::format was exported---but I get the
same results as you with g++, which doesn't support export.)

If I put the << operator in namespace std (illegal, and
undefined behavior, but you can often get away with it), the
code works (even if the declaration is moved after the include
of boost/format.hpp), but my understanding of the standard is
that this should not be necessary if the declaration precedes
the template definition (which, of course, is in the file
boost/format.hpp).

It seems weird to me, too.

 

[Martin T.]

Overload resolution on the set of visible functions.

You have to be very careful here with user defined << operators
on basic types. When << is invoked for a standard type from
within a template, only std:: will be searched at the point of
instantiation, so if the << wasn't defined at the point of
definition, it won't be found.

Ah ok. This kindof makes sense. Thanks for pointing it out explicitly.

Except that here, you've declared (and defined, but that doesn't
matter) the operator before including the header. So even in a
template, the compiler should find your declaration, and use it.
(Unless, of course, boost::format was exported---but I get the
same results as you with g++, which doesn't support export.)

If I put the << operator in namespace std (illegal, and
undefined behavior, but you can often get away with it), the
code works (even if the declaration is moved after the include
of boost/format.hpp), but my understanding of the standard is
that this should not be necessary if the declaration precedes
the template definition (which, of course, is in the file
boost/format.hpp).

It seems weird to me, too.

Thanks a lot for this explanation!
I sadly had to give up further experiments for now, as my work time is
only limited atm. :/

(The reason I wanted to do this was some functions that return wchar_t*
but my logging is done in char* and in these cases I will get the
wchar_t* formatted as void*, which is obv. not what I want, but the
compiler will not find the code parts where the explicit conversion
wchar_t->char has been forgotten. (as there exists a conversion from
wchar_t* to void*) ... grml ...)


Maybe someone else can clear-up this "weirdness" as it apparently seems
to happen on g++ (you) and on MSVC8 (VS2005) (me) ...

br,
Martin

 

[James Kanze]

Maybe someone else can clear-up this "weirdness" as it apparently seems
to happen on g++ (you) and on MSVC8 (VS2005) (me) ...

Yes. I'm curious myself, because it does seem to be
intentional. Maybe there's something particular in
boost::format which affects name lookup as well; I didn't have
time to search it in detail to see, but if so, I can't off hand
imagine what it could be.

 

[Martin T]

Yes. I'm curious myself, because it does seem to be
intentional. Maybe there's something particular in
boost::format which affects name lookup as well; I didn't have
time to search it in detail to see, but if so, I can't off hand
imagine what it could be.

I have played around with the problem some more.
I certainly do not know WHY it behaves that way, but I have found a
workaround that does not require me to put the operator<< into the std
namespace. It appears it is sufficient when the operator is defined in
the boost::io::details namespace, where the put_last(..) function will
be invoked.
I have to admit that I am not really someone who gets the namespace
lookup rules, but in this case it appears that the operator<< has to be
defined in _exactly_ the namespace where it will be used.
If used from the :: global namespace it has to be defined there.
If used in ::boost::io::details it has to be defined there.
If used in ::some_other_ns it has to be defined there. (using namespace
<namespace>; will NOT work it appears.

Or, put differently: If it is defined in exactly the namespace where it
is used it will be found, Else the compiler will first look in the std
namespace (or maybe, first look at the std::basic_ostream class which
defines a matching operator) before looking in the global namespace.

does that make sense?

br,
Martin


-- updated test code --
----
#pragma warning( disable : 4996)

#include <iostream>

// Include boost format header:
#include <boost/format.hpp>

#ifndef MY_OWN_BOOST_IO_STD
# define MY_OWN_BOOST_IO_STD ::std::
#endif

// Class in separate namespace outputting a template Type to
std::basic_ostream:
namespace boost {
// namespace io {
// namespace detail {
// (exact copy from boost/feed_args.hpp)
// it will NOT work if we put it into the boost namespace
template< class Ch, class Tr, class T> inline
void my_own_boost_put_last( MY_OWN_BOOST_IO_STD basic_ostream<Ch, Tr>
& os, const T& x ) {
os << x ;
}
// };
// };
};

namespace my_own_namespace {
namespace io {
namespace detail {
// (exact copy from boost/feed_args.hpp)
// it will work if we do not put it into the boost namespace
template< class Ch, class Tr, class T> inline
void my_own_put_last( MY_OWN_BOOST_IO_STD basic_ostream<Ch, Tr> & os,
const T& x ) {
os << x ;
}
};
};
};

namespace my_other_namespace {
void func() {
using namespace std;
using namespace boost;
cout << "in other namespace: " << L"wchar-string" << endl;
}
};


// (global namespace) user defined ostream operator<< for wchar_t*
template< class Ch, class Tr>
inline std::basic_ostream<Ch, Tr>& operator<<(std::basic_ostream<Ch,
Tr>& os, const wchar_t*const str)
{
// printf("\n--\nThe type of the stream is: %s\n--\n", typeid(os).name());
os << "[wchar_t*]";
return os;
}

#define DEFINE_OS_OUTP_OPERATOR_AND_USE_GLOBAL();
\
template< class Ch, class Tr>
\
inline std::basic_ostream<Ch, Tr>& operator<<(std::basic_ostream<Ch,
Tr>& os, const wchar_t*const str) \
{
\
return :perator<<(os, str);
\
}

namespace boost {
DEFINE_OS_OUTP_OPERATOR_AND_USE_GLOBAL();
};

namespace boost {
namespace io {
namespace detail {
DEFINE_OS_OUTP_OPERATOR_AND_USE_GLOBAL();
};
};
};


// Main execution:
int main(int argc, char* argv[])
{
using namespace std;

cout << " Global: ";
cout << L"String 1"; // Placeholder text ... OK
cout << endl;

cout << " boost: ";
boost::my_own_boost_put_last(cout, L"String 2"); // Output as void* ... ??
cout << endl;

cout << " my_namespace: ";
my_own_namespace::io::detail::my_own_put_last(cout, L"String 3"); //
Placeholder text ... OK
cout << endl;

boost::io::detail:ut_last(cout, L"String 4"); // Output as void* ... ??
cout << endl;

my_other_namespace::func();

}
----

 

[Martin T]

Yes. I'm curious myself, because it does seem to be
intentional. Maybe there's something particular in
boost::format which affects name lookup as well; I didn't have
time to search it in detail to see, but if so, I can't off hand
imagine what it could be.

I have played around with the problem some more.
I certainly do not know WHY it behaves that way, but I have found a
workaround that does not require me to put the operator<< into the std
namespace. It appears it is sufficient when the operator is defined in
the boost::io::details namespace, where the put_last(..) function will
be invoked.
I have to admit that I am not really someone who gets the namespace
lookup rules, but in this case it appears that the operator<< has to be
defined in _exactly_ the namespace where it will be used.
If used from the :: global namespace it has to be defined there.
If used in ::boost::io::details it has to be defined there.
If used in ::some_other_ns it has to be defined there. (using namespace
<namespace>; will NOT work it appears.

Or, put differently: If it is defined in exactly the namespace where it
is used it will be found, Else the compiler will first look in the std
namespace (or maybe, first look at the std::basic_ostream class which
defines a matching operator) before looking in the global namespace.

does that make sense?

br,
Martin


-- updated test code --
----
#pragma warning( disable : 4996)

#include <iostream>

// Include boost format header:
#include <boost/format.hpp>

#ifndef MY_OWN_BOOST_IO_STD
# define MY_OWN_BOOST_IO_STD ::std::
#endif

// Class in separate namespace outputting a template Type to
std::basic_ostream:
namespace boost {
// namespace io {
// namespace detail {
// (exact copy from boost/feed_args.hpp)
// it will NOT work if we put it into the boost namespace
template< class Ch, class Tr, class T> inline
void my_own_boost_put_last( MY_OWN_BOOST_IO_STD basic_ostream<Ch, Tr>
& os, const T& x ) {
os << x ;
}
// };
// };
};

namespace my_own_namespace {
namespace io {
namespace detail {
// (exact copy from boost/feed_args.hpp)
// it will work if we do not put it into the boost namespace
template< class Ch, class Tr, class T> inline
void my_own_put_last( MY_OWN_BOOST_IO_STD basic_ostream<Ch, Tr> & os,
const T& x ) {
os << x ;
}
};
};
};

namespace my_other_namespace {
void func() {
using namespace std;
using namespace boost;
cout << "in other namespace: " << L"wchar-string" << endl;
}
};


// (global namespace) user defined ostream operator<< for wchar_t*
template< class Ch, class Tr>
inline std::basic_ostream<Ch, Tr>& operator<<(std::basic_ostream<Ch,
Tr>& os, const wchar_t*const str)
{
// printf("\n--\nThe type of the stream is: %s\n--\n", typeid(os).name());
os << "[wchar_t*]";
return os;
}

#define DEFINE_OS_OUTP_OPERATOR_AND_USE_GLOBAL();
\
template< class Ch, class Tr>
\
inline std::basic_ostream<Ch, Tr>& operator<<(std::basic_ostream<Ch,
Tr>& os, const wchar_t*const str) \
{
\
return :perator<<(os, str);
\
}

namespace boost {
DEFINE_OS_OUTP_OPERATOR_AND_USE_GLOBAL();
};

namespace boost {
namespace io {
namespace detail {
DEFINE_OS_OUTP_OPERATOR_AND_USE_GLOBAL();
};
};
};


// Main execution:
int main(int argc, char* argv[])
{
using namespace std;

cout << " Global: ";
cout << L"String 1"; // Placeholder text ... OK
cout << endl;

cout << " boost: ";
boost::my_own_boost_put_last(cout, L"String 2"); // Output as void* ... ??
cout << endl;

cout << " my_namespace: ";
my_own_namespace::io::detail::my_own_put_last(cout, L"String 3"); //
Placeholder text ... OK
cout << endl;

boost::io::detail:ut_last(cout, L"String 4"); // Output as void* ... ??
cout << endl;

my_other_namespace::func();

}
----

 

[James Kanze]

I have played around with the problem some more.
I certainly do not know WHY it behaves that way, but I have
found a workaround that does not require me to put the
operator<< into the std namespace. It appears it is sufficient
when the operator is defined in the boost::io::details
namespace, where the put_last(..) function will be invoked.

In otherwords, a namespace related to its use.

I think I've figured out the problem. The compiler does do two
name lookups: one "normal" (using both unqualified name lookup
and argument-dependent name lookup at the point of definition,
and one using just argument-dependent name lookup at the point
of instantiation). The problem is that the point of definition
is in boost (or a namespace nested in namespace boost), and
boost::format defines a free function overload of the operator<<
there. And since in unqualified lookup, "name lookup ends as
soon as a declaration is found for the name", this lookup
doesn't find any other operator<<; neither yours nor the ones in
std. (The argument-dependent lookup, of course, finds the ones
in std.)

Note that the problem would be the same if you defined the
operator in one of the boost namespaces, and tried to use it in
a template function (e.g. with std:stream_iterator) from the
standard library.

And the reason the operator works outside of boost::format is
that you are invoking it from the global namespace, and outside
of a template, name lookup uses both unqualified name lookup at
the point the operator is invoked and argument-dependent name
lookup; the unqualified name lookup starts from the namespace
you are in, and works out, stopping in the first namespace it
finds the operator in. In the case of operators, it's a little
bit more complicated, since the compiler will in fact try two
different alternatives, one for free functions, as described
here, and a second for the case where the operator might be a
member, using class member access lookup. This is why you
always get the member operator<< of std:stream---including <<
of void*---in the overload set.

It's too late to consider the possibility now (and the
possibility didn't exist when iostream was being invented), but
the best solution would probably have been to make the
operator<< a template member function of std:stream, with
explicit specializations (and of course, you can specialize a
function template member of std, provided that the
specialization includes a user defined type of your own,
somewhere). That would ensure that anytime the compiler found a
<< for ostream, it found all of the ones you wanted it to,
regardless (because of member lookup), and that you wouldn't
have the problem of name lookup giving different results
depending on the namespace you happened to be in.

But of course, the current situation only creates a problem when
you try to define a << operator for a type which has no
dependencies on a user defined type. Which you're normally not
supposed to do.

I have to admit that I am not really someone who gets the
namespace lookup rules, but in this case it appears that the
operator<< has to be defined in _exactly_ the namespace where
it will be used.

The normal rule is to define the << function in the same
namespace as that in which the user defined type is defined.
That way, argument-dependent lookup picks it up, always. The
problem here is that wchar_t is a built-in type, and built-in
types are not defined in any namespace (not even global), and
don't cause any additional namespaces to be considered. So you
have to define the operator in a namespace which will be
considered by argument-dependent lookup. And the only one you
can really count on is std. (The presence of std:stream, or a
class derived from std:stream, as the first operand guarantees
that argument-dependent lookup will search std.)

In your case, quite frankly, I think I'd just go ahead and
define the operator in std::. The standard says undefined
behavior, but in practice, it will either work, or you'll get a
compiler error about double definitions. (To be sure of getting
the double definition error, make your operator a function
template as well, with exactly the same template arguments, used
in exactly the same way, as the standard operator<< for char.)
I'd throw in some comments about why you're doing this (i.e. why
you need to overload << for wchar_t in the first place, and a
mention of the name lookup problems which caused you to put it
into std:, and be done with it.

If used from the :: global namespace it has to be defined
there. If used in ::boost::io::details it has to be defined
there. If used in ::some_other_ns it has to be defined there.
(using namespace <namespace>; will NOT work it appears.

Nope. A using directive only makes names that are not already
declared in the namespace visible. (Formally, it makes the
names in the class visible "as if" they were declared in the
"nearest enclosing namespace which contains both the
using-directive and the nominated namespace". Presumably an
imaginary enclosing namespace if the using directive is in the
global namespace.) A using declaration would work, but it must
be present in every namespace in which you want the << operator
to be found: in the boost namespace, in std, etc.

Or, put differently: If it is defined in exactly the namespace
where it is used it will be found, else the compiler will
first look in the std namespace (or maybe, first look at the
std::basic_ostream class which defines a matching operator)
before looking in the global namespace.

does that make sense?

Except that it's not quite right. For an operator like <<:
first the compiler considers two different rewrites of a << b:
a.operator<<( b ) and operator<<( a, b ), doing name lookup for
both, and adding everything the name lookup finds to the
overload set. In the first case, it does class member lookup,
starting in the class of a, and working down through the base
classes until it finds a declaration. (Obviously, if a is not
of class type, this step is skipped, since only class types can
have member functions.) In the second case, the exact steps
depend on whether the function is being invoked from a template
instantiation or not. If not, both unqualified name lookup and
argument-dependent name lookup are done from the point where the
function in invoke (i.e. where the operator was used).
Unqualified name lookup starts in the namespace in which the
code is found, and works outwards, stopping (like class member
lookup) as soon as it finds the name. Argument-dependent lookup
creates a set of associated namespaces for each argument: the
namespace in which the argument type is defined, of course, but
also any namespaces in which base classes of the argument type
is defined, and in the case of templates, both the namespaces
associated with the template itself, and those associated with
type parameters of the template. Any declarations in any of
these namespaces are added to the overload set. If the operator
is invoked in a function template, then this name lookup occurs
at the point of definition of the template---if the expression
is dependent, an additional argument-dependent name lookup
occurs at the point of instantiation.

That is, of course, a much simplified explination. And the
rules for overload resolution are even more complicated. On the
other hand, if you follow standard operating procedures, and
just do things cleanly the way they're supposed to be done (e.g.
defining the << operator for a class in the same namespace where
you defined the class), the results are pretty much what you'd
want and would naïvely expect. The problems really only occur
when for some reason, you need to do something you're not
supposed to do (like overloading an << operator for a built-in
type). (Or when you have to write a compiler, and have to
implement all this without any errors.)

 

[Martin T.]

>
(snipped)
>
That is, of course, a much simplified explination. And the
rules for overload resolution are even more complicated. On the
other hand, if you follow standard operating procedures, and
just do things cleanly the way they're supposed to be done (e.g.
defining the << operator for a class in the same namespace where
you defined the class), the results are pretty much what you'd
want and would naïvely expect. The problems really only occur
when for some reason, you need to do something you're not
supposed to do (like overloading an << operator for a built-in
type). (Or when you have to write a compiler, and have to
implement all this without any errors.)

Thank you very much for this explanation and your tips!
Just goes to show again that I know more of C++ than I like and less
than I need ;-)

best regards,
Martin