What is the flaw in this benchmark, other than not unrolling loops
(which is probably a big flaw)?
I haven't looked to closely at the source code but eg. in "sieve.cpp" the
program is not correct: it causes undefined behavior because an iterator
for a vector is placed beyond the end of the vector (further than the
past the end iterator). No big deal since it will normally work on
typical platforms but I'd think the programs should be correct.
But I think the results are still startling, considering that
Java is interpreted.
It is a misconception that Java is interpreted: it is not. Java is a
compiled language. The major difference to C++ (if you leave out
Microsoft's C++ for .Net) is that it uses some form of platform
independent assembler which is transformed into a platform's native
assembler at run-time (when it is loaded and possible later again when
hotspot optimization is done). The major differences between Java and
C++ are in their respective object models, not that one is compiled and
the other is not. And most of the benchmarks stay clear of the object
model differences operating on the low-level built-in types like 'int'
where the object models are identical. One of the examples where there
is a difference in the object model (methcall.*) the test forces the
Java model on the C++ code while making sure that a Java specific
optimization can and will kick in.
Maybe the fault is with g++'s optimization.
The gcc optimizations are apparently not as good as those of other C++
compilers owing to the needs and history of gcc. However, most of
these objective benchmarks are designed to get a false impression - and
even fail to do so when using more appropriate optimization flags.
This is not a valid way of benchmarking. To compare the performance,
you need to test the same algorithm coded in both languages.
This is interesting... I would guess that good functional languages will
optimize eg. the Fibonacci example to effectively avoid the recursion:
although the source looks like the same algorithm is used, the system
effectively does very different things. Is this a valid way of
benchmarking? I could easily contrive a set of benchmarks relying heavily
on recursion most functional programming language would win. Does this
benchmark tell me anything about using the languages for problems for
algorithms which are inherently non-recursive (of course, they can be
formulated recursively but the optimizations I used to win the benchmarks
do not kick in)?
You can't code a logarithmic time algorithm in C++ and an
exponential-time one in Java for the same problem and use the results
to claim that C++ is faster.
If the only thing the Java programmers are up to is an exponential-time
algorithm while the C++ people can come up with a logarithmic one, I
think this is a pretty reasonable approach! The point is, that people
comfortable in a language should do the best they can for the respective
benchmark. If the algorithms turn out to have different complexities
this might expose flaws in the respective languages and this is exactly
what benchmarks are about: to expose flaws. It is not the point of
benchmarks to tune them to let something particular look good. ... and
other benchmarks don't make any assumption about algorithms. For example,
a typical database benchmark tosses a set of SQL statements at a data
base and determines what it takes to process them. Is this unfair because
some database uses inferior algorithms and thus the others should be
disqualified? The point about the Fibonacci example was not that C++ is
much faster using this other algorithm. The point was that nobody would
implement this computation the way it was done in the first place -
except in a functional language where it is a rather sensible approach
to do.
You would have to
compare the above code with the *same* algorithm, coded in Java.
I don't think so. Substitute "*same*" by "*best*" and we agree.
The
algorithm used in the benchmarks is well-known to be a particularly
bad one for generating Fibonacci numbers, but the point is moot. In
the benchmark, you have to test the same algorithm in both languages,
not a fast algorithm coded in C++ against a slow one coded in Java.
Do you want me to design a few benchmarks particularily suitable for C++
which nobody in their right mind would implement that way in Java?
One of the things this "benchmark" highlights is that the developers for
other languages are working on improving the performance. I think this
should also apply to C++. Especially in the area of the standard C++
library I'm aware of a large potential for improvements. Eg. the fastest
version of the wc-benchmark I could come up with is only fast with my
own implementation of the standard C++ library where for_each() takes
advantage of the "segmented sequence optimization" and this code is not
[yet] available publically (I don't have the resources to finish up this
stuff...).