Employer looking 4 online C++ aptitude tests; recommendations?

[Matt]

Hello Ivan,

I very much agree with your comments below.

As part of the interview process, I will go further and give an actual
coding assignment. A company I worked for had a standardized assignments,
asking candidates to maintain and evolve an initial program through
multiple iterations of requirements.

I find this in particular to be an excellent idea. Unfortunately, my
current environment (the very early stages of a startup company) is
prohibiting me from spending much time on test development. I may
attempt to develop and use such a mechanism in the future.

Alternatively, if anyone can refer me to existing, already-developed
mechanisms like this one (the changing-requirements stuff is
particularly useful), I would love to see it.

I suppose I could draw from past projects (...but I can't find them
anywhere...I did my C++ work a *long* time ago) or look through a
couple of my favorite books...although I'd be concerned with
plagiarism for the latter.

-Matt

 

[Dietmar Kuehl]

Jerry Coffin wrote:
[ranting about the tests removed]

Thanks for sharing your experience: I had scheduled taking the test
later today just to see how it compares to Brainbench. I made a C++
test at Brainbench a few years ago and it was similar crap: it
tested MSVC++ specifics (the '__try' and '__finally' look like
particular extensions to me), had multiple choices questions with
the right answer being absent, etc. I managed to get a rating as
"master" anyway (and I was distrubed twice by phone calls, too...).
It seems that this site is similar crap.

 

[Ivan Vecerina]

As I've said here before, you are opening yourself up to lawsuit from an
alert candidate. If the candidate can show that he/she did well on the
written exam and that the exam is not predictive of performance, not
hiring
that candidate would (allegedly) show a bias in not hiring. The argument
goes that if you use a test and the candidate passes that test, then if
you
don't hire it's because of some other determination. Why bother using the
test?

IANAL of course, but I wouldn't take this as realistic legal advice.
Yes there are hiring caveats (avoiding sensitive topics, and ensuring
the fairness of the process), but the above is total FUD in my opinion.

Besides which, and more to the point, your questions are (so far) flawed
and/or meaningless. Predictive exams are hard to come by, and most on-line
quizes are pinpoint-limited to a very narrow range of syntax and language
topices --- hardly predictive of performance. I'd recommend you give up
the
idea of a test and stick with (admitedly) subjective determination, stated
this way: the candidate appears to have the experience based on resume and
references and an attitude which fits our way of operating. A trial
employment will determine if the candidate can do the job within the
normal
range of expertise we require. Then you conditionally hire for 30 days.

This is incredibly expensive! Plus someone you lay-off after 30 days
is much more likely to sue you for dicrimination or whatever other motive...

Sorry to sound so negative, but I've been down this road for almost 44
years. (My first experience was with the Programmer's Apptitude Test
given by IBM, in 1961). It still bothers me.

Screening tests are needed by large corporations that may receive
thousands of applications for a job opening. You can't hire them all
for 30 days based on a résumé and a nice interview chat!

Yes, tests are always flawed and biased. But by Bayesian logic, they
do help you get a vague opinion of someone's capabilities.

Lab or imaging exams made by your physician have flaws as well, but
in most cases they are much cheaper and safer than starting surgery
right away to see what is going on!


Kind regards,
Ivan

 

[Ondrej Brablc]

I think that it isn't very fortunate to hire programmers testing their
knowledge of a specific language. A better approach is to test their
algorithmic skills, their thinking. A language is just a tool and it's
usage can be learned quite fast.

Raw intelligence is of no use if it is not accompanied with knowledge.
I have seen many people claiming to be experts in C++ and having zero
knowledge of STL. Such people would reinvent the wheel again and again,
programming their own linked lists, ... - and they would do it damned
good and in relatively short time, but would you like to maintain that
many implementations?

Of course you do not need to know all constructors from STL, but you
should now that something like this exists and you should be able to
find it in manual or in TC++PL in matter of seconds. But you must have
an overview of both the C and C++ standard library.

I use following 4 points in interviews:

1) Fill one or both tests from
http://www.cpp-home.com/index/tiki-list_quizzes.php (after site upgrade
there something wrong with the tests)
2) Go through the test(s) and discuss the mistakes or good answers
which you know are difficult.
3) Unnamed C function of 10 lines. Give it correct name, tell me what
it does and show me how would you do it in your favorite language.
4) Optimize 12 lines long C function with known functionality.
Reference Manual can be used.

The points 3 and 4 are because of our legacy C code we have to
maintain. Of course the most important is the point 2 and if possible I
would skip the part 1 . If someone fails with operator precedence
question and tells me that he prefers to use more parenthesis, than he
is my man. The test can even be used for people who do not know the
language. You ask them how would they do it in their favorite language.

Ondrej

 

[Randy]

Hello,

I'm a hiring C++ developer employer looking for existing, online C++
aptitude tests.

I have not yet extensively researched this yet, but as an example, I
thought this test looked pretty good:

http://expertrating.com/c++test.asp

I'm curious if anyone can offer any other recommendations? If so,
could you please offer your experience with your recommended resource
and why you recommend them...or might recommend others?

Fyi, My philosophy falls in line with this post:

http://groups-beta.google.com/group/comp.lang.c++.moderated/msg/f840d9074af466c1

-Matt
--

I have to agree with several other posters who see little value in such
aptitude tests.

Instead, the approach that I have used in evaluating developer
candidates is to get them in a room with a white board and a couple of
other developers and ask the candidate to describe and explain a past
project of their choosing. If they can draw a diagram on the white
board showing what the pieces of that project were, and how they fit
together, and can provide reasonable answers as to what was done, and
more importantly why it was done that way, then they are likely to get
the job.

The things I look for most in such an interview are if the candidate
understands how and why C++ features such as inheritance or templates
were used in a real project, and if they can intelligently discuss the
alternatives and trade-offs involved in making such choices.

I myself do not necessarily recall off the top of my head all the
intricacies of C++, nor would I expect anyone else to do so. But I do
have a feel for C++ and how to apply it to real-world programming
problems, and I have a copy of the Standard, plus a pretty good library
of other books, on hand for reference whenever I need more detailed
info.

So another thing I look for in a developer interview is what C++ books
they are familiar enough with to actually discuss some of the content.
If they are not familiar with at least some of the mainstream
literature on C++, then they are not keeping their knowledge up to
date, and are not likely to get the job.

Finally I like to know if a candidate developer is familiar at all with
comp.lang.c++.moderated and/or comp.std.c++. If they follow these
newsgroups enough to be familiar with some of the ideas being
discussed, and with some of the names we see here on a regular basis,
then that is a big plus as well.

Just my two cents worth.

Randy.

 

[kanze]

called directly

It has undefined behaviour before it tries to call a dtor (it
has the wrong return type for main().

Which means that the compiler must emit a diagnostic.
Logically, the code shouldn't even compile.

This kind of thing says far more about the testers than it
does about the tested. If the examples you give are in anyway
typical I would want my money back because they claim that
every question has at least one correct answer, that claim is
demonstrably false.

It seems obvious to me that the test is worthless, because the
authors don't really know C++ themselves. It also means that
any company using it will probably succeed in filtering out any
competent applicants.

 

[Ivan Vecerina]

Hello Matt,

I find this in particular to be an excellent idea. Unfortunately, my
current environment (the very early stages of a startup company) is
prohibiting me from spending much time on test development. I may
attempt to develop and use such a mechanism in the future.

Working for a start-up company as well now, I know very well
what you mean. What I now try to do is have a few code samples
related to my field of activity, and somehow use them as
a basis for discussion (e.g. How would you fix that function?
What would you do if you now had to add this feature? ).

I know some who have also been looking for inspiration in a collection
of coding contest problems, such as http://acm.uva.es/problemset/

Alternatively, if anyone can refer me to existing, already-developed
mechanisms like this one (the changing-requirements stuff is
particularly useful), I would love to see it.

The problem with those company-tests is:
- companies want to keep them private (they are typically subject
to confidentiality agreements) - to avoid cheating, and as
a competitive asset.
- the best test would be geared towards the kind of development
you are doing internally (e.g. data processing, event-based
systems and state machines, graph-based algos, etc...).
(The latter has prevented me from developing a similar test
myself...).

I suppose I could draw from past projects (...but I can't find them
anywhere...I did my C++ work a *long* time ago) or look through a
couple of my favorite books...although I'd be concerned with
plagiarism for the latter.

Can you tell us more about the type of application that the candidate
will be working on? (e.g. part of the job description?)


Kind regards,
Ivan

 

[kanze]

Jerry Coffin wrote:
[...]

Even if the question were rephrased to deal with C++, it's
still downright stupid anyway! Given a limited number of
questions, which subject should be included: goto or (for
example) partial specialization?

150% agree with that!

From experience: just asking why you might want a virtual

destructor in a class is enough to eliminate most candidates.
With regards to templates, I suspect that just plain
specialization (not even partial specialization) or even a
non-type template parameter would be just as effective.

On the other hand, do you want to weed people out because
they've been using older compilers, or because they don't know
how to program in general. Unless I had special requirements,
I'd throw in a few questions about C++, but only enough to prove
that you actually have written and compiled a correct program
(i.e. that your CV isn't totally a piece of fiction), but I
don't think I'd go much farther, on the grounds that a competent
programmer will be able to learn the more complex things if
needed, but an incompetent programmer will still be an
incompetent programmer, even if he's seen all of the latest
novelties.

The hardest thing to test, of course, is his ability to
communicate with others and work in a team. Which remain the
most important qualities.

 

[Ivan Vecerina]

Jerry Coffin wrote:
[ranting about the tests removed]

Thanks for sharing your experience: I had scheduled taking the test
later today just to see how it compares to Brainbench. I made a C++
test at Brainbench a few years ago and it was similar crap: it
tested MSVC++ specifics (the '__try' and '__finally' look like
particular extensions to me), had multiple choices questions with
the right answer being absent, etc. I managed to get a rating as
"master" anyway (and I was distrubed twice by phone calls, too...).
It seems that this site is similar crap.

I sympathize. I think that the Brainbench test has improved since
you tried it. A couple of years ago, I have had the opportunity
to 'filter' the list of questions (hundreds of them!) and make a
large batch of corrections to improve style and conformance with
the C++ standard.
But this was made with very limited time and resources, and I
failed to agree with Brainbench on funding a project for renewing
the questions. I was still unhappy with the quality of many
questions, and especially with the coverage of the test
(e.g. lack of emphasis on the standard library). But I have not
been updated on the progress made since on the test.

Even though the Brainbench test may have changed quite a bit since
I reviewed it, I don't think you would find in it horrors such
as those described by Jerry in the "expertrating" one.
But it won't be flawless either...


Kind regards,
Ivan

 

[James Talbut]

I find this in particular to be an excellent idea. Unfortunately,
my
current environment (the very early stages of a startup company) is
prohibiting me from spending much time on test development. I may
attempt to develop and use such a mechanism in the future.

It doesn't take very long to prepare them, it takes quite a bit of
time to administer them.
The tests themselves can be quite straightforward (we used the
travelling salesman for quite a while, but you can only use something
like that until the candidates find out about it advance).

Almost noone finished the problems set in the available time (I tried,
knowing the problem in advance and knowing simmulated annealing and it
took me about ten minutes longer than was allocated but that's not
important.
You are not interested in whether or not they can solve the problem,
you are interested in how they go about trying to solve it.
Going through the candidates workings can take quite a bit of time,
but it is very informative.
Unless the candidate is a complete numpty you don't come to any final
conclusions this way, but you get information to guide your interview.

 

[Jerry Coffin]

Carlos Moreno wrote:

[ mention of __try, __finally, __except ... ]

Actually, d) is correct, IMHO -- If there is no such thing
as a __try statement block in C++, then a statement that
says you can do whatever thing into a __try statement block
is false: it is false because there is no such thing as
a __try block, and therefore NO, you can NOT use a goto to
jump into a __try block.

Even that isn't quite true though: since all these identifiers are in
the implementor's name space, it's perfectly legitimate for a fully
conforming compiler to include them, and C++ programs that make use of
them are subject to the implementation's rules. As such, none of the
statements is really even necessarily false.

 

[Jerry Coffin]

Andrew Peter Marlow wrote:

[ ... ]

brainBench seems quite good to me.
It draws from a very large database, making it harder
for agents to cheat, and although it is multiple-choice
the choices have subtle differences. Also the responses
have a time limit.

BrainBench may or may not be stronger on testing methodology --
ExpertRating does have a time-limit, but I'm not sure whether the test
is a fixed set of questions or drawn from a larger pool.

At least based on my recollections from taking their C++ test years ago
(under circumstances much like this time) BrainBench's questions were
even worse though. I'd say around 70-75% of ExpertRating's questions
had reasonably correct answers among the choices given, and only about
15% or so covered truly mind-bendingly stupid subjects. In the case of
BrainBench, my recollection is that those numbers were barely over 50%
and around 30% respectively.

 

[Mike Wahler]

Raw intelligence is of no use if it is not accompanied with knowledge.

and skill.

I have seen many people claiming to be experts in C++ and having zero
knowledge of STL. Such people would reinvent the wheel again and again,
programming their own linked lists, ... - and they would do it damned
good and in relatively short time, but would you like to maintain that
many implementations?

I feel that the paramount skill needed of a programmer is
problem-solving. A good problem solver will already know
to look for existing tools before trying to invent them.
The more astute problem solvers will also add 'standardized'
or 'in common use' to the description of sought-after tools.
(this will give a 'guideline' about how much confidence one
can place in such tools when discovered).

Of course you do not need to know all constructors from STL, but you
should now that something like this exists and you should be able to
find it in manual or in TC++PL in matter of seconds. But you must have
an overview of both the C and C++ standard library.

Yes, I agree one should have an overall view of the major
features of the standard library, and that memorizing them
is not at all necessary, they can be looked up when used.

-Mike

 

[Jerry Coffin]

[ ... ]

1) Fill one or both tests from
http://www.cpp-home.com/index/tiki-list_quizzes.php (after site upgrade
there something wrong with the tests)

Hmm....still somewhat short of perfect. E.g:

Q: What will !((1 || 0) && 0) evaluate to?

A. Wrong code
B. Undefined behaviour
C. I don't know
D. 1
E. 0

Without even working through the logic, it's clear that none of these
choices can possibly be correct in C++, in which the result of a
logical expression is a bool (i.e. either 'true' or 'false'). Now, if
this was about C, then D and E would be possibilities, but in C++
they're clearly not.

OTOH, this is clearly a better test than the commercial ones I've seen.

 

[Mike Wahler]

[ ... ]

1) Fill one or both tests from
http://www.cpp-home.com/index/tiki-list_quizzes.php (after site upgrade
there something wrong with the tests)

Hmm....still somewhat short of perfect. E.g:

Q: What will !((1 || 0) && 0) evaluate to?

A. Wrong code
B. Undefined behaviour
C. I don't know
D. 1
E. 0

F. Why the heck would someone write that expression anyway?



-Mike

 

[Gary Labowitz]

news:[email protected]...

Screening tests are needed by large corporations that may receive
thousands of applications for a job opening. You can't hire them all
for 30 days based on a résumé and a nice interview chat!

Yes, tests are always flawed and biased. But by Bayesian logic, they
do help you get a vague opinion of someone's capabilities.

Well, first, that's what HR departments are for. They eliminate people who
apply for jobs. They do the initial screening and send the postcard that
says "Thanks, but no thanks" to candidates that don't look promising. (And
some people think HR departments hire employees!)

Tests I know of test such specific items that they prove little except the
test-passer knows those items. On the job you see what someone knows how to
do, and if he knows how to look up what he doesn't know. You get an example
of work done under the actual conditions. You see the interaction of the new
employee with your staff, etc. This is the only way to actually find out if
a candidate can do what you are willing to pay for. And, yes, it is
expensive to hire and then let go. But every job I've ever had was
conditional on performance. Any employer who hires and then is stuck with
who them picked is either a fool or the government.

It is just as important for the candidate to evaluate the company and job
she is applying for. We have had new hires come in and say that they didn't
think this was what they wanted and quit on the spot. Actually, this was
better for everyone in the long run, but it is tough on all the parties when
this happens.

Selecting new employees is not an easy job in any case, and shouldn't be
left to amateurs. (P.S. I have no solution.)
And we are getting way OT.

 

[Gary Labowitz]

message
Kinda an aside:
Some years ago I was interviewing for a contract job and the nice man asked
me "What is a PCB?" Well, it didn't seem to fit the job they wanted, but I
started to explain the Program Control Block, when it's built, what it
contains, some discussion of TIOT's, the save area chain, etc. and then
asked him why this was important for this job. He said he had just heard the
term around the shop and was curious as to what it was. (I didn't get the
job anyway.)

Maybe a better technique for today is to google the candidate and see what
ng posts he has. I have made some decisions about most of YOU guys and which
of you I would like to have on my team.

How about these questions:
What is the STL?
How would you go about looking up information about vectors?
What programming language ng's do you read regularly?
Have you ever had to use bit shift operators? Tell me about it.

 

[Walter]

It seems obvious to me that the test is worthless, because the
authors don't really know C++ themselves. It also means that
any company using it will probably succeed in filtering out any
competent applicants.

I'd explain to them what is wrong with the test. If they're not interested
and not willing to listen, *then* I'd walk away.

 

[Ivan Vecerina]

It doesn't take very long to prepare them, it takes quite a bit of
time to administer them.

Depends. At some company, years ago, I had to make a very well-designed
interview test. I can't give details, it was a complex kind of state
machine. You start with a source file, part of a large application,
that 'joe hacker' wrote in a rush for v1.0 of the product.
Then you get a storyboard with multiple iterations, with new
requirements that have to be implemented for each iteration.
To end each iteration, you run a specified makefile that will compile
your updated source and link it with a test application that runs
a bunch of unit tests (each attempt is logged, and results + source
are e-mailed to the supervisor).
You are left alone for a few hours, with some books at hand if you
wish, and a phone to call your supervisor if you would like some help.

The test had been in use for several years, and data was carefully
collected. The supervisor could immediatly place you on a 'performance
percentile' based on how far you got. He could also inspect the
quality of the code produced under relative stress, and how the changing
requirements were dealt with. The reaction when in
difficulty could also be assessed (well, unless the candidate was
in the <1% who was able to get reach the last iteration on his own,
and tried to race for the virtually unreachable finish line).

The tests themselves can be quite straightforward (we used the
travelling salesman for quite a while, but you can only use something
like that until the candidates find out about it advance).

Yes, but a test problem related to your field of application
will be of greater value.
But if all you are looking for is a coding challenge, yes,
there are a number of examples available for free. Just pick one
from http://acm.uva.es/problemset/ (there are >1000, with an
online judge that will run submissions through a test harness).
These can actually serve as a good example. You could even take
a chance and remotely ask your candidate to complete one of the
provided exercices (but beware of solution code available on
various forums, and other ways of cheating...).

Such a 'coding challenge' might actually be better than a multiple
choice test for screening job candidates - but its predictive value
(w.r.t. on-the-job performance) also can be disputed...


Cheers,
Ivan

 

[Bob Hairgrove]

Q: What will !((1 || 0) && 0) evaluate to?

A. Wrong code
B. Undefined behaviour
C. I don't know
D. 1
E. 0

Without even working through the logic, it's clear that none of these
choices can possibly be correct in C++, in which the result of a
logical expression is a bool (i.e. either 'true' or 'false'). Now, if
this was about C, then D and E would be possibilities, but in C++
they're clearly not.

OTOH, this is clearly a better test than the commercial ones I've seen.

Well, it does compile and run correctly under Comeau, returning 1:

#include <iostream>
#include <ostream>

int main()
{
std::cout << "Result of !((1 || 0) && 0):\t";
std::cout << !((1 || 0) && 0) << std::endl;
return 0;
}

After all, bool is defined by the standard to be "an integral type".
Question is, whether other output != 0 (e.g. 4711) should be allowed.

Does the standard specify what the integral value of "true" should be?