Clarity vs. code reuse/generality

[pdpi]

That is completely confusing.  I get the, er, impression that "sense"
is supposed to change during the loop, and it takes much head
scratching to tell whether what you have there is right or not.  Also,
it calls func 3 times on each loop, which could be extremely slow.
You don't know what func does, so eliminating 2/3 of the calls to it
is not a micro-optimization--it could be a major time saving.

Yet another version:

    def _binary_search(x0, x1, func, target, epsilon):
        y0,y1 = func(x0), func(x1)
        while abs(y1 - target) > epsilon:
            if x0 == x1 or cmp(y0,target) == cmp(y1,target):
                return None
            xn = (x0 + x1) / 2.
            yn = func(xn)
            if cmp(yn,target) == cmp(y0,target):
               x0,y0 = xn,yn
            else:
               x1,y1 = xn,yn
        return x1

micro-optimization was perhaps too harsh, but it is an optimization
that's not obvious for the newbie, and one best learnt from experience
rather than having it spoon fed. I'll restate: you should start with
the cleanest code possible and mangle it for performance. Then again,
that implicitly assumes that f(x) is more readable than y, which is
really a matter of taste...

 

[Tim Rowe]

Maybe the reason for "so much buggy software" is that people
inappropriately use assert, thus changing the behaviour of code depending
on whether it is run with the -O flag or not.

I've done my share of code review and process audits, and assertions
seem *far* to rare for that distinction.

I don't know what "hostility" you're seeing. The only hostility I'm
seeing is from the OP, which is bizarre considering that he asked for
advice and we gave it. What I see is a bunch of people concerned that the
OP is teaching novices a bad habit, namely, using assert for error
checking. He claims -- angrily and over and over again -- that in his
code, the assertions should never fail. Great. Good for him for knowing
when to use assert. But are the novices going to learn that lesson, or
will they simply learn "use assert for error checking"?

They are rather more likely to learn if they are taught, aren't they?
Or do you think it's better to keep them in the dark? "There are these
things called assertions -- work them out for yourselves".

I am convinced that the time to teach programmers to consider under
what circumstances a routine can be called and who is responsible for
ensuring that those conditions are met is as soon as they hit the
concept of calling routines. And assertions provide an excellent way
of doing that, fostering good habits for the rest of their careers.
Any hostility from the OP seems to be a response to the persistent
refusal to accept his assurances that he is not using the assertions
for run-time error checking, nor teaching the students to do that,

 

[kj]

In general, code clarity is more important than reusability.
Unfortunately, many novice programmers have the opposite impression. I
have seen too much convoluted code written by beginners who try to
make the code generic. Writing simple, clear, to-the-point code is
hard enough as it is, even when not aiming at making it reusable.

If in the future you see an opportunity to reuse the code, then and
only then is the time to make it generic.

YAGNI is a wonderful principle.

Thanks!

kynn

 

[kj]

First, a quote which took me a bit to find:
Thomas William Körner paraphrasing Polya and Svego
in A Companion to Analysis:
Recalling that 'once is a trick, twice is a method,
thrice is a theorem, and four times a theory,' we
seek to codify this insight.

Good stuff.

Let us apply this insight:
Suppose in writing code, we pretty much go with that.
A method is something you notice, a theorem is a function, and
a theory is a generalized function.

Even though we like DRY ("don't repeat yourself") as a maxim, let
it go the first time and wait until you see the pattern (a possible
function). I'd go with a function first, a pair of functions, and
only then look to abstracting the function.

Thanks!

kynn

 

[kj]

2009/7/4 kj <[email protected]>:

Good for you. I'm convinced that you have used the assertion
appropriately, and the fact that so many here are unable to see that
looks to me like a good case for teaching the right use of assertions.
For what it's worth, I read assertions at the beginning of a procedure
as part of the specification of the procedure, and I use them there in
order to document the procedure. An assertion in that position is for
me a statement to the user of the procedure "it's your responsibility
to make sure that you never call this procedure in such a way as to
violate these conditions". They're part of a contract, as somebody
(maybe you) pointed out.

As somebody who works in the safety-critical domain, it's refreshing
to see somebody teaching students to think about the circumstances in
which a procedure can legitimately be called. The hostility you've
received to that idea is saddening, and indicative of why there's so
much buggy software out there.

Thanks for the encouragement.

When I teach programming, the students are scientists. For the
stuff they do, correctness of the code trumps everything else. I
teach them to view assertions as way of translating their assumptions
into code. And by this I mean not only assumptions about the
correctness of their code (the typical scope of assertions), but
also, more broadly, assumptions about the data that they are dealing
with (which often comes from external sources with abysmal quality
control).

My scientific code is jam-packed with assertions. I can't count
the number of times that one such lowly assertion saved me from a
silent but potentially disastrous bug. And yes I find it distressing
that so so few programmers in my line of work use assertions at
all.

kynn

 

[Gabriel Genellina]

In <[email protected]> Tim Rowe




Thanks for the encouragement.

When I teach programming, the students are scientists. For the
stuff they do, correctness of the code trumps everything else. I
teach them to view assertions as way of translating their assumptions
into code. And by this I mean not only assumptions about the
correctness of their code (the typical scope of assertions), but
also, more broadly, assumptions about the data that they are dealing
with (which often comes from external sources with abysmal quality
control).

Nobody says you shouldn't check your data. Only that "assert" is not the
right way to do that. Ok, it's easier to write:

assert x>0 and y>0 and x<y

instead of:

if not (x>0 and y>0 and x<y):
raise ValueError, "x=%r y=%r" % (x,y)

but the assert statement is completely ignored if your script is run with
the -O option, and then no check is done.
assert should be used only to verify internal correctness only - to detect
wrong assumptions in the *code* itself. Once you're confident the code is
OK, you may turn off assertions (and improve program speed). But you must
*always* validate input data no matter what - so assert is not the right
tool.

Even worse, never DO anything inside an assertion - it won't be done in
the optimized version. This method becomes almost empty when assertions
are removed (not the intended behavior!):

def process_all(self):
for item in self.items:
assert self.process(item)

My scientific code is jam-packed with assertions. I can't count
the number of times that one such lowly assertion saved me from a
silent but potentially disastrous bug. And yes I find it distressing
that so so few programmers in my line of work use assertions at
all.

I'm just saying that some of those assertions probably should be real "if
.... raise" statements instead - not that you remove the checks. You may
use this short function, if you prefer:

def fassert(condition, message='', exc_type=AssertionError):
if not condition:
raise exc_type(message)

 

[Ethan Furman]

My scientific code is jam-packed with assertions. I can't count
the number of times that one such lowly assertion saved me from a
silent but potentially disastrous bug.

Now imagine that asserts had been disabled for that run...

The issue is not "should you validate your inputs", the issue is "do you
want your validation able to be turned off?"

~Ethan~

 

[Nobody]

Nobody says you shouldn't check your data. Only that "assert" is not the
right way to do that.

"assert" is not the right way to check your *inputs*. It's a perfectly
reasonable way to check data which "should" be valid, as well as a way to
document what variables are supposed to contain.

 

[Steven D'Aprano]

"assert" is not the right way to check your *inputs*. It's a perfectly
reasonable way to check data which "should" be valid, as well as a way
to document what variables are supposed to contain.

Where are those variables coming from?

The distinction really boils down to this:

* asserts should never fail. If there is any chance that an assertion
might fail outside of test suites, then don't use assert.


You can't control what input the caller provides to a function, so unless
the data is generated inside the function, a caller might provide
something unexpected. Therefore, assert is inappropriate for checking
function parameters, even if that function is "private" and only called
by your own functions.

(1) So-called "private" or "internal" functions have a habit of becoming
public, and then you have asserts in public functions.

(2) If public() calls _private(x), and _private uses assert to check the
value of x, there is a risk that if public() is buggy and supplies an
invalid x, the assertion will never be checked and _private() will return
incorrect results.

(3) assert is absolutely unsuitable for enforcing pre-conditions and post-
conditions, unless such conditions are mere "guidelines", because assert
can be switched off at runtime.



It's a fine line to draw. Obviously, if we think something *truly* can
never fail, we don't bother checking it:

def add(x, y):
result = x+y
assert result-y == x
assert result-x == y
return result

is probably overkill for most programs, and yet surprisingly those
assertions will commonly fail for float arguments!

 

[Jean-Michel Pichavant]

"assert" is not the right way to check your *inputs*. It's a perfectly
reasonable way to check data which "should" be valid, as well as a way to
document what variables are supposed to contain.

Maybe, one of the assert problem is a semantic issue. Assert is a
convenient way to write in the code "This is a requirement, get lost if
you don't fit in". However it seems we often forget that builtin asserts
can be disabled.
One possible solution for those who prefer to write assertions instead a
'if A then Exception' is to write their own assertion statement and
**raise an exception** that will not be disabled at runtime.

Jea-Michel

 

[Tim Rowe]

Thanks for the encouragement.
[snip]

into code.  And by this I mean not only assumptions about the
correctness of their code (the typical scope of assertions), but
also, more broadly, assumptions about the data that they are dealing
with (which often comes from external sources with abysmal quality
control).

There we diverge. A lot. If "correctness of the code trumps everything
else" (in fact, if it matters at all) and the external data has
"abysmal quality control" then it *must* be checked for correctness
before it is used. If it is not, you have no idea whether your output
is correct or not. And assertions *will* *not* reliably provide that
checking (because they may not be executed). You *must* actively check
the data, using good old-fasioned "if" statements and so on, because
not to do so is to declare that you *don't* care about correctness.
You *know* the input is often wrong, but you're not bothering to check
it?

 

[Charles Yeomans]

Where are those variables coming from?

The distinction really boils down to this:

* asserts should never fail. If there is any chance that an assertion
might fail outside of test suites, then don't use assert.


You can't control what input the caller provides to a function, so
unless
the data is generated inside the function, a caller might provide
something unexpected. Therefore, assert is inappropriate for checking
function parameters, even if that function is "private" and only
called
by your own functions.

(1) So-called "private" or "internal" functions have a habit of
becoming
public, and then you have asserts in public functions.

(2) If public() calls _private(x), and _private uses assert to check
the
value of x, there is a risk that if public() is buggy and supplies an
invalid x, the assertion will never be checked and _private() will
return
incorrect results.

(3) assert is absolutely unsuitable for enforcing pre-conditions and
post-
conditions, unless such conditions are mere "guidelines", because
assert
can be switched off at runtime.

Unless, of course, you want to switch off such checking at runtime, as
you might when using a design-by-contract approach.

Charles Yeomans

 

[J. Cliff Dyer]

Where are those variables coming from?

The distinction really boils down to this:

* asserts should never fail. If there is any chance that an assertion
might fail outside of test suites, then don't use assert.

I'm no expert, but the more I read this thread, and the more I think on
it, the more I believe that asserts don't really need to exist outside
of test suites. The function that assertions provide is handled in a
far more robust and maintainable way by unit tests and doctests.
Anything else should be handled by more descriptive exceptions.

The use of assertions in regular code may just be a historical baby step
on the way to real code testing.

To play devils advocate for a moment, one possible use case for assert
statements is if you need to test something that you can't easily get
under a proper unittest or doctest. Maybe you need to know the value of
a variable halfway through a method. A judicious assertion can help
supplement your test suite A better solution would be to refactor so
you can get the needed value under test, but if time constraints won't
allow it, then make your assertion and move on, but only to help you
debug the code, not to protect your code at runtime. Even then, why not
use a proper Exception (unless speed is a major issue)?

Even if it is only used internally in a module, I would still prefer a
ValueError to an AssertionError. It tells you what's happening more
clearly. And it protects you if another caller (even one internal to
the class) calls it with a different set of assumptions.

At minimum, I think there's a heavy burden on an author to justify the
use of AssertionErrors rather than other kinds of Exceptions.

Cheers,
Cliff

 

[Robert Kern]

I'm no expert, but the more I read this thread, and the more I think on
it, the more I believe that asserts don't really need to exist outside
of test suites.

Actually, there is a good argument that one shouldn't use an assert statement in
test suites: code can have bugs that only show up under -O so you want to be
able to run your test suite under -O.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[J. Cliff Dyer]

Actually, there is a good argument that one shouldn't use an assert statement in
test suites: code can have bugs that only show up under -O so you want to be
able to run your test suite under -O.

That's an interesting point. Presumably TestCase.assert_() doesn't
suffer from this defect? Otherwise the entire unittest suite is
essentially broken by your argument. I suppose I should have said one
should only raise AssertionErrors in test suites. Practically speaking,
that's what a test suite is: The place where you assert what the code
does.

 

[Robert Kern]

That's an interesting point. Presumably TestCase.assert_() doesn't
suffer from this defect? Otherwise the entire unittest suite is
essentially broken by your argument.

It explicitly raises AssertionErrors. It does not use the assert statement.

I suppose I should have said one
should only raise AssertionErrors in test suites. Practically speaking,
that's what a test suite is: The place where you assert what the code
does.

Yup.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Ethan Furman]

On Mon, 06 Jul 2009 14:32:10 +0200, Jean-Michel Pichavant wrote:

kj wrote:

sense = cmp(func(hi), func(lo))
assert sense != 0, "func is not strictly monotonic in [lo, hi]"

As already said before, unlike other languages, sense in english does
**not** mean direction. You should rewrite this part using a better
name. Wrong informations are far worse than no information at all.

Absolutely.

From Webster's Dictionary:

8. (Geom.) One of two opposite directions in which a line,
surface, or volume, may be supposed to be described by the motion
of a point, line, or surface.
[1913 Webster]


And from WordNet:

2: the meaning of a word or expression; the way in which a word
or expression or situation can be interpreted

Both meanings are relevant to the way KJ is using the word. Please take
your own advice and stop giving wrong information. As a native English
speaker, I had no difficulty understanding the meaning of "sense" in the
sense intended by KJ.

As another native English speaker, I agree with Jean-Michel; this is the
first time I've seen "sense" used to mean direction.

Just goes to show you learn something new all the time.

http://www.merriam-webster.com/dictionary/sense

7: one of two opposite directions especially of motion (as
of a point, line, or surface)


http://dictionary.reference.com/browse/sense

18. Mathematics. one of two opposite directions in which
a vector may point.



Paraphrasing the Collins Dictionary of Mathematics:

The sense of a vector is the sign of the measure, contrasted with the
magnitude. Thus the vectors AB and BA have the same direction but
opposite sense. Sense is also used to distinguish clockwise and anti-
clockwise.

Sense is, if you like, a "signed direction". "Towards north" (say) as
opposed to "along the north-south axis".

This also illustrates the importance of knowing your target audience. I
have also not seen "sense" used this way before, and from the placement
in the dictionaries I would venture to say it's not common usage outside
of mathematics and the sciences.

Of course, since kj is teaching biologists, odds are decent they know
what he's talking about.

~Ethan~

 

[Steven D'Aprano]

Unless, of course, you want to switch off such checking at runtime, as
you might when using a design-by-contract approach.

So is design-by-contract just another way of saying "let's hope the data
is valid, because if it's not, we're screwed"?

Perhaps Python should have a new statement, `assume`, used just like
`assert` except it never actually performs the test or raises an error.

 

[Dennis Lee Bieber]

Maybe, one of the assert problem is a semantic issue. Assert is a
convenient way to write in the code "This is a requirement, get lost if
you don't fit in". However it seems we often forget that builtin asserts
can be disabled.

I look on asserts as programmer hints... That is, if the assert ever
fails, the programmer is responsible for finding where in the code PRIOR
to the assert he/she failed to provide sanity checks and/or data
validation on the input that feeds the code with the assertion.

An assert should NEVER trigger in production code.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Terry Reedy]

So is design-by-contract just another way of saying "let's hope the data
is valid, because if it's not, we're screwed"?

Perhaps Python should have a new statement, `assume`, used just like
`assert` except it never actually performs the test or raises an error.

The manual says quite clearly

"The simple form, assert expression, is equivalent to
if __debug__:
if not expression: raise AssertionError"

It should be used when and only when one actually wants the double
condition check, and not as an abbreviation of the second conditional only.

tjr