Exception as the primary error handling mechanism?

[Dave Angel]

<snip>

Maybe, although I recently learned on here that one can't rely on assert
statements in production code, their intended use is to aid debugging
and testing really.

Hopefully, what you learned is that you can't use assert() in production
code to validate user data. It's fine to use it to validate program
logic, because that shouldn't still need testing in production.

<snip>

DaveA

 

[r0g]

Hopefully, what you learned is that you can't use assert() in production
code to validate user data. It's fine to use it to validate program
logic, because that shouldn't still need testing in production.

<snip>

DaveA

Well maybe I didn't quite get it then, could you explain a bit further?

My understanding was that asserts aren't executed at all if python is
started with the -O or -OO option, or run through an optimizer. If
that's the case how can you expect it to validate anything at all in
production? Do you mean for debugging in situ or something? Could you
maybe give me an example scenario to illustrate your point?

Cheers,

Roger.

 

[Roy Smith]

No, but that's why I try not to use languages where you can only return
a single result, I always found that an arbitrary and annoying
constraint to have. I leads to ugly practices like "magic" return values
in C or explicitly packing things into hashtables like PHP, yuk!

Python only lets you return a single result, just like C or C++.

The difference is that in Python it's trivial to build tuples on the fly
and return them. About the closest you get to that in C++ is std:air,
and that's about 5 bucks short and a week late.

 

[Steven D'Aprano]

Well maybe I didn't quite get it then, could you explain a bit further?

My understanding was that asserts aren't executed at all if python is
started with the -O or -OO option,
Correct.


or run through an optimizer.

I don't know what you mean by that.

If
that's the case how can you expect it to validate anything at all in
production?

The asserts still operate so long as you don't use the -O switch.

Do you mean for debugging in situ or something? Could you
maybe give me an example scenario to illustrate your point?

There are at least two sorts of validation that you will generally need
to perform: validating user data, and validating your program logic.

You *always* need to validate user data (contents of user-editable config
files, command line arguments, data files, text they type into fields,
etc.) because you have no control over what they put into that. So you
shouldn't use assert for validating user data except for quick-and-dirty
scripts you intend to use once and throw away.

Program logic, on the other hand, theoretically shouldn't need to be
validated at all, because we, the programmers, are very clever and
naturally never make mistakes. Since we never make mistakes, any logic
validation we do is pointless and a waste of time, and therefore we
should be able to optimise it away to save time.

*cough*

Since in reality we're not that clever and do make mistakes, we actually
do want to do some such program validation, but with the option to
optimise it away. Hence the assert statement.

So, a totally made-up example:


def function(x, y):
if x < 0:
raise ValueError("x must be zero or positive")
if y > 0:
raise ValueError("y must be zero or negative")
z = x*y
assert z < 0, "expected product of +ve and -ve number to be -ve"
return 1.0/(z-1)



This example cunningly demonstrates:

(1) Using explicit test-and-raise for ensuring that user-supplied
arguments are always validated;

(2) Using an assertion to test your program logic;

(3) That the assertion in fact will catch an error in the program logic,
since if you pass x or y equal to zero, the assertion will fail.


Any time you are tempted to write a comment saying "This can't happen,
but we check for it just in case", that is a perfect candidate for an
assertion. Since it can't happen, it doesn't matter if it goes away with
the -O flag; but since we're imperfect, and we want to cover ourselves
just in case it does happen, we perform the test when not optimized.

From my own code, I have a global constant:

UNICODE_NUMERALS = u'\uff10\uff11\uff12\uff13\uff14\uff15\uff16\uff17
\uff18\uff19'


And then to make sure I haven't missed any:

assert len(UNICODE_NUMERALS) == 10


In another function, I validate a mapping {key:value} to ensure that all
the values are unique:

seen_values = set()
for k,v in mapping.items():
if v in seen_values:
raise ValueError('duplicate value %s' % k)
seen_values.add(v)
# If we get here without error, then the mapping contains no
# duplicate values.
assert len(seen_values) == len(mapping)


The assertion acts as a double-check on my logic, not the data. If my
logic is wrong (perhaps there is a way to get past the for-loop while
there is a duplicate?) then the assertion will catch it.

 

[D'Arcy J.M. Cain]

shouldn't use assert for validating user data except for quick-and-dirty
scripts you intend to use once and throw away.

A mythcial beast that has yet to be spotted in the wild.

 

[r0g]

No, there is nothing inherent to the ‘return’ statement for dealing with
multiple values.

The ‘return’ statement *always* returns a single value: whatever value
you specify as the argument to the statement (or the value ‘None’ if no
argument is specified. If you specify a tuple — and Python has a nice
syntax for creating a literal tuple — that's the single value the
statement will use.

That's what I said Ben...


See how I agree that "The ‘return’ statement *always* returns a single
value"?



You're confusing literal and conceptual returns. You can tell there are
two senses of return at play because the two returns have different
possessives within the same sentence. Look...

"Yes, IT returns a tuple" - Speaking literally about Python...

return 1, 2, 3 # Python returned one value

"YOU return more than one value" - Speaking conceptually from a
programmers perspective.

return 1, 2, 3 # I returned three values

Both are valid in their own context. The second (conceptual) statement
MIGHT be ambiguous if it weren't for the first literal one. Both cannot
be literally true and the first one clearly IS a literal factual
statement about Python so the second cannot also be interpreted as
literal. So let's disgard it...

The first two "it"s are Python, the third could either be...

(a) The construction of tuples

or at a stretch...

(b) The act of returning tuples i.e. some special syntax for returning
tuples.


You seem to think I meant (b) - I actually meant (a)

Maybe I could have made that clearer but I don't think people want to
read legalese and I think it takes a particular pedantic, nitpicking
disposition to even spot such small ambiguities.

Of course I'm now guilty of pedantry too :/ I might have let it slip had
you not started your reply with the word "No", that just p***** me off.

Having said that I find the mental image of you slamming your fist on
the table and shouting it out loud whenever you read something you
disagree with on usenet quite amusing!

Roger.

 

[r0g]

I don't know what you mean by that.

I've never used them but I heard there are optimizers for python
(psycho?). I assumed these would do everythin -O does and more,
including losing the asserts.

The asserts still operate so long as you don't use the -O switch.



There are at least two sorts of validation that you will generally need
to perform: validating user data, and validating your program logic.

<snipped very detailed and clear response, thanks >


Cool, that's what I thought i.e. you can't rely on asserts being there
so don't use them for anything critical but it's still a good idea to
use them for logic/consistency checking in production code as, should
you be running your production code unoptimised, it might catch
something you'd otherwise miss.

Thanks for responding is such detail

Roger.

 

[Dave Angel]

Well maybe I didn't quite get it then, could you explain a bit further?

My understanding was that asserts aren't executed at all if python is
started with the -O or -OO option, or run through an optimizer. If
that's the case how can you expect it to validate anything at all in
production? Do you mean for debugging in situ or something? Could you
maybe give me an example scenario to illustrate your point?

Cheers,

Roger.

You understand the -O and -OO options fine. But the point is that you
should not use assert() for anything that will be properly debugged
before going to the user. You use if statements, and throw's to catch
the error, and print to stderr, or GUI dialog boxes, or whatever
mechanism you use to tell your user. But those errors are ones caused
by his data, not by your buggy code. And the message tells him what's
wrong with his data, not that you encountered a negative value for some
low level function.

I agree with Steve's pessimistic view of the state of most released
software. But if you view a particular internal check as useful for
production, then it should be coded in another mechanism, not in
assert. Go ahead and write one, with a UI that's appropriate for your
particular application. But it should do a lot more than assert does,
including telling the user your contact information to call for support.

def production_assert(expression, message):
if not expression:
dialog_box("Serious internal bug,
call NNN-NNN-NNNN immediately", message)


For an overly simplified example showing a user validation, and an assert :

import sys
def main():
try:
text = raw_input("Enter your age, between 1 and 22 ")
age = int(text)
except ValueError, e:
age = -1
if not 1 <= age <= 22: #not an assert
print "Age must be between 1 and 22"
print "Run program again"
sys.exit(2)
grade = calc_grade(age)
print "Your grade is probably", grade

table = [0, 0, 0, 0, 0, "K", "First", "2nd", 3]
def calc_grade(age):
""" calculate a probable grade value, given an
i2nteger age between 1 and 22, inclusive
"""
assert(1 <= age <= len(table))
grade = table[age] #assume I have a fixed-length table for this
return grade

main()

Note a few things. One I have a bug, in that the table isn't as big as
the limit I'm checking for. With defensive coding, I'd have another
assert for that, or even have the table size be available as a global
constant (all uppers) so that everyone's in synch on the upper limit.
But in any case, the test suite would be checking to make sure the code
worked for 1, for 22, for a couple of values in between, and that a
proper error response happened when a non-integer was entered, or one
outside of the range. That all happens in separate code, not something
in this file. And the test suite is run after every change to the
sources, and certainly before release to production.

Next, see the docstring. It establishes a precondition for the
function. Since the function is called only by me (not the user), any
preconditions can be checked with an assert. An assert without a
supporting comment (or docstring) is almost worthless.

And finally, notice that I check the user's input *before* passing it on
to any uncontrolled code. So any asserts after that cannot fire, unless
I have a bug which was not caught during testing.

All opinions my own, of course.

DaveA

 

[Lie Ryan]

<snipped very detailed and clear response, thanks >


Cool, that's what I thought i.e. you can't rely on asserts being there
so don't use them for anything critical but it's still a good idea to
use them for logic/consistency checking in production code as, should
you be running your production code unoptimised, it might catch
something you'd otherwise miss.

Steven described the traditional approach to using assertions; another
approach to when to use assertion is the one inspired by
Design-by-Contract paradigm. DbC extends the traditional approach by
focusing on writing a contract (instead of writing assertions) and
generating assertions[1] to validate the contract. Just like assertions,
these contracts are meant to be removed in production releases.

In Design-by-Contract, only codes that interacts with the outer-world
(e.g. getting user/file/network input, etc) need to do any sort of
validations. Codes that doesn't interact directly with outside world
only need to have a "contract" and simplified by *not* needing argument
checking, since the function relies on the caller obeying the
contract[2] and never calling it with an invalid input.

DbC uses assertions[1] spuriously, unlike the traditional approach which
is much more conservative when using assertions.

[1] or explicit language support which is just syntax sugar for assertions
[2] of course, on a debug release, the contract validation code will
still be enforced to catch logic/consistency bugs that causes the violation

 

[Steve Holden]

A mythcial beast that has yet to be spotted in the wild.

Not true (he wrote, picking nits). Such programs are written all the
time. The fact that they invariably get used more often than intended
doesn't negate the intentions of the author. ;-)

regards
Steve

 

[Steve Holden]

A mythcial beast that has yet to be spotted in the wild.

Not true (he wrote, picking nits). Such programs are written all the
time. The fact that they invariably get used more often than intended
doesn't negate the intentions of the author. ;-)

regards
Steve

 

[Steven D'Aprano]

Well, if being told “no†is going to piss you off, I think you're in for
a rough time.

Oh, you're in trouble now! If you think he gets upset at being told no,
you should see how upset he gets at being told he's in for a rough time!!!

*wink*

 

[Phlip]

Otherwise, could some python expert explain to me why exception is
widely used for error handling in python? Is it because the efficiency
is not the primary goal of python?

It's not about efficiency, it's about making assumptions for the
programmer about what kind of rigor they need.

Why can't int('nonnumeric') return None?

Why can't a Django Record.objects.get(pk=-1) return a None? That's
what it's for.

(A related question - why can't I just go 'if record = method(): use
(record)'. Why extra lines just to trap and assign the variable before
using it?)

There are workarounds that sometimes benefit the code. In the case of
collections, like recordsets, you might be better off using for ... all
():

Then your controlled block efficiently does not happen if it saw no
records. "Efficiently" in terms of programmer complexity - the number
and meaning of lines that a programmer must comprehend.

And why can't Record.objects.get(pk='nonnumeric') return None?
Because, of course, deep inside it calls int(). I can't simplify the
calling code, and rely on garbage-in-None-out, because Python decided
which simplifications I should avoid with self-righteous indignation.

The Samurai Principle (return victorious, or not at all) is very
useful, sometimes. But other times it just prematurely depletes your
supply of Samurai...

 

[r0g]

Oh, you're in trouble now! If you think he gets upset at being told no,
you should see how upset he gets at being told he's in for a rough time!!!

*wink*

NO! It's a rude way to start a sentence don't you think? Just because
you're correcting someone doesn't mean you have to be combative and try
and make them feel small. Unless they've adopted a hostile or wilfully
ignorant tone there's no reason to be so brusqe with people. You can be
both nice AND terse you know.

I can't imagine why I expect good manners on usenet though, AFAICT it's
never been like that (well not since I got on it anyway).

Roger.

 

[Chris Rebert]

It's not about efficiency, it's about making assumptions for the
programmer about what kind of rigor they need.

Why can't int('nonnumeric') return None?

Errors should never pass silently.
Unless explicitly silenced.
-- The Zen of Python (http://www.python.org/dev/peps/pep-0020/)

Better to throw an exception and ensure the case is specifically dealt
with one way or another than to silently return an error flag result
which may only delay the error until later in the program, making it
harder to debug. Is it that much of a burden to write and use the
small function that does what you want?

def int_or_None(string):
try:
return int(string)
except ValueError:
return None

Heck, you can even write it inline and dispense with the function if you want:

try: foo = int(bar)
except ValueError: foo = None

Quibbling over a mere one more line of code (or writing one short
function) seems a bit petty.

(A related question - why can't I just go 'if record = method():  use
(record)'. Why extra lines just to trap and assign the variable before
using it?)

I believe that's disallowed so as to prevent the subtle bugs seen in C
code which result from when someone makes a typo and omits the second
"=" in their `if foo == bar():` test.

Cheers,
Chris

 

[r0g]

checking, since the function relies on the caller obeying the
contract[2] and never calling it with an invalid input.

DbC uses assertions[1] spuriously, unlike the traditional approach which
is much more conservative when using assertions.

[1] or explicit language support which is just syntax sugar for assertions
[2] of course, on a debug release, the contract validation code will
still be enforced to catch logic/consistency bugs that causes the violation

Thanks for the responses Steven/Dave/Lie, that's some really insightful
stuff

Roger.

 

[Phlip]

Errors should never pass silently.

You are saying I, as the programmer, cannot decide what is an error
and what is a pass-thru. The decision is made for me. (Yes yes I can
write int_or_None(), etc...)

Here's a super easy example:

{ 42: 'forty two' }.get(41, None)

Because I can supply a default, I can decide what is an error and what
is .

Now the equivalent in a language that does not enjoy this false "Zen":

{ 42: 'forty two' }[41] # returns None
{ 42: 'forty two' }.fetch(41, None) # ibid
{ 42: 'forty two' }.fetch(41) # raises an exception

The quicky validation is available if I _request_ it.

Quibbling over a mere one more line of code (or writing one short
function) seems a bit petty.

Because that "Zen of Python" is an empty sophistry that forces me to
add a "mere one more line of code" over and over again...

I believe that's disallowed so as to prevent the subtle bugs seen in C
code which result from when someone makes a typo and omits the second
"=" in their `if foo == bar():` test.

Don't prevent me from using a technique just because others had
trouble with it.

And if bar() == foo is the superior technique anyway, because the ==
happens in chronological and lexical order after the bar() call.

 

[Phlip]

Errors should never pass silently.

Unless explicitly silenced.
-- The Zen of Python (http://www.python.org/dev/peps/pep-0020/)

"The person who says it cannot be done should never interrupt the
person doing it"

 

[Chris Rebert]

You are saying I, as the programmer, cannot decide what is an error
and what is a pass-thru. The decision is made for me. (Yes yes I can
write int_or_None(), etc...)

No, you can certainly decide yourself. You merely need be explicit
about it; you're leaving out the other tandem half of the couplet:
"*Unless* explicitly silenced." Throwing exceptions is only the
default because it tends to lead to more reliable code in that you're
forced to deal with the error condition by either catching an
exception or preventing one from being thrown (by, say, using a
different method in the API that just returns a default value
instead).

As an aside, I would guess the built-in types don't have default value
parameters for conversions partly because it'd be a bit less elegant
to implement; since None is a commonly used default value, they'd have
to use a different sentinel as said parameter's default value to
indicate that the caller wanted an exception raised, and having a
hierarchy of nil values in the core language detracts from the
language's elegance.

At any rate, if you like dict.get(), I don't see why, say,
my_int('nonnumeric', None) should be such a problem.

Here's a super easy example:

 { 42: 'forty two' }.get(41, None)

Because I can supply a default, I can decide what is an error and what
is .

Exactly, that's explicitly silencing the error; no one ever said an
`except` clause was the only silencing mechanism. You're making your
intention clear by using .get() and passing in a desired default. I
agree this is a fine thing.

Now the equivalent in a language that does not enjoy this false "Zen":

 { 42: 'forty two' }[41]  # returns None
 { 42: 'forty two' }.fetch(41, None)  # ibid
 { 42: 'forty two' }.fetch(41)  # raises an exception

The quicky validation is available if I _request_ it.

Seems like rather "fast-and-loose" programming if you don't care about
validation enough of the time to merit it being default. If your
programming is indeed so fast-and-loose, I refer you to the recent
comment about PHP (and other less "exception-happy" languages).

Anyway, I do totally agree that you should, if feasible, be provided
an easy way to designate a common error-handling strategy (for
example, in this case, by using a default value via .fetch()).

However, go too loose on error handling and exceptions and one ends up
with something like JavaScript with its infamous `undefined` value
which can make debugging a nightmare (whoever came up with the idea of
JS's `undefined` should be slapped upside the head).

Cheers,
Chris

 

[Steven D'Aprano]

NO! It's a rude way to start a sentence don't you think?

No.

I'm not Japanese, I don't feel any social prohibition at saying No in
this context. I'm also happy to start a sentence with "You're wrong", and
occasionally I give in to temptation to start it with "Are you on crack?".

Just because
you're correcting someone doesn't mean you have to be combative and try
and make them feel small. Unless they've adopted a hostile or wilfully
ignorant tone there's no reason to be so brusqe with people. You can be
both nice AND terse you know.

Now I feel hurt that you're publicly rebuking me and making me feel as if
I've done something wrong by trying to lighten the mood with a small
joke...

Nah, just kidding.