Exception handling in Python 3.x

[Steven D'Aprano]

Consider the following common exception handling idiom:

def func(iterable):
it = iter(iterable)
try:
x = next(it)
except StopIteration:
raise ValueError("can't process empty iterable")
print(x)

The intention is:

* detect an empty iterator by catching StopIteration;
* if the iterator is empty, raise a ValueError;
* otherwise process the iterator.

Note that StopIteration is an internal detail of no relevance whatsoever
to the caller. Expose this is unnecessary at best and confusing at worst.


In Python 2.6 this idiom works as intended:

func([])

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in func
ValueError: can't process empty iterable

There is no sign of the StopIteration, and nor should there be.

But Python 3.1 changes this behaviour, exposing the unimportant
StopIteration and leading to a more complicated and confusing traceback:

func([])

Traceback (most recent call last):
File "<stdin>", line 4, in func
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in func
ValueError: can't process empty iterable


I understand the rational of this approach -- it is to assist in
debugging code where the except block is buggy and raises an error. But a
deliberate and explicit call to raise is not a buggy except block. It is
terribly inappropriate for the common use-case of catching one exception
and substituting another.

I note that the PEP explicitly notes this use-case, but merely sweeps it
under the carpet:

Open Issue: Suppressing Context
As written, this PEP makes it impossible to suppress '__context__',
since setting exc.__context__ to None in an 'except' or 'finally'
clause will only result in it being set again when exc is raised.

http://www.python.org/dev/peps/pep-3134/


Apart from this horrible idiom:

def func(iterable):
it = iter(iterable)
failed = False
try:
x = next(it)
except StopIteration:
failed = True
if failed:
raise ValueError("can't process empty iterable")
print(x)


or similar, is there really no way to avoid these chained exceptions?

 

[Hrvoje Niksic]

Consider the following common exception handling idiom:

def func(iterable):
it = iter(iterable)
try:
x = next(it)
except StopIteration:
raise ValueError("can't process empty iterable")
print(x)

Not exactly what you're looking for, but another way to express the
above is:

def func(iterable):
for x in iterable:
break
else:
raise ValueError("can't process empty iterable")
print(x)

Otherwise, I completely agree that being unable to completely replace
the original exception is an annoyance at best.

 

[Peter Otten]

Consider the following common exception handling idiom:

def func(iterable):
it = iter(iterable)
try:
x = next(it)
except StopIteration:
raise ValueError("can't process empty iterable")
print(x)

The intention is:

* detect an empty iterator by catching StopIteration;
* if the iterator is empty, raise a ValueError;
* otherwise process the iterator.

Note that StopIteration is an internal detail of no relevance whatsoever
to the caller. Expose this is unnecessary at best and confusing at worst.

http://mail.python.org/pipermail/python-list/2010-October/1258606.html
http://mail.python.org/pipermail/python-list/2010-October/1259024.html

 

[Hrvoje Niksic]

http://mail.python.org/pipermail/python-list/2010-October/1258606.html
http://mail.python.org/pipermail/python-list/2010-October/1259024.html

Both of these involve suppressing the chaining at the wrong place,
namely in the outer handler or, worse yet, in the exception display
mechanism. Steven, on the other hand, wants his *inner* handler to
express that the original exception was an implementation detail, a
business exception such as StopIteration, that is completely irrelevant
to the actual exception being raised. The outer handler is the wrong
place to suppress the chaining because it has no way of distinguishing
Steven's case from a genuine case of a new exception unexpectedly
occurring during handling of the original exception.

One solution would be for "raise" inside except to not use the context.
For example:

try:
{}[1]
except KeyError:
1/0

would behave as before, but:

But:

try:
{}[1]
except KeyError:
raise Exception("my error")

....would raise the custom error forgetting the KeyError.

 

[Peter Otten]

Both of these involve suppressing the chaining at the wrong place,
namely in the outer handler or, worse yet, in the exception display
mechanism. Steven, on the other hand, wants his *inner* handler to
express that the original exception was an implementation detail, a
business exception such as StopIteration, that is completely irrelevant
to the actual exception being raised. The outer handler is the wrong
place to suppress the chaining because it has no way of distinguishing
Steven's case from a genuine case of a new exception unexpectedly
occurring during handling of the original exception.

To quote the Rolling Stones: You can't always get what you want.

After rereading the original post I still don't get why the workarounds
provided in those links aren't worth considering.

Peter

 

[Paul Rubin]

def func(iterable):
it = iter(iterable)
failed = False
try:
x = next(it)
except StopIteration:
failed = True
if failed:
raise ValueError("can't process empty iterable")
print(x)

Untested:

from itertools import islice

def func(iterable):
xs = list(islice(iter(iterable), 1))
if len(xs) == 0:
raise ValueError(...)
print xs[0]

 

[Ethan Furman]

> http://mail.python.org/pipermail/python-list/2010-October/1258606.html
http://mail.python.org/pipermail/python-list/2010-October/1259024.html

I found #6210 on bugs.python.org -- does anyone know if there are any
others regarding this issue? Or any progress on MRAB's idea?

> Suggestion: an explicit 'raise' in the exception handler excludes the
> context, but if you want to include it then 'raise with'. For example:
>
> # Exclude the context
> try:
> command_dict[command]()
> except KeyError:
> raise CommandError("Unknown command")
>
> # Include the context
> try:
> command_dict[command]()
> except KeyError:
> raise with CommandError("Unknown command")

~Ethan~

 

[Ethan Furman]

To quote the Rolling Stones: You can't always get what you want.

After rereading the original post I still don't get why the workarounds
provided in those links aren't worth considering.

For me at least it's a matter of simplicity, clarity, and the Way of the
Python

The workarounds are boiler-plate for a fairly common situation, and one
of the things i _love_ about python is the *lack* of boilerplate.

I think the real question is is there any progress on dealing with the
Open Issue in the PEP?

Open Issue: Suppressing Context
As written, this PEP makes it impossible to suppress '__context__',
since setting exc.__context__ to None in an 'except' or 'finally'
clause will only result in it being set again when exc is raised.

http://www.python.org/dev/peps/pep-3134/

~Ethan~

 

[Steven D'Aprano]

def func(iterable):
it = iter(iterable)
failed = False
try:
x = next(it)
except StopIteration:
failed = True
if failed:
raise ValueError("can't process empty iterable")
print(x)

Untested:

from itertools import islice

def func(iterable):
xs = list(islice(iter(iterable), 1))
if len(xs) == 0:
raise ValueError(...)
print xs[0]

If you're intention was to make me feel better about the version above
that sets a flag, you succeeded admirably!

 

[Steven D'Aprano]

Thanks for the links Peter.

Both of these involve suppressing the chaining at the wrong place,
namely in the outer handler or, worse yet, in the exception display
mechanism. Steven, on the other hand, wants his *inner* handler to
express that the original exception was an implementation detail, a
business exception such as StopIteration, that is completely irrelevant
to the actual exception being raised.

Yes, exactly! Python 3.x exposes completely irrelevant and internal
details in the traceback.

The outer handler is the wrong
place to suppress the chaining because it has no way of distinguishing
Steven's case from a genuine case of a new exception unexpectedly
occurring during handling of the original exception.

One solution would be for "raise" inside except to not use the context.

I would have thought that was such an obvious solution that I was
gobsmacked to discover the PEP 3134 hadn't already considered it. If you
*explicitly* raise an exception inside an exception handler, surely it's
because you want to suppress the previous exception as an internal detail?

If not, and you want to chain it with the previous exception, the
solution is simple, obvious and straight-forward: explicit chaining.

try:
something()
except SomeException as exc:
raise MyException from exc

For example:

try:
{}[1]
except KeyError:
1/0

would behave as before, but:

Yes, that presumably would be a bug and should chain exceptions.

But:

try:
{}[1]
except KeyError:
raise Exception("my error")

...would raise the custom error forgetting the KeyError.

That's exactly the behaviour I would expect and I'm surprised that this
feature was put into production without some simple way to support this
idiom.

 

[Steven D'Aprano]

After rereading the original post I still don't get why the workarounds
provided in those links aren't worth considering.

The first work-around:

http://mail.python.org/pipermail/python-list/2010-October/1258606.html

is unsuitable because it requires the caller to install a custom
excepthook. It would be rude and unacceptable for arbitrary functions to
install hooks, possibly stomping all over the caller's own custom
excepthook. And even if I did, or the caller did, it has the unfortunate
side-effect of suppressing the display of *all* chained exceptions,
including those that come from the bugs in exception handlers.


The second work-around might be worth considering:

http://mail.python.org/pipermail/python-list/2010-October/1259024.html

however it adds unnecessary boilerplate to what should be a simple
try...except...raise block, it obscures the intention of the code. As a
work-around, it might be worth considering, but it's hardly elegant and
it could very well be a fluke of the implementation rather than a
guaranteed promise of the language.


In the absence of a supported way to suppress exception chaining, I'm
leaning towards my original work-around: set a flag in the except block,
then raise the exception once I leave the block.

But thanks again for the links.

 

[John Nagle]

Consider the following common exception handling idiom:

def func(iterable):
it = iter(iterable)
try:
x = next(it)
except StopIteration:
raise ValueError("can't process empty iterable")
print(x)

The intention is:

* detect an empty iterator by catching StopIteration;
* if the iterator is empty, raise a ValueError;
* otherwise process the iterator.

Note that StopIteration is an internal detail of no relevance whatsoever
to the caller. Expose this is unnecessary at best and confusing at worst.

Right. You're not entitled to assume that StopIteration is
how a generator exits. That's a CPyton thing; generators were
a retrofit, and that's how they were hacked in. Other implementations
may do generators differently.

John Nagle

 

[Paul Rubin]

Apart from this horrible idiom:

def func(iterable):
it = iter(iterable)
failed = False
try:
x = next(it)
except StopIteration:
failed = True
if failed:
raise ValueError("can't process empty iterable")
print(x)


or similar, is there really no way to avoid these chained exceptions?

Seems like yet another example of people doing messy things with
exceptions that can easily be done with iterators and itertools:

from itertools import islice

def func(iterable):
xs = list(islice(iter(iterable), 1))
if len(xs) == 0:
raise ValueError(...)
print xs[0]

It's really unfortunate, though, that Python 3 didn't offer a way to
peek at the next element of an iterable and test emptiness directly.

 

[Mark Wooding]

Right. You're not entitled to assume that StopIteration is how a
generator exits. That's a CPyton thing; generators were a retrofit,
and that's how they were hacked in. Other implementations may do
generators differently.

This is simply wrong. The StopIteration exception is a clear part of
the generator protocol as described in 5.2.8 of the language reference;
the language reference also refers to 3.5 of the library reference,
which describes the iterator protocol (note, not the generator
implementation -- all iterators work the same way), and explicitly
mentions StopIteration as part of the protocol.

-- [mdw]

 

[Steven D'Aprano]

It's really unfortunate, though, that Python 3 didn't offer a way to
peek at the next element of an iterable and test emptiness directly.

This idea of peekable iterables just won't die, despite the obvious flaws
in the idea.

There's no general way of telling whether or not a lazy sequence is done
except to actually generate the next value, and caching that value is not
appropriate for all such sequences since it could depend on factors which
have changed between the call to peek and the call to next.

If you want to implement a peek method in your own iterables, go right
ahead. But you can't make arbitrary iterables peekable without making a
significant class of them buggy.

 

[MRAB]

This idea of peekable iterables just won't die, despite the obvious flaws
in the idea.

There's no general way of telling whether or not a lazy sequence is done
except to actually generate the next value, and caching that value is not
appropriate for all such sequences since it could depend on factors which
have changed between the call to peek and the call to next.

If you want to implement a peek method in your own iterables, go right
ahead. But you can't make arbitrary iterables peekable without making a
significant class of them buggy.

Perhaps Python could use Guido's time machine to check whether the
sequence will yield another object in the future.

 

[John Nagle]

Right. You're not entitled to assume that StopIteration is how a
generator exits. That's a CPyton thing; generators were a retrofit,
and that's how they were hacked in. Other implementations may do
generators differently.

This is simply wrong. The StopIteration exception is a clear part of
the generator protocol as described in 5.2.8 of the language reference;
the language reference also refers to 3.5 of the library reference,
which describes the iterator protocol (note, not the generator
implementation -- all iterators work the same way), and explicitly
mentions StopIteration as part of the protocol.

-- [mdw]

PEP 255, like too much Python literature, doesn't distinguish clearly
between the language definition and implementation detail. It says
"The mechanics of StopIteration are low-level details, much like the
mechanics of IndexError in Python 2.1". Applications shouldn't be
explicitly using StopIteration.

IronPython doesn't do StopIteration the same way CPython does.

http://ironpython.codeplex.com/wikipage?title=IPy1.0.xCPyDifferences

Neither does Shed Skin.

John Nagle

 

[John Nagle]

This idea of peekable iterables just won't die, despite the obvious flaws
in the idea.

There's no general way of telling whether or not a lazy sequence is done
except to actually generate the next value, and caching that value is not
appropriate for all such sequences since it could depend on factors which
have changed between the call to peek and the call to next.

Right.

Pascal had the predicates "eoln(file)" and "eof(file)", which
were tests for end of line and end of file made before reading. This
caused much grief with interactive input, because the test would
stall waiting for the user to type something. Wirth originally
intended Pascal for batch jobs, and his version didn't translate
well to interactive use. (Wirth fell in love with his original
recursive-descent compiler, which was simple but limited. He
hated to have language features that didn't fit his compiler model
well. This held the language back and eventually killed it.)

C I/O returned a unique value on EOF, but there was no way to
test for it before reading. Works much better. The same issues apply
to pipes, sockets, qeueues, interprocess communication, etc.

John Nagle

 

[Dennis Lee Bieber]

Pascal had the predicates "eoln(file)" and "eof(file)", which
were tests for end of line and end of file made before reading. This

As I recall the documentation, the Pascal I/O system by default did
a one-element "pre-read" on opening a "file". One would have to copy the
element before incrementing to the next...

caused much grief with interactive input, because the test would
stall waiting for the user to type something. Wirth originally

And yes, most implementations did have to play games with regards to
interactive terminals.

intended Pascal for batch jobs, and his version didn't translate
well to interactive use. (Wirth fell in love with his original
recursive-descent compiler, which was simple but limited. He
hated to have language features that didn't fit his compiler model
well. This held the language back and eventually killed it.)

Pascal was designed as a teaching language; I don't think Wirth ever
expected it to become a business application language. <G>

 

[Mark Wooding]

PEP 255, like too much Python literature, doesn't distinguish
clearly between the language definition and implementation detail. It
says "The mechanics of StopIteration are low-level details, much like
the mechanics of IndexError in Python 2.1". Applications shouldn't be
explicitly using StopIteration.

You've twisted the words by quoting them out of context, and have
attempted to force a misinterpretation of `low-level details' as
`implementation detail'.

That text comes from a question-and-answer section, in response to the
question `why not force termination to be spelled "StopIteration"?'.
This is a fine answer to the question: the details of the (preexisting
-- see PEP 234) iteration protocol are abstracted by the generator
syntax. But it doesn't at all mean that the StopIteration exception
isn't an official, use-visible part of Python.

IronPython doesn't do StopIteration the same way CPython does.

http://ironpython.codeplex.com/wikipage?title=IPy1.0.xCPyDifferences

IronPython's behaviour when you try to fetch items from a spent
generator is different. It still implements the same iterator protocol,
and raises StopIteration when it has no more items to yield.

You're not stupid, but you'd have to be in order to think that these
references support your claim that

I don't want to conclude that you're not arguing in good faith but I'm
not seeing many other possibilities.

-- [mdw]