cascading python executions only if return code is 0

[Steven D'Aprano]

I don't understand what you're trying to say there.

I'm trying to say that code that relies on side-effects is usually a sign of
poor design. Is that more clear now?

I'm not a functional programming zealot, but imperative programming that
relies on side-effects is often harder to reason about than functional
style, due to lack of locality in its effects. With functions that
communicate only through their input arguments and output result, you don't
have to look far to see the effects the function call has: it is *only* in
the return result. But if it has side-effects, you potentially have to
inspect the entire program and environment to see what it has done.

Now of course sometimes the whole point of the function is to have a
side-effect (print something, delete or save a file, start up the car's
engine, ...) and even functional languages usually have some facility for
side-effects. And we can often limit the harm of relying on side-effects
but narrowly constraining what those side-effects are.

E.g. list.append has a very narrow and well-defined side-effect, which makes
it relatively easy to reason about it. But still not as easy as perhaps we
would like:

alist = blist = [1, 2, 4, 8]
# later on
alist.append(16) # operates by side-effect
# and still later on
assert blist == [1, 2, 4, 8] # FAILS!!!


A side-effect free language might make list.append return a new list with
the extra value appended, and then the above would not occur. But I
digress.

The point is, I'm not saying that imperative code that operates via
side-effects is always harmful. There are degrees of badness.

A bit later in your
post, you wrote:

try:
a()
b()
c()
except SomeError:
handle_error()

Clearly, since the return values of a(), b(), and c() aren't saved, the
only reason they're getting called is for their side effects.

That's not my design

And I don't see anything wrong with that.

And quite frankly, neither do I. But I don't know what a, b and c actually
do.

BTW, there's a pattern we use a bunch in the Songza server code, which
is sort of this, but in reverse. We'll have a bunch of possible ways to
do something (strategies, to use the pattern vernacular), and want to
try them all in order until we find one which works.

Sounds reasonable. You're not operating by side-effect, since you actually
do want the result generated by the strategy. Presumably the strategy
signature is to return None on failure, or instance on success. (I see
below that's exactly what you do.)

So, for example:

classes = [ClientDebugPicker,
StatefulSongPicker,
SWS_SequentialSongPicker,
StandardSongPicker]
for cls in classes:
picker = cls.create(radio_session, station, artist)
if picker:
return picker

This seems perfectly reasonable up. The strategy either returns a picker, in
which case you are done, or it returns None and you continue. No
side-effects are involved.

else:
assert 0, "can't create picker (classes = %s)" % classes

¡Ay, caramba! I was with you until the very last line. The above code is
possibly buggy and inappropriately designed. (I may be completely
misinterpreting this, in which case feel free to ignore the following
rant.)

First, the bug: there are circumstances where no exception is raised even if
all the strategies fail. (If the StandardSongPicker is guaranteed to
succeed, then obviously my analysis is wrong.) In this case, then any code
following the for-else will execute. Since this is in a function, and there
doesn't seem to be any following code, that means that your function will
return None instead of a valid picker. Does the rest of your code check for
None before using the picker? If not, you have a bug waiting to bite.

Second, the poor design. When it works as expected, failure is indicated by
AssertionError. Why AssertionError? Why not ImportError, or
UnicodeEncodeError, or ZeroDivisionError, or any other of the dozens of
inappropriate errors that the code could (but shouldn't) raise? I guess
nobody here would write code that looks like this by design:

try:
picker = select_picker(radio_session, station, artist)
except EOFError:
handle_no_picker_case()


just because "raise EOFError" required less typing than raising some more
appropriate exception. So why would we write:

try:
picker = select_picker(radio_session, station, artist)
except AssertionError:
handle_no_picker_case()


just because "assert" required less typing than raising some more
appropriate exception? assert is not a short-cut for "raise
SomeGenericException". AssertionError has a specific meaning, and it is
sloppy to misuse it. An assertion error means that an assertion has failed,
and assertions only fail when your code contains a logic error or a program
invariant is violated. Both should never happen in production code, and if
they do they ought to be unrecoverable errors.

Anyway, I may be completely misinterpreting what I'm reading. Perhaps the
assertion is checking a function invariant ("one of the strategies will
always succeed") in which case you're doing it exactly right and I should
shut up now

 

[Chris Angelico]

Anyway, I may be completely misinterpreting what I'm reading. Perhaps the
assertion is checking a function invariant ("one of the strategies will
always succeed") in which case you're doing it exactly right and I should
shut up now

Or maybe it has a lengthy piece of error trapping code that wasn't
germane to the discussion, and for brevity was squished down to a
quick little assert

ChrisA

 

[Dennis Lee Bieber]

Now, I do take the point that these functions seem to take the
unix-exit-code convention that zero is success (leaving the many
values of "non-zero" to indicate flavours of failure as desired,
as we have many types of exceptions).

Having spent 22 years with VMS, 0 -> success is still a problem to me.
Odd result codes (aka True) were 1-success/3-information, even results
(False) were 0-warning/2-error/4-fatal

"""
Each condition code corresponds to a system message. The command
interpreter saves the condition code as a 32-bit longword defined as the
reserved global symbol $STATUS. The condition code stored in $STATUS is a
hexadecimal number conforming to the format of an OpenVMS message code:

Bits 0--2 contain the severity level of the message.
Bits 3--15 contain the number of the corresponding message.
Bits 16--27 contain a number for the software component, or facility,
that generated the message.
Bits 28--31 contain internal control flags.
"""

(Boolean testing only examined bit 0)

 

[Chris Angelico]

Having spent 22 years with VMS, 0 -> success is still a problem to me.
Odd result codes (aka True) were 1-success/3-information, even results
(False) were 0-warning/2-error/4-fatal

Having spent a similar amount of time with Unix, C, and various APIs,
I'm quite used to 0 meaning success and nonzero meaning error.

ChrisA

 

[Mark Lawrence]

Having spent a similar amount of time with Unix, C, and various APIs,
I'm quite used to 0 meaning success and nonzero meaning error.

ChrisA

Another C thing to complain about, with functions like malloc the status
code and value returned are one and the same thing, except that NULL is
failure in this case.

 

[Chris Angelico]

Another C thing to complain about, with functions like malloc the status
code and value returned are one and the same thing, except that NULL is
failure in this case.

How's that a problem? Python has the same:

memory.get(1234)

will return a bit of memory, or None if it can't get any. Not
materially different. Remember, C doesn't have exceptions. In C++, the
'new' operator will throw an exception if it can't provide memory.

ChrisA

 

[Mark Lawrence]

How's that a problem? Python has the same:

Remembering to check it?

memory.get(1234)

You learn something new every day or April 1st come early?

 

[Chris Angelico]

You learn something new every day or April 1st come early?

memory = {1:"Foo", 12:"Bar", 123:"Quux"}

ChrisA

 

[Roy Smith]

Anyway, I may be completely misinterpreting what I'm reading. Perhaps the
assertion is checking a function invariant ("one of the strategies will
always succeed") in which case you're doing it exactly right and I should
shut up now

Yes

More specifically, the assertion exception will get caught way up in
some django middleware which will log a stack trace and return a HTTP
50-something. This will typically be followed by somebody like me
noticing the stack dump and trying to figure out WTF happened.

Assertions are great tools. People should use them more often. In a
sense, they're executable comments. They're a programmer's way of
standing on a hilltop and shouting to all the world, "I swear to you,
this is true. There may be a gazillion lines of code out there and
GBytes of program state, but right here, right now, within this circle
I've drawn in the sand, I know this to be true, and you can depend on
that. You can make it part of the foundation on which you begin to
reason about program behavior. Furthermore, if it turns out not to be
true, you don't have to worry about figuring out who's fault it is. I
hereby declare that it's my fault".

 

[Chris Angelico]

More specifically, the assertion exception will get caught way up in
some django middleware which will log a stack trace and return a HTTP
50-something. This will typically be followed by somebody like me
noticing the stack dump and trying to figure out WTF happened.

Assertions are great tools. People should use them more often.

You do have to be careful though, because they can be compiled out. If
it really is a "can't happen", then sure, but encouraging people to
use them for potentially obscure cases may be dangerous - if you never
happen to hit on it during development and then run with assertions
disabled, you won't see that stack trace.

However, they're still a lot more executable than other forms of comment

ChrisA

 

[Ned Batchelder]

Even IF you are presented with a small snippet of code that
include the identifier for the return value, you still
cannot be certain that extrapolating meaning from an
identifier*alone* will give you the insight to remove all
ambiguity.

For example, let's say the return value is "proceed" and
attached to name spelled "flag". Do you really think you can
extrapolate the purpose of that value being returned?

"proceed" with what?

Proceed counting unicorns?
Proceed raising exceptions?
Proceed extrapolating to infinity and beyond?!

You can guess, but that is all. Which brings us full circle
to the problem at hand.

You seem to be arguing that the original question didn't have enough
information to be usefully answered. You are doing this after the OP
got an answer he was pleased with.

Yes, the question had some ambiguity. That's why other people engaged
with him to get to a useful point. Waxing philosophically about the
impossibility of asking the question isn't helpful.

 

[Ethan Furman]

else:
assert 0, "can't create picker (classes = %s)" % classes

¡Ay, caramba! I was with you until the very last line. The above code is
possibly buggy and inappropriately designed. [...]

First, the bug: there are circumstances where no exception is raised even if
all the strategies fail.

Since Steven wasn't explicit about the circumstances, I'll list the two I'm aware of: if Python is started with -O or
-OO then assertions are cut from the code and do not run:

ethan@media:~/source/python/issue19995$ ./python -O
Python 3.4.0b1 (default:be22ffb4fdf1, Dec 20 2013, 12:26:10)
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
--> assert 0, "Testing"
--> # no exception raised

 

[Mark Lawrence]

[snip molehill turned into Himalayas]

Again Frank's original question.

"
I have a requirement where I need to sequentially execute a bunch of
executions, each execution has a return code. the followed executions
should only be executed if the return code is 0. is there a cleaner or
more pythonic way to do this other than the following ?

if a() == 0:
if b() == 0:
c()
"

What is so difficult to understand about "is there a cleaner or more
pythonic way to do this other than the following?" Peter Otten for one
certainly managed to get it, and as always managed to beat me to the draw

 

[Ethan Furman]

Yes

More specifically, the assertion exception will get caught way up in
some django middleware which will log a stack trace and return a HTTP
50-something. This will typically be followed by somebody like me
noticing the stack dump and trying to figure out WTF happened.

This is completely misusing what assertions are for. I hope this bit of middleware (or django itself) is very clear
about never running with assertions turned off.

Assertions are great tools.

Only when used properly.

People should use them more often.

I see them being (mis)used too much as it is.

In a
sense, they're executable comments. They're a programmer's way of
standing on a hilltop and shouting to all the world, "I swear to you,
this is true. There may be a gazillion lines of code out there and
GBytes of program state, but right here, right now, within this circle
I've drawn in the sand,

Considering how easy it is to disable assertions, a circle in the sand is an amazingly appropriate metaphor.

 

[Mark Lawrence]

Considering how easy it is to disable assertions, a circle in the sand
is an amazingly appropriate metaphor.

It might be easy, but surely it's far more fun providing an itertools
solution

 

[Roy Smith]

This is completely misusing what assertions are for. I hope this bit of
middleware (or django itself) is very clear
about never running with assertions turned off.

Sigh. Sometimes I'm not sure which is worse. The anti-assertion
zealotry on this list, or the anti-regex zealotry.

The use of an assertion here is perfectly reasonable. It is a statement
that "this can never happen".

The bit of middleware I was talking about catches ALL uncaught
exceptions. We design our code so that all exceptions are supposed to
get caught and handled at some appropriate place in application code.
Nothing is ever supposed to leak up to the point where that middleware
catches it, and anything caught there is evidence of a bug. This is a
common pattern in any kind of long-lived server.

And, if we didn't have the piece of middleware, the assertion (or any
other uncaught exception) would get caught at some deeper level inside
the django core. The result would still be a 500 HTTP response, but
without the added diagnostic logging we do, so it would be a lot harder
to understand what happened.

And, if django didn't have that "except Exception" block wrapping
everything, the gunicorn process would exit and upstart would catch that
and restart it.

And, yes, I know that assertions get turned off with -O (frankly, I
think that's a design flaw). We don't run with -O.

 

[Dennis Lee Bieber]

Roy Smith wrote:



¡Ay, caramba! I was with you until the very last line. The above code is
possibly buggy and inappropriately designed. (I may be completely
misinterpreting this, in which case feel free to ignore the following
rant.)

I'd think the biggest problem with this is that if one runs Python with
optimization turned on, the assert statement itself vanishes, leaving one
with an empty else clause...

Try debugging that problem!

 

[Mark Lawrence]

I'd think the biggest problem with this is that if one runs Python with
optimization turned on, the assert statement itself vanishes, leaving one
with an empty else clause...

Try debugging that problem!

I'll pass on that one

 

[Ethan Furman]

Sigh. Sometimes I'm not sure which is worse. The anti-assertion
zealotry on this list, or the anti-regex zealotry.

I am not a zealot (I'm not! Really!! . I just find it alarming to have major pieces of software rely on a feature
that can be so easily tuned out, and it wasn't clear from your comment that it was /any/ exception.

Mostly I don't want newbies thinking "Hey! I can use assertions for all my confidence testing!"

Just as one data point OpenERP, which has a lot of good features, unfortunately uses assert to test user input.

 

[Steven D'Aprano]

Sigh.  Sometimes I'm not sure which is worse.  The anti-assertion
zealotry on this list, or the anti-regex zealotry.

I'm not anti-assertions. I love assertions. I wouldn't be surprised if I use
assert more than you do. What I dislike is people misusing assertions.

And, yes, I know that assertions get turned off with -O (frankly, I
think that's a design flaw).  We don't run with -O.

Until such time as somebody decides they can speed up your code by 5% by
running with optimizations turned on.

Unfortunately, it's not always clear when an assert is appropriate and when
it isn't. Fundamentally, if an assertion can never fail[1], then there's no
point testing for it:

x = []
assert x == [] and len(x) == 0

So there's always tension between "why am I testing something that can't
fail?" and "but what if it does?", and different people are going to draw
the line in different places. So even if you and I draw the line in
different places, I can acknowledge that you seem to be using asserts the
same way I would, as executable comments and for testing program
invariants, and I'm sorry that my rant was aimed in your direction
unnecessarily.





[1] Short of hardware failure or serious bugs in the compiler, but in both
those cases you can't trust the assertions to trigger correctly either.