Attack a sacred Python Cow

[Russ P.]

Sis, boom, rah rah rah!

You're kind of skipping over the point where three other people had the
same misunderstanding about your original statement and correction, so

Another whopper, but who's counting?

maybe the reader is not the problem but rather the writer, but hey,
don't let that get in the way of a good public shitfit.

You're winning!

And you're a professional of some sort? Man, I can't even imagine
working in an environment with people like you. I guess I'm pretty
lucky to work with real professionals who don't play petty little
games like the one you played here -- and are still playing. Go ahead,
have the last word, loser -- then get lost.

 

[Iain King]

Another whopper, but who's counting?



And you're a professional of some sort? Man, I can't even imagine
working in an environment with people like you. I guess I'm pretty
lucky to work with real professionals who don't play petty little
games like the one you played here -- and are still playing. Go ahead,
have the last word, loser -- then get lost.

You understand this is usenet, right? Where we can all read the
entire thread? So trying to spin the situation just doesn't work?
Just checking...

Iain

 

[Chris Mellon]

I would accept as "evidence" something that satisfies my criteria,
which your example did not: it could have easily (and more robustly)
been written with a simple explicit test. I am looking for one that
can't.

You keep bringing up this notion of "more complex with no benefit",
which I'm simply not interested in talking about that at this time,
and I won't respond to any of your points. I am seeking the answer to
one question: whether "if x" can usefully do something a simple
explicit test can't. Everyone already knows that "if x" requires
fewer keystrokes and parses to fewer nodes.

I'm really not sure where you're going with this or what you're trying
to prove. "if x" is a duck-type test for a boolean value. Obviously if
you know the type and want a more *specific* test, then you can use an
explicit one. Any time you don't know or don't care about a more
specific type than "something which probably is boolean true", or any
time where you know the boolean semantics of the type and want to drop
some typing, you can use "if x". The more specific test is what you'd
use if you want more specific results. What's complicated about this
idea?

 

[Matthew Fitzgibbons]

If you're expecting a list (and only a list)
then your point makes sense. 'if x' can get you into trouble if you
_don't_ want its polymorphism.

"if x" is hardly unique in that way. If you're expecting a list, and only
a list, "len(x) != 0" will get you in trouble if somebody passes a string
or a dictionary. I don't see any reason why we should single out "if x"
as dangerous in the face of invalid types. With the exception of the "is"
and "is not" operators, nothing in Python is guaranteed to work with any
imaginable object. Even print can fail, if the object's __str__ method
raises an exception.

Although, if my function is expecting a list, my preference is to do:

if not isinstance(x, list):
raise SomeMeaningfulException()
# do stuff with the list

I put my type checking at the top of the function, so readers can
reference it easily.

And thus you break duck-typing and upset anybody who wants to pass a
sequence that doesn't inherit directly from list.

There are other (and arguably better, although more labour-intensive)
techniques for defensive programming that don't break duck-typing. You
can google for Look Before You Leap and Easier To Ask Forgiveness Than
Permission for more information. Alex Martelli has a fine recipe in the
Python Cookbook -- search for the recipe "Checking if an object has the
necessary attributes".

But in a nutshell, here's a toy example:

def spam(seq):
try:
seq.append
seq.extend
seq[0] = seq[0]
except Exception:
raise TypeError("argument isn't sufficiently sequence-like")
# No exceptions expected from here on
seq.append(seq[0])
seq.extend([1,2,3])
seq[0] = "spam"

Yes, I know it breaks duck typing, which is why I do it only very
rarely, and never with e.g. sequence types. If I use ifinstance for type
checking, it's because I need some _very_ specific class, almost always
one that I wrote. My more usual use case for ifinstance is to figure out
how to handle a particular object using introspection.

That said, your example is still helpful. It's a good approach, and I
should use it more often. Although I'd also lean toward hasattr and
iscallable and ordinary ifs instead of the try ... except where
possible. Depending on what you put in the try ... except, you risk
masking legit exceptions.

Ahhh, Programing. Where no rule of thumb seems to last five minutes.

-Matt

 

[Matthew Fitzgibbons]

"if x" is hardly unique in that way. If you're expecting a list, and only
a list, "len(x) != 0" will get you in trouble if somebody passes a string
or a dictionary. I don't see any reason why we should single out "if x"
as dangerous in the face of invalid types. With the exception of the "is"
and "is not" operators, nothing in Python is guaranteed to work with any
imaginable object. Even print can fail, if the object's __str__ method
raises an exception.

Forgot respond to this point. I heartily agree. I singled out 'if x'
only because that's the specific example under scrutiny. I think you and
I are pretty much in agreement.

-Matt

 

[Matthew Fitzgibbons]

If you're expecting a list (and only a list)
then your point makes sense. 'if x' can get you into trouble if you
_don't_ want its polymorphism.

"if x" is hardly unique in that way. If you're expecting a list, and
only a list, "len(x) != 0" will get you in trouble if somebody passes
a string or a dictionary. I don't see any reason why we should single
out "if x" as dangerous in the face of invalid types. With the
exception of the "is" and "is not" operators, nothing in Python is
guaranteed to work with any imaginable object. Even print can fail, if
the object's __str__ method raises an exception.

Although, if my function is expecting a list, my preference is to do:

if not isinstance(x, list):
raise SomeMeaningfulException()
# do stuff with the list

I put my type checking at the top of the function, so readers can
reference it easily.

And thus you break duck-typing and upset anybody who wants to pass a
sequence that doesn't inherit directly from list.

There are other (and arguably better, although more labour-intensive)
techniques for defensive programming that don't break duck-typing. You
can google for Look Before You Leap and Easier To Ask Forgiveness Than
Permission for more information. Alex Martelli has a fine recipe in
the Python Cookbook -- search for the recipe "Checking if an object
has the necessary attributes".

But in a nutshell, here's a toy example:

def spam(seq):
try:
seq.append
seq.extend
seq[0] = seq[0]
except Exception:
raise TypeError("argument isn't sufficiently sequence-like")
# No exceptions expected from here on
seq.append(seq[0])
seq.extend([1,2,3])
seq[0] = "spam"

Yes, I know it breaks duck typing, which is why I do it only very
rarely, and never with e.g. sequence types. If I use ifinstance for type
checking, it's because I need some _very_ specific class, almost always
one that I wrote. My more usual use case for ifinstance is to figure out
how to handle a particular object using introspection.

That said, your example is still helpful. It's a good approach, and I
should use it more often. Although I'd also lean toward hasattr and
iscallable and ordinary ifs instead of the try ... except where
possible. Depending on what you put in the try ... except, you risk
masking legit exceptions.

Ahhh, Programing. Where no rule of thumb seems to last five minutes.

-Matt

Where by ifinstance I mean isinstance and by iscallable I mean callable.

*hides*

-Matt

 

[Matthew Woodcraft]

How do you propose telling whether an iterator is empty?
That's a generic problem with any sort of lazy function. You don't know
if it has finished unless you try grabbing data from it.

Of course.

The point is that if you tell people that "if x" is the standard way to
check for emptiness, and also support a general principle along the
lines of "write your function using the interfaces you expect, and call
it using the object you have", you should expect to end up with bugs of
this sort.

AFAIK there's no great solution to this problem. It's inherent in the way
lazy functions work. Certainly you can't replace the call to "if widgets"
with "if len(widgets)", because iterators don't have a length.

I'm not a fan of len() for testing emptiness. But it would have been
better in this case, because it would have converted a hard-to-find
performance bug into an obvious exception.

However, there is a good (but not great) solution:

def frob(widgets, power):
widgets = iter(widgets) # in case widgets is a sequence
try:
first = widgets.next()
except StopIteration:
# empty iterator, nothing to do
return None
frobber = Frobber(power) # expensive call
frobber.frob(widget)
for widget in widgets:
frobber.frob(widget)

I would use the following low-tech way:

def frob(widgets, power):
frobber = None
for widget in widgets:
if frobber is None:
frobber = Frobber(power)
frobber.frob(widget)

-M-

 

[Steven D'Aprano]

Of course.

The point is that if you tell people that "if x" is the standard way to
check for emptiness, and also support a general principle along the
lines of "write your function using the interfaces you expect, and call
it using the object you have", you should expect to end up with bugs of
this sort.

I'm not sure that an occasional performance hit justifies calling it a
bug. I suppose you might come up with a scenario or two where it really
is a problem, but then I'm also free to imagine scenarios where calling
len(obj) takes a massive performance hit, or has irreversible side
effects.


But note that the code still does the right thing even if widgets is
empty. The only problem is that it needlessly calls Frobber. Frankly,
that would have to be *really* expensive before I would even bother using
the more complicated code. This is a good example of premature
optimization.

I'm not a fan of len() for testing emptiness. But it would have been
better in this case, because it would have converted a hard-to-find
performance bug into an obvious exception.

At the cost of wrongly raising a exception for perfectly good arguments,
namely non-empty iterators.

 

[Paul McGuire]

Nothing glues a community together so well as a common enemy. Or even
better: two enemies i.e. Perl and Java in Pythons case. On the other
hand, some enemies have to be ignored or declared to be not an enemy
( Ruby ), although oneself is clearly an enemy for them. The same
antisymmetry holds for Python and Java. Java is an enemy for Python
but Python is not worth for Java to be an enemy as long as it can be
ignored. C++ and Java are enemies for each other. Same holds for Java
and C#.- Hide quoted text -

- Show quoted text -

Help... being... sucked into... black hole... inside a... Klein...
bottle...

 

[Matthew Woodcraft]

I'm not sure that an occasional performance hit justifies calling it a
bug. I suppose you might come up with a scenario or two where it really
is a problem, but then I'm also free to imagine scenarios where calling
len(obj) takes a massive performance hit, or has irreversible side
effects.

Of course performance problems can be bugs! And it was explicit in my
example that this was supposed to be such a case.

But here, have an example where it's a non-performance bug:

def load_widgets(self, widgets):
if not widgets:
raise ValueError("no widgets specified")
self._prepare_for_widgets()
for widget in widgets:
self._load_widget(widget)
self._is_widgetty = True

This way you can end up with an object in an invalid state.

At the cost of wrongly raising a exception for perfectly good arguments,
namely non-empty iterators.

Sure, which will be fixed when the code passing an iterator is first
tested, whether the iterator is empty or not. It's the errors which pass
silently that are most dangerous.


As I said, I'm not a fan of len() for testing emptiness. But "if x"
isn't good either. Often I end up rearranging things so I can get rid of
the explicit test altogether, but there can be a cost in readability.

I think any Python-like language would be better off with an explicit
way of asking 'is this container empty?'.

-M-

 

[Carl Banks]

I'm really not sure where you're going with this or what you're trying
to prove. "if x" is a duck-type test for a boolean value. Obviously if
you know the type and want a more *specific* test, then you can use an
explicit one. Any time you don't know or don't care about a more
specific type than "something which probably is boolean true", or any
time where you know the boolean semantics of the type and want to drop
some typing, you can use "if x". The more specific test is what you'd
use if you want more specific results. What's complicated about this
idea?

Many people trumpet that "if x" makes your code more polymorphic
whenever this comes up--in fact you just repeated the claim--without
ever considering how rarely this more extensive polymorphism comes up
in practice. I was calling them out to say "prove to me that it
actually happens".

I believe it's very rare not to know enough about the expected type
that explicit tests won't work. We've uncovered a few examples of it
in this thread, but IMO we haven't uncovered any sort of broad, wide-
ranging use cases.


Carl Banks

 

[Carl Banks]

Hmmm... well, I see your point. Unfortunately, even though it feels
incorrect to me, I do not (yet) have the breadth and depth of Python
experience to come up with an example that would display such exquisite
polymorphism. It also seems to me that such an example would be
non-trivial in nature.

Wrong on both counts. Let me give you a hint.

Take integers and lists. Count how many methods the two types have in
common. By my reckoning, they are:

__add__, __iadd__, __mul__, _imul__, __str__, __repr__, and
__nonzero__.

Also there are __getattribute__ and __setattr__, but lists and ints
share no common attributes. I don't think ints have any at all aside
from the special methods.

What this means is: any code that can work for both ints and lists
must use only these methods (this is not quite correct, but mostly
is). That eliminates probably 99.9% of code right off the bat.

Then you have to consider that some code that does use only these
attributes does something useful only for either ints or lists, not
both. Take this simple example:

def foil(a,b,c,d):
return a*b + a*d + b*c + b*d

This code works if a and c are both ints, or are both lists (while b
and d would have to be ints). It does something useful for ints and
other numeric types, but does it do anything useful for lists?

Third, there are cases where you have code that works for both ints
and lists, that does do something useful for both, but it makes no
sense to apply "if x" to it.

def print_if_true(x):
if x:
print x

This little piece of code might actually come in handy here and there,
but would it end up being used for numeric types a whole lot? It
seems like if you have an empty list you might reasonably want to
print nothing at all, but when you have a zero integer you usually
want to print the zero.

(However, you might reasonably want to print some object unless it's
None. You could then use this one function to handle that case as
well as the case with empty lists, instead of defining two functions,
like so:

print_if_not_empty(x):
if len(x)!=0:
print x

print_if_not_None(x):
if x is not None:
print x

But wait! What if you had a value that could be either an integer or
None? You can't use the "if x" variant because x could be False. So
you'd have to define the "if x is not None" variant anyway. I might
use the "if x" variant in throwaway scripts, but for anything
production I would prefer the explicit variants, partially because, as
shown above, even when "if x" does rear it's polymoprhic head, it
doesn't necessarily buy you much.

Nevertheless, I think this is probably the best example of the
enhanced polymorphism of "if x" yet. I'm kind of surprised no one
came up with it.)

In general, the ability to take advantage of "if x" polymorphism
across numeric, container, and other types depends on the something/
nothing dichotomy to be applicable. Something versus nothing is a
useless concept most of the time, but occasionally finds use in human
interaction cases such as printing.

So there you have it. This is why I didn't expect anyone to come up
with a good example: you just don't have a lot to work with.



Carl Banks

 

[Carl Banks]

Yes, all people are idiots for reading what you wrote, reading your
later realization that it was wrong, and taking both at face value.
I'll be sure never to make that mistake again!

I thought it was obvious that he was paraphrasing. I also think that,
among the people who took it literally, those who are not an ass would
have accepted his explanation that he was paraphrasing it and moved
on, rather than rubbing it in.

If you recall, I agreed with his statement. Would you like to claim
that I don't understand the fundamentals of Python?


Carl Banks

 

[Carl Banks]

[snip excellent explanation of why it's hard to for "if x" to be
extensively polymorphic]


By the way, one thing I forgot to mention is Matt Fitzgibbons' filter
example.

As I said, it's hard to write code that works for both numeric and
container types because they share so few methods. However, sometimes
you don't know ahead of time what methods are used! When you're doing
functional programming you might pass in a method that calls the
appropriate method, like so:

def apply_if_true(func,x):
if x:
func(x)

The function f could use methods appropriate for ints when x is an
int, and for lists when x is an int.

I did downplay this, because frankly, not many programmers use
functional methods this extensively. But one should also note that
you could pass in the test as well:

def apply_if_condition(func,cond,x):
if cond(x):
func(x)

Thus this usage of "if x" arguably could be considered "replaceable
with a simple explicit test". But in the interests of harmony I
didn't push the point, and it really would have been stretching the
spirit of what I was trying to prove.


Carl Banks

 

[Antoon Pardon]

Many people trumpet that "if x" makes your code more polymorphic
whenever this comes up--in fact you just repeated the claim--without
ever considering how rarely this more extensive polymorphism comes up
in practice. I was calling them out to say "prove to me that it
actually happens".

I believe it's very rare not to know enough about the expected type
that explicit tests won't work. We've uncovered a few examples of it
in this thread, but IMO we haven't uncovered any sort of broad, wide-
ranging use cases.

I was reading this thread and was wondering about the following problem.

I have a loop that continuously calculates something. I also have a
producer that may produce extra input that can influence the
calculations. The producer can also indicate the loop should finish.

So the result of produce can be three things:

1) A not empty sequence; indicating available input.

2) An empty sequence; indicating no input available now.

3) None; indicating the end of the calculations.


So the loop essentially looks like this:

while 1:
extra = produce()
if extra is None:
break
for el in extra:
adjust_with(el)
calculate()


I now have the following question for people who argue that "if x"
is more polymorphic. I could subclass list, so that instances
of this new sequence would always behave as true, even if they are
empty. I could then rewrite my loop as follows:

while 1:
extra = produce()
if not extra:
break
for el in extra:
adjust_with(el)
calculate()

Is this second loop now more polymorphic as the first?

Personnaly I would argue that the first loop with the more specific
test is more polymorphic in this case, as it works with the standard
sequence semantics of python; so the first loop will work with
produce, producing any kind of sequence while the second loop
will only work with produce producing a specific kind of sequence.

 

[Antoon Pardon]

It's more confusing since you've changed the standard behavior of a
standard type, which doesn't really have anything to do with
polymorphism. It's more confusing, if that's a benefit.

So you accept my point that "if x" can be less polymorphic
and in fact can be more confusing than a more specific test.

 

[Matthew Fitzgibbons]

So you accept my point that "if x" can be less polymorphic
and in fact can be more confusing than a more specific test.

I think your example is more related to a trap with polymorphism in
general rather than an argument against 'is x' specifically. Any time
you override a method, you have an opportunity to change the behavior in
unexpected ways. Especially in languages like Python that don't do type
checking. For example, you write a __cmp__ method like this:

class ReallyBadPractice(object):
def __cmp__(self, other):
return -cmp(self, other)

Of course any code that does comparisons on this class is going to
behave unexpectedly!

-Matt

 

[Matthew Fitzgibbons]

[snip excellent explanation of why it's hard to for "if x" to be
extensively polymorphic]


By the way, one thing I forgot to mention is Matt Fitzgibbons' filter
example.

As I said, it's hard to write code that works for both numeric and
container types because they share so few methods. However, sometimes
you don't know ahead of time what methods are used! When you're doing
functional programming you might pass in a method that calls the
appropriate method, like so:

def apply_if_true(func,x):
if x:
func(x)

I find myself doing things like this surprisingly often. All you've done
is move the decision as to what function is applied to x elsewhere. Like
a factory, for example. I could see using something like this where func
prepares object x to be inserted into a database, and you want to make
sure x is meaningful first.

def add_to_db(prep_func, x):
if x:
entry = prep_func(x)
add_to_db(entry)

'if x' strikes me as better for this case because you might want to
accept a non-empty list (or some other objects) but reject non-empty
lists. 'if x is None' would not work. It still may be susceptible to the
empty iterator problem, depending on what prep_func does.

The function f could use methods appropriate for ints when x is an
int, and for lists when x is an int.

I did downplay this, because frankly, not many programmers use
functional methods this extensively. But one should also note that
you could pass in the test as well:

def apply_if_condition(func,cond,x):
if cond(x):
func(x)

Thus this usage of "if x" arguably could be considered "replaceable
with a simple explicit test". But in the interests of harmony I
didn't push the point, and it really would have been stretching the
spirit of what I was trying to prove.


Carl Banks

Are you using cond to (a) determine is x is a suitable type to pass to
func, or (b) whether or not to call func, based on some other
characteristics of x?

If (a), I think using a condition is this way is a little goofy. Why not
just allow func to decide is x is an acceptable argument?

(b) is not really what we've been discussing, so I assume not. I would
seriously consider refactoring cond outside the function anyway.

-Matt

 

[Matthew Fitzgibbons]

'if x' strikes me as better for this case because you might want to
accept a non-empty list (or some other objects) but reject non-empty
lists. 'if x is None' would not work. It still may be susceptible to the
empty iterator problem, depending on what prep_func does.

I mean reject empty lists. I need to be slower on hitting send.

-Matt

 

[Antoon Pardon]

I think your example is more related to a trap with polymorphism in
general rather than an argument against 'is x' specifically.

I didn't want to argue against "if x". I just wanted to give a
counter point to the argument that "if x" is more polymorphic.

Whether more or less polymorphic is good or bad, depends on
what kind of polymophism and circumstances.