Interface and duck typing woes

[Joe Junior]

While designing a simple library, I found myself asking a
philosophical question: to check or not to check the parameter's
interface?

I think that, considering it is Python, the usual answer would be
"no", but here is the situation that got me thinking:

class Flock:

def __init__(self):
self.ducks= []

def do_stuff(self):
for duck in self.ducks:
duck.quack()

class Duck:

def quack(self):
#quack-quack
pass

f = Flock()
d = Duck()
f.ducks.append(d)
f.do_stuff()

Ok, no big deal there, the problem is if the user forgets to implement
the quack() method. The stack trace would complain that "duck.quack()"
is wrong, but that can happen hundreds of lines after the user
actually added the object to the Flock, and it can be hard to find out
what is happening and which object is wrong.

Of course I don't want to check isistance(), I like duck typing, but
should I check if hasattr() and callable() before adding to the
container? What is the pythonic way to deal with it? Am I worrying too
much ;-)?

Thanks,

Joe

 

[Steven D'Aprano]

While designing a simple library, I found myself asking a philosophical
question: to check or not to check the parameter's interface?

The only correct answer to that is, "Yes no maybe".

I think that, considering it is Python, the usual answer would be "no",
but here is the situation that got me thinking:

class Flock:

def __init__(self):
self.ducks= []

def do_stuff(self):
for duck in self.ducks:
duck.quack()

class Duck:

def quack(self):
#quack-quack
pass

f = Flock()
d = Duck()
f.ducks.append(d)
f.do_stuff()

Ok, no big deal there, the problem is if the user forgets to implement
the quack() method. The stack trace would complain that "duck.quack()"
is wrong, but that can happen hundreds of lines after the user actually
added the object to the Flock, and it can be hard to find out what is
happening and which object is wrong.

True, but that's a good case for improving the error message, or using a
debugger. Here is Flock.do_stuff re-written to give a more verbose/useful
error message:


def do_stuff(self):
for i, duck in enumerate(self.ducks):
try:
duck.quack()
except AttributeError:
raise DuckError(
'object %r at index %d has no quack' % (duck, i)
)


Okay, seeing the index is useful. But we would have got nearly as much
information from the AttributeError traceback, minus the index:

py> (42).quack()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute 'quack'

So how much extra work are you prepared to put in to make a rare
occurrence (passing a magpie instead of a duck) easier to debug? Since
errors are presumably rare, maybe the answer is "not a lot of extra
work". But if the consequence of an error is catastrophic ("for want of a
duck quacking, the nuclear reactor explodes, making the northern
hemisphere uninhabitable"), maybe the answer is "as much as it takes".

Other questions: what happens if duck.quack is buggy and raises
AttributeError? A good question, but just how far should we go in
worrying about things like this? What happens if duck.quack is buggy and
raises StopIteration? Sometimes the right reaction is to deal with it if
and when it actually occurs. In other words, wait for the bug report
before trying to fix it.

("Fix it" may mean telling people "don't do that!".)

Of course I don't want to check isistance(), I like duck typing, but
should I check if hasattr() and callable() before adding to the
container? What is the pythonic way to deal with it? Am I worrying too
much ;-)?

Yes

Except in the (rare?) case that you aren't worrying enough, in which case
you can check hasattr and callable up front, or do whatever other tests
you feel the need to check. It depends on the specific code you are
writing.

 

[Nobody]

Of course I don't want to check isistance(), I like duck typing, but
should I check if hasattr() and callable() before adding to the container?

That won't tell you if the object has a quack() method but with
incompatible semantics (e.g. wrong number or types of arguments).

What is the pythonic way to deal with it?

Ignore it. If you want early type checking, use a statically-typed
language.

 

[Joe Junior]

Well, the main reason for me asking this question here was because of
the Java/C#/Whatever developer in me craving for an Interface for the
container's items, and I noticed that I'm not alone in this. But I was
actually expecting the "We're all consenting adults, here", I guess I
just needed the confirmation

Another reason for this question is that I read some people saying
they wouldn't use python for large projects, and they always point at
the lack of Interfaces as a concern. I actually disagree, but I can
see their point. What do you think?

@Nobody

That won't tell you if the object has a quack() method but with
incompatible semantics (e.g. wrong number or types of arguments).

Yeah, didn't think about that, it's kinda swimming upstream!
Definitely it is more hassle than it is worth.

@ChrisA

Do you believe that you can write code to catch every bug you might
make? If so, you are naive and probably haven't spent much time
programming yet And if not, then you must acknowledge that bugs
WILL happen; therefore you will need to cope with them after the
event. So rather than trying to prevent them all, just improve your
means of coping, and you'll accomplish the same end with much less
trouble.

Oh, no! I'm not that presumptuous (or naive)! But what do you mean by
"improve means of coping"? Do you know any article on the subject you
could point me?

@Steven

Yes

Well, thanks! And thanks for the article.

 

[Chris Angelico]

@ChrisA

Oh, no! I'm not that presumptuous (or naive)! But what do you mean by
"improve means of coping"? Do you know any article on the subject you
could point me?

Hmm. l don't know of any good articles off-hand. But what I'm talking
about is simply developing the skill of reading exceptions, plus a few
simple things like knowing where it's appropriate to catch-and-log;
sometimes, what that means is actually writing some code to (for
example) email you whenever there's an exception, but more likely it
means writing no code at all, and just looking at STDERR of your live
usage. Works really well for >95% of Python scripts.

The most important thing to consider is: What happens if my code
doesn't run all the way through? Is it safe for this to run part way,
then bomb with an exception? For many scripts, it's pretty easy: fix
the problem and rerun the script, and it'll completely rewrite its
output file. For others, this is a good reason for putting all your
"real data" into a transactional database - you begin a transaction at
the top, don't commit till the end, and if an exception kills your
script, your transaction will be rolled back. I have a system for
patching our database based on a script (written in Pike, not Python,
but the same applies); if I have any sort of critical failure in the
patch script, it'll bomb out as soon as I test it - but since I use
PostgreSQL, all that DDL (eg "ALTER TABLE") is covered by
transactional integrity (which it isn't with MySQL - another reason to
be wary of MySQL), so my patch will be backed out, and I can fix it
and start over. I don't need to have a Look Before You Leap approach
to database changes - I can simply do stuff, and if it crashes, all's
well. (That same script also has a system for catching errors at a
mid-level point that means that the process doesn't terminate when
there's an error; it supports full code reload, so once I fix the
patch, I send the process a SIGHUP and it fetches from disk again.)
*That* is error handling the safe way.

ChrisA

 

[Joe Junior]

Hmm. l don't know of any good articles off-hand. But what I'm talking
about is simply developing the skill of reading exceptions, plus a few
simple things like knowing where it's appropriate to catch-and-log;
sometimes, what that means is actually writing some code to (for
example) email you whenever there's an exception, but more likely it
means writing no code at all, and just looking at STDERR of your live
usage. Works really well for >95% of Python scripts.

The most important thing to consider is: What happens if my code
doesn't run all the way through? Is it safe for this to run part way,
then bomb with an exception? For many scripts, it's pretty easy: fix
the problem and rerun the script, and it'll completely rewrite its
output file. For others, this is a good reason for putting all your
"real data" into a transactional database - you begin a transaction at
the top, don't commit till the end, and if an exception kills your
script, your transaction will be rolled back. I have a system for
patching our database based on a script (written in Pike, not Python,
but the same applies); if I have any sort of critical failure in the
patch script, it'll bomb out as soon as I test it - but since I use
PostgreSQL, all that DDL (eg "ALTER TABLE") is covered by
transactional integrity (which it isn't with MySQL - another reason to
be wary of MySQL), so my patch will be backed out, and I can fix it
and start over. I don't need to have a Look Before You Leap approach
to database changes - I can simply do stuff, and if it crashes, all's
well. (That same script also has a system for catching errors at a
mid-level point that means that the process doesn't terminate when
there's an error; it supports full code reload, so once I fix the
patch, I send the process a SIGHUP and it fetches from disk again.)
*That* is error handling the safe way.

Oh, I get it! Thanks.

 

[alex23]

Another reason for this question is that I read some people saying
they wouldn't use python for large projects, and they always point at
the lack of Interfaces as a concern. I actually disagree, but I can
see their point. What do you think?

Having worked on large Python projects both with & without interfaces,
it's definitely possible with either approach. It certainly isn't
lacking in for support for them:

http://docs.zope.org/zope.interface/

 

[Steven D'Aprano]

Well, the main reason for me asking this question here was because of
the Java/C#/Whatever developer in me craving for an Interface for the
container's items, and I noticed that I'm not alone in this. But I was
actually expecting the "We're all consenting adults, here", I guess I
just needed the confirmation

Another reason for this question is that I read some people saying they
wouldn't use python for large projects, and they always point at the
lack of Interfaces as a concern. I actually disagree, but I can see
their point. What do you think?

Interfaces aren't a built-in part of the language, but big frameworks
like Zope and Twisted include them. See for example discussion here:

http://dirtsimple.org/2004/12/python-interfaces-are-not-java.html

In a more ad-hoc manner, there are recipes for interface-like
functionality. For example, from the Python Cookbook, we have this:

http://code.activestate.com/recipes/52291


It's a myth that Python is entirely opposed to type-checking. Many built-
ins do it. Way back in Python 1.5, Python's creator Guido van Rossum
wrote an essay describing a way to implement Eiffel-like pre- and post-
condition checking:

http://www.python.org/doc/essays/metaclasses/


These days, it would be relatively simple to implement pre- and post-
condition checking using decorators, and indeed one of the motivating use-
cases for function annotations in Python 3 is to allow such things.

http://www.python.org/dev/peps/pep-3107/

(Function annotations are perhaps the best Python feature that nobody
uses.)

 

[jussi.santti]

While designing a simple library, I found myself asking a

philosophical question: to check or not to check the parameter's

interface?

Design by contract discipline says: do not.

I think that, considering it is Python, the usual answer would be

"no", but here is the situation that got me thinking:



class Flock:



def __init__(self):

self.ducks= []



def do_stuff(self):

for duck in self.ducks:

duck.quack()



class Duck:



def quack(self):

#quack-quack

pass



f = Flock()

d = Duck()

f.ducks.append(d)

f.do_stuff()



Ok, no big deal there, the problem is if the user forgets to implement

the quack() method. The stack trace would complain that "duck.quack()"

is wrong, but that can happen hundreds of lines after the user

actually added the object to the Flock, and it can be hard to find out

what is happening and which object is wrong.



Of course I don't want to check isistance(), I like duck typing, but

should I check if hasattr() and callable() before adding to the

container? What is the pythonic way to deal with it? Am I worrying too

much ;-)?



Thanks,



Joe

 

[Roy Smith]

These days, it would be relatively simple to implement pre- and post-
condition checking using decorators, and indeed one of the motivating use-
cases for function annotations in Python 3 is to allow such things.

http://www.python.org/dev/peps/pep-3107/

(Function annotations are perhaps the best Python feature that nobody
uses.)

This is awesome.

 

[Steven D'Aprano]

This is awesome.

Heh, everybody has one of two reactions:

"This is awesome!"

"You'll add type checking to my Python code over my dead body!!!"

But I'm still to see a practical use for annotations in real world code.
Or indeed to think of a use for them other than type checking.

 

[Joshua Landau]

Heh, everybody has one of two reactions:

"This is awesome!"

"You'll add type checking to my Python code over my dead body!!!"

But I'm still to see a practical use for annotations in real world code.
Or indeed to think of a use for them other than type checking.

I occasionally use them for documentation. I think that there some are
cases where the return type (encoded as a string) is as good an indicator
of functionality as a short docstring, so use both.

 

[Roy Smith]

Heh, everybody has one of two reactions:

"This is awesome!" [[i.e. what I said]]

"You'll add type checking to my Python code over my dead body!!!"

Duck typing is a funny thing. Sure, I don't have to give you a Duck, I
just have to give you something that quacks like a Duck. But, at some
point, you and I need to agree on what that means. If you're expecting
a https://en.wikipedia.org/wiki/Duck and I give you a
https://en.wikipedia.org/wiki/DUKW, we've had a failure to communicate.

To take a quasi-realistic example, let's say I've got this musical
masterpiece in my database:

{
"_id" : ObjectId("4ccb7052e5f37551d479add6"),
"album" : "My World",
"album_comp_id" : NumberLong(34732133),
"artist" : "Justin Bieber",
"avail_date_aac" : ISODate("1970-01-01T00:00:00Z"),
"avail_date_mp3" : ISODate("1970-01-01T00:00:00Z"),
"duration" : 192,
"genre" : "Pop",
"mn_comp_id" : NumberLong(34732147),
"seq_num" : 1297,
"song_id" : 6544798,
"title" : "Love Me",
"track" : -1
}

If I want to ask you, "Please return to me a url from which I can stream
this song as an mp3", I could look at your Song class and find:

@staticmethod
def get_stream_url(song):
[some code goes here]

but that would leave me wondering what you mean by "song". You could
legitimately mean 6544798, "4ccb7052e5f37551d479add6",
ObjectId("4ccb7052e5f37551d479add6"), an instance of the Song class
itself, or possibly even 34732147. Depending on the context, any of
those might very well be the right answer.

And, we haven't even gotten to describing what I should expect to get
back.

So, to clear things up, you had to go and write something in the doc
string:

@staticmethod
def get_stream_url(song):
"""Song is an instance of class Song. This will return
an absolute URL as a string."""

But, why not just embed that information in some way which is both
compact and machine readable?

Of course, when I say, "Song is an instance of class Song", what I
really (in full duck typing glory) mean is, "Song is something which has
a "mn_comp_id" attribute whose value is something which I can pass to
str() and get back a properly formatted decimal integer string. So,
this would work, wouldn't it?

class Platypus:
def __getattr__(self, name):
return 34732147

duck = Platypus()
Song.get_stream_url(duck)

Hey, it met the description you gave me, didn't it? And, sure enough,
if you do with duck what I expect you will, we will soon hear Justin
Bieber's, "Love Me" coming out the speaker. But, in reality, I suspect
we would quickly get into an argument about just what exactly did you
mean when you said "Duck".