How about "pure virtual methods"?

[Noam Raphael]

This is false. For instance:

class A(object):
def method(self, a):
print a

class B(A):
def method(self, a, b):
print a, b

B implements a different interface than A. Statically typed OO
languages either use multi-methods or disallow changing the signature
of an overridden method.

A tool to detect such cases would probably be almost as useful as the
tool you've proposed.

<mike

I agree that such a tool would be very useful. In fact, I think it
exists - I'm sure pychecker checks for mistakes like that. I understand
that it checks for not implementing functions which just raise an
exception too, so you can say, "why all this mess? Run pychecker and
everything will be good." However, I think that there is a difference
between these two tests, which explains why one should be done by the
language itself and one should be done by an analysis tool.

The difference is that handling arguments, in Python, can be seen as a
part of the *implementation*, not the interface. The reason is that you
can write a method which simply gets a (*args, **kwargs), and raises a
TypeError if the number of args isn't two, and it would be completely
equivalent to a function which is defined using def f(a, b). Of course,
even in statically typed languages, you can't enforce an implementation
to do what it should (too bad - it would have made debugging so much
easier...)

So checking whether the argument list of a method of a subclass suits
the argument list of the original implementation is nice, but should be
left to external analysis tools, but checking whether a method is
defined at all can be done easily by the language itself.

Noam

 

[Alex Martelli]

However, I'm not sure if this solves my practical problem - testing
whether all abstract methods were implemented. I think that usually, you
can't write a test which checks whether an abstract method did what it
should have, since different implementations do different things. I

Still, there must be some _protocol_ about what those abstract methods
are all about... otherwise you _are_ just talking about an *interface*,
the way I use the terms (I didn't invent that usage, I just find it very
useful in practice). An interface is "just syntax" -- method names and
signatures; a protocol adds semantics and pragmatics (the distinction
between the last two may only be meaningful to somebody with a
background in linguistics, and it's not important here anyway).

If some method is a hook which may do anything including nothing then
it's most often more useful not to make it abstract (or absent) in the
base class, but rather to define it there as a no-op. This way, you
reduce the boilerplate in all subclasses: they only override the methods
they NEED to override. Besides no-op, this could also apply to other
semantics that were "a very likely possibility" -- but it's unusual for
any given semantics to be "very likely", except "first, do no harm";-)

When you design a protocol, _make it testable_. Say that you start with
the idea of the protocol having two methods:

def frambozzle(self):
''' must make the instance frambozzled '''
def unframbozzle(self):
''' must make the instance not-frambozzled '''

This protocol is not complete. Add a third method:

def isframbozzled(self):
''' return a true value iff the instance is frambozzled '''

NOW you have a testable protocol! Sorry for the toy-ness of the
example, but I don't want to post for hundreds of lines. Beyond the
direct advantages of being able to write much better tests, experience
teaches that a testable protocol tends to have extra resilience and
usefulness for perfectly reasonable uses that the protocol designer
might not have anticipated. E.g., with the testable protocol one can
write

def toggle_frambozzling(some_instance):
if some_instance.isframbozzled():
some_instance.unframbozzle()
else:
some_instance.frambozzle()

Note that we're only using semantics here, aka abstract semantics, not
pragmatics (just to illuminate the semi-relevant distinction I was using
above). Abstract semantics are those you can fully capture by axioms
related to methods that are part of the protocol itself, without for
example needing to refer to the external world, or other objects. You
get pragmatics when such references are there, e.g., informal specs of
the kind "the shipping costs of a frambozzled instance are expected to
not exceed that of an unframbozzled instance, and normally to be below;
on the other hand, frambozzled instances need only be marshalable on a
single platform and version of Python and of this framework, while
unframbozzled instances must ensure marshaling cross-platform on any
version of Python >= 2.3.4 and version of this framework >= 0.8.2". In
practice, only projects with relatively high formality tend to write
things down to this level, of course, but generally there is some less
formal pragmatic understanding of what it's all about.

For example, in a recent thread about hashing &c I pointed out that
adding a method such as "def __hash__(self): return 23" can ensure
semantically correct hashing of any class; OTOH, it DOES badly violate
pragmatics (and produces horrid performance!-), because hash values are
MEANT to "differentiate" among different instances (they don't _hafta_,
but they're expected to do SOME level of reasonable effort!-).

don't even know how you can test whether an abstract method was
implemented - should you run it and see if it raises a
NotImplementedError?

Makes sense to me, yes. assertRaises seems good;-).

But with what arguments?

If you're going to use your abstract base class for ANY purpose, you'd
better have SOME idea of the signature with which you can call those
methods which its subclasses must implement! Otherwise, it's all for
naught: you get zero polymorphism, period. Say that subclass A has
def whoa(self): ...
while subclass B has
def whoa(self, andmore): ...
then you can NEVER call whoa without knowing which subclass you have an
instance of -- no polymorphism! It's fine if a subclass wants to have a
WIDER signature (add OPTIONAL args to those specified in the base class,
even a *any if you wish), but there must be SOME commonality. Well
then, call according to that commonality, just like any USE of you
abstract base class will do.

This points out that, if you wish to do checks at 'class' statement time
which determine whether a class is instantiable, the signature of the
methods should be part of those checks. Fortunately, module inspect
makes it reasonably easy to get and check methods' signatures.

And even if you find a way
to test whether a method was implemented, I still think that the
duplication of code isn't very nice - you have both in your class
definition and in your test suite a section which says only "method
so-and-so should be implemented."

Then it may be best to ONLY have the test suite -- don't put the method
in the ABC at all. If you WANT the method in the ABC, for documentation
purposes, well then, that's not duplication of code, it's documentation,
which IS fine (just like it's quite OK to have some of the same info in
a Tutorial document, in a Reference one, AND in a class's docstring!).

If you don't want to have the duplication your unit tests become easier:
you just getattr from the class (don't even have to bother instantiating
it, ain't it great!), and check the result with inspect.

I think that making abstract methods a different object really makes
sense - they are just something else. Functions (and methods) define
what the computer should do. Abstract methods define what the
*programmer* should do.

Unit tests define what the programmer should do, to a large extent, much
better than other approaches do (until it comes to the integration of
components, but then, abstract methods ain't much use there either;
tests of other ilks may be preferable;-).


ITRW (in the real world) you MAY find yourself in situations where you
know unit-tests WILL be skimped on. Then you're in crisis mode from the
word go, and taking what little precautions you can (including custom
metaclasses that in practice won't be bypassed and can enforce a tiny
sliver of sanity in what will inevitably remain a crazy environment) may
be among the band-aids you can try. Besides, part of your investment in
building that infrastructure does get repaid in getting some errors
diagnosed earlier (though not as thoroughly and early as unit tests
might, but, we're starting from the sad RW assumption that _some_ deuced
coders won't do them, and we're just trying to reduce "totally dead" to
"mostly dead"...;-).


Alex

 

[Alex Martelli]

This is false. For instance:

class A(object):
def method(self, a):
print a

class B(A):
def method(self, a, b):
print a, b

B implements a different interface than A. Statically typed OO
languages either use multi-methods or disallow changing the signature
of an overridden method.

A tool to detect such cases would probably be almost as useful as the
tool you've proposed.

Agreed, and besides, the two tools can usefully be one. Look at the
inspect module for signature checking helpers...


Alex

 

[Mike Meyer]

Of course, even in statically typed languages, you can't enforce
an implementation to do what it should (too bad - it would have made
debugging so much easier...)

That's what DbC languages are for. You write the contracts first, then
the code to fullfill them. And get exceptions when the implementation
doesn't do what the contract claims it does.

<mike

 

[Steve Holden]

Alex Martelli wrote:

[...]

Sorry for the toy-ness of the
example, but I don't want to post for hundreds of lines.

[...]

Can we take it this is some kind of early New Year's resolution?

happy-christmas-ly y'rs - steve

 

[Noam Raphael]

That's what DbC languages are for. You write the contracts first, then
the code to fullfill them. And get exceptions when the implementation
doesn't do what the contract claims it does.

<mike

Can you give me a name of one of them? This is a very interesting thing
- I should learn one of those sometime. However, I'm pretty sure that
programming in them is hell, or at least, takes a very long time.

Noam

 

[Noam Raphael]

Thank you very much for this answer! I learned from you about unit
tests, and you convinced me that "testing oriented programming" is a
great way to program.

You made me understand that indeed, proper unit testing solves my
practical problem - how to make sure that all the methods which should
be implemented were implemented. However, I'm still convinced that this
feature should be added to Python, for what may be called "aesthetic
reasons" - I came to think that it fills a gap in Python's "logic", and
is not really an additional, optional feature. And, of course, there are
practical advantages to adding it.

The reason why this feature is missing, is that Python supports building
a class hierarchy. And, even in this dynamically-typed language, the
fact that B is a subclass of A means that B is supposed to implement the
interface of A. If you want to arrange in a class hierarchy a set of
classes, which all implement the same interface but don't have a common
concrete class, you reach the concept of an "abstract class", which
can't be instantiated. And the basestring class is exactly that.

The current Python doesn't really support this concept. You can write in
the __new__ of such a class something like "if cls == MyAbstractClass:
raise TypeError", but I consider this as a patch - for example, if you
have a subclass of this class which is abstract too, you'll have to
write this exception code again. Before introducing another problem, let
me quote Alex:

... If you WANT the method in the ABC, for documentation
purposes, well then, that's not duplication of code, it's documentation,
which IS fine (just like it's quite OK to have some of the same info in
a Tutorial document, in a Reference one, AND in a class's docstring!).

If you don't want to have the duplication your unit tests become easier:
you just getattr from the class (don't even have to bother instantiating
it, ain't it great!), and check the result with inspect.

That's actually right - listing a method which should be implemented by
subclasses, in the class definition is mainly a matter of
*documentation*. I like the idea that good documentation can be derived
from my documented code automatically, and even if I provide an external
documentation, the idea that the code should explain itself is a good
one. The problem is, that the current convention is not a good
documentation:

def frambozzle(self):
''' must make the instance frambozzled '''
raise NotImplementedError

The basic problem is that, if you take this basic structure, it already
means another thing: This is a method, which takes no arguments and
raises a NotImplementedError. This may mean, by convention, that this
method must be implemented by subclasses, but it may also mean that this
method *may* be implemented by subclasses. I claim that a declaration
that a method must be implemented by subclass is simply not a method,
and since Python's "logic" does lead to this kind of thing, it should
supply this object (I think it should be the class "abstract"). Two of
Python's principles are "explicit is better than implicit", and "there
should be (only?) one obvious way to do it". Well, I think that this:

@abstract
def frambozzle(self):
"""Must make the instance frambozzled"""
pass

is better than the previous example, and from

def frambozzle(self):
raise NotImplementedError, "You must implemented this method, and
it must make the instance frambozzled"

and from

def frambozzle(self):
"""Must make the instance frambozzled.

PURE VIRTUAL
"""
pass

and from maybe other possible conventions. Note also that making this
explicit will help you write your tests, even if Python would allow
instantiation of classes which contain abstract methods - you will be
able to simple test "assert not isinstance(MyClass.frambozzle, abstract)".
(I don't like the solution of not mentioning the method at all, which
makes the test equally simple, because it doesn't document what the
method should do in the class definition, and I do like in-code
documentation.)

To summarize, I think that this feature should be added to Python
because currently, there's no proper way to write some code which fits
the "Python way". As a bonus, it will help you find errors even when
your unit tests are not sufficient.

I plan to raise this issue in python-dev. If you have any additional
comments, please post them here. (I will probably be able to reply only
by the weekend.)

Have a good day,
Noam

 

[Mike Meyer]

Can you give me a name of one of them? This is a very interesting
thing - I should learn one of those sometime. However, I'm pretty sure
that programming in them is hell, or at least, takes a very long time.

Eiffel. Google for SmartEiffel for a free portable implementation. If
you want an IDE/etc., look for EiffelStudio - though it's only
available in binary form. And the two implement different languages
(EiffelStudio implements things accepted as standard for OOSCv3,
whereas SmartEiffel implements things that are still under
consideration in their own variants, and fails to implement some
critical features), though the DbC parts are identical.

I find Eiffel fun to program in. SmartEiffel's libraries are as
flexible as Python's built in types. And debugging with all the
contract checking turned on causes exceptions for truly obscure bugs.

<mike

 

[Mike Meyer]

The current Python doesn't really support this concept. You can write
in the __new__ of such a class something like "if cls ==
MyAbstractClass: raise TypeError", but I consider this as a patch -
for example, if you have a subclass of this class which is abstract
too, you'll have to write this exception code again. Before
introducing another problem, let me quote Alex:

You can do *much* better than that. You can finger the class that the
object is an instance of. You don't need a new language construct to
do it, either:
.. def virtualmethod(self):
.. raise NotImplementedError("virtualmethod not implemented in class %s" \
% self.__class__.__name__)
.. pass
.. Traceback (most recent call last):
File "<stdin>", line 1, in ?

I think that error message pinpoints the class of the instance that's
missing the virtual method, which is as good as your proposal would
do, adequately documents that this method needs to be implemented by
subclasses, and has the advantage that you can instantiate only part
of the interface to an abstract class if you only need part of it.

<mike

 

[Alex Martelli]

Can you give me a name of one of them? This is a very interesting thing
- I should learn one of those sometime. However, I'm pretty sure that
programming in them is hell, or at least, takes a very long time.

If you want to experiment with DbC within Python, I think you might want
to try <http://www.pypackage.org/packages/python-dbc> -- not sure it's
maintained and up-to-date with today's Python, but I suspect the answer
is "yes", and it would give you a chance to try DbC as a methodology,
quite separately from the issue of changing programming languages.


Alex

 

[Noam Raphael]

Thanks for your suggestion, but it has several problems which the added
class solves:

* This is a very long code just to write "you must implement this
method". Having a standard way to say that is better.
* You can instantiate the base class, which doesn't make sense.
* You must use testing to check whether a concrete class which you
derived from the base class really implemented all the abstract methods.
Testing is a good thing, but it seems to me that when the code specifies
exactly what should happen, and it doesn't make sense for it not to
happen, there's no point in having a separate test for it.

About the possibility of implementing only a subset of the interface:
You are perfectly welcomed to implement any part of the interface as you
like. Function which use only what you've implemented should work fine
with your classes. But you can't claim your classes to be instances of
the base class - as I see it, subclasses, even in Python, guarantee to
behave like their base classes.

Have a good day,
Noam

 

[George Sakkis]

Thanks for your suggestion, but it has several problems which the added
class solves:

* This is a very long code just to write "you must implement this
method". Having a standard way to say that is better.
* You can instantiate the base class, which doesn't make sense.
* You must use testing to check whether a concrete class which you
derived from the base class really implemented all the abstract methods.
Testing is a good thing, but it seems to me that when the code specifies
exactly what should happen, and it doesn't make sense for it not to
happen, there's no point in having a separate test for it.

Here's a more refined implementation of the one posted before that
addresses these issues. It defines 'abstractclass', which would be
a nice class decoraror when (and if) class decorators make it into
the language. The extra benefit compared to the MustImplement metaclass
is that it allows a class with no abstract methods to be defined as abstract.
This makes no sense if one defines an abstract class as "a class with one
or more abstract (deferred) methods". Perhaps a more useful and general
definition of an abstract class is "a class that is not allowed to be instantiated".
Deferred methods are one reason for making a class uninstantiable, but it's
not the only one. Sometimes it is useful to define a base class that provides
a default implementation of a full protocol, yet disallow its direct instantiation,
as in the example below.

George

#===========================================================
# test_abstract.py

import unittest
from abstract import abstractclass

class AbstractTestCase(unittest.TestCase):

def test_no_abstractmethods(self):
class SomeBase(object):
# This class has no abstract methods; yet calling foo() or bar()
# on an instance of this class would cause infinite recursion.
# Hence it is defined as abstract, and its concrete subclasses
# should override at least one of (foo,bar) in a way that breaks
# the recursion.
def __init__(self, x):
self._x = x
def foo(self, y):
return self.bar(self._x + y)
def bar(self, y):
return self.foo(self._x - y)
SomeBase = abstractclass(SomeBase)

class Derived(SomeBase):
def __init__(self,x):
SomeBase.__init__(self,x)
def foo(self,y):
return self._x * y

self.assertRaises(NotImplementedError, SomeBase, 5)
self.assertEquals(Derived(5).bar(2), 15)

if __name__ == '__main__':
unittest.main()

#===========================================================
# abstract.py

import inspect

__all__ = ["abstractclass", "abstractmethod", "AbstractCheckMeta"]


def abstractclass(cls):
'''Make a class abstract.

Example::
# hopefully class decorators will be supported in python 2.x
# for some x, x>4
#@abstractclass
class SomeBase(object):
@abstractmethod
def function(self):
"""Implement me"""
# the only way as of python 2.4
SomeBase = abstractclass(SomeBase)

@param cls: A new-style class object.
@return: A surrogate of C{cls} that behaves as abstract. The returned
class raises NotImplementedError if attempted to be instantiated
directly; still its subclasses may call its __init__. A subclass of
the returned class is also abstract if it has one or more abstract
methods, or if it is also explicitly decorated by this function. A
method is declared abstract by being assigned to NotImplemented (or
decorated by L{abstractmethod}).
@raise TypeError: If there is a metaclass conflict between C{type(cls)}
and L{AbstractCheckMeta}, or if C{cls} has an C{__abstractmethods__}
attribute.
'''
# check if cls has AbstractCheckMeta (or a subtype) for metaclass
metaclass = type(cls)
if not issubclass(metaclass, AbstractCheckMeta):
# it doesn't; try to make AbstractCheckMeta its metaclass by
# inheriting from _AbstractCheck
cls = metaclass(cls.__name__, (_AbstractCheck,) + cls.__bases__,
dict(cls.__dict__))
# replace __init__ with a proxy ensuring that __init__ is called by a
# subclass (but not directly)
old_init = getattr(cls,'__init__',None)
def new_init(self,*args,**kwds):
if self.__class__ is cls:
raise NotImplementedError("%s is an abstract class" % cls.__name__)
if old_init is not None:
old_init(self,*args,**kwds)
setattr(cls,'__init__',new_init)
return cls


def abstractmethod(function):
'''A method decorator for those who prefer the parameters declared.'''
return NotImplemented


class AbstractCheckMeta(type):
'''A metaclass to detect instantiation of abstract classes.'''

def __init__(cls, name, bases, dict):
if '__abstractmethods__' in cls.__dict__:
raise TypeError("'__abstractmethods__' is already defined in "
"class '%s': %s" % (cls.__name__,
cls.__dict__['__abstractmethods__']))
type.__init__(cls, name, bases, dict)
cls.__abstractmethods__ = [name for name, value in
inspect.getmembers(cls)
if value is NotImplemented]

def __call__(cls, *args, **kwargs):
if cls.__abstractmethods__:
raise NotImplementedError(
"Class '%s' cannot be instantiated: Methods %s are abstract."
% (cls.__name__,", ".join(map(repr,cls.__abstractmethods__))))
return type.__call__(cls, *args, **kwargs)


class _AbstractCheck(object):
'''
A class to stick anywhere in an inheritance chain to make its
descendants being checked for whether they are abstract.
'''
__metaclass__ = AbstractCheckMeta