super() and multiple inheritance failure

[Steven D'Aprano]

I don't understand why I'm getting the following behaviour when using
super() with multiple inheritance. The following is a minimal example
demonstrating the behaviour.

I have a diamond class hierarchy as follows:

o
|
B
/ \
P N
\ /
M

where:
o = object
B = BaseClass
P = PClass
N = NClass
M = MyClass

Inside MyClass().method(n), I dispatch to either NClass.method() or
PClass.method() depending on the value of the argument n. The correct
class is called, but then the *other* class method is called as well.
E.g. this is what I expect:

MyClass().method(2)
-> calls PClass.method
-> calls BaseClass.method

but this is what I get:

MyClass().method(2)
-> calls PClass method
-> calls NClass.method
-> calls BaseClass.method


and similarly for negative arguments, swapping PClass and NClass.

First off, is this the expected behaviour? I seems strange to me, can
somebody explain why it is the nominally correct behaviour?

Secondly, how should I deal with this situation? Avoid super()
altogether? Avoid multiple inheritance? Do something else?


Demonstration code follows, using doctest to demonstrate the failure and
the call pattern.


### fail.py

import sys

class BaseClass(object):
def method(self, n):
"""Return something. n must not be 0, 1 or -1.
-4
"""
assert int(n) == n and n not in (-1, 0, 1)
return n + 1

class PClass(BaseClass):
"""Deal with positive n."""
def method(self, n):
"""Return something. n must be strictly > 1.
6
"""
print >>sys.stderr, "Called from PClass"
assert int(n) == n and n > 1
return 1 + super(PClass, self).method(n)

class NClass(BaseClass):
"""Deal with negative n."""
def method(self, n):
"""Return something. n must be strictly < -1.
-2
"""
print >>sys.stderr, "Called from NClass"
assert int(n) == n and n < -1
return 1 + super(NClass, self).method(n)


class MyClass(PClass, NClass):
def method(self, n):
"""Return something useful.
-10
"""
#print >>sys.stderr, "Calling itoa with base=%d" % base
if n > 0:
print >>sys.stderr, "About to call PClass"
parent = PClass
else:
print >>sys.stderr, "About to call NClass"
parent = NClass
return parent.method(self, n)


if __name__ == '__main__':
import doctest
doctest.testmod()

 

[Chris Rebert]

I don't understand why I'm getting the following behaviour when using
super() with multiple inheritance. The following is a minimal example
demonstrating the behaviour.

I have a diamond class hierarchy as follows:

 o
 |
 B
/ \
P  N
\ /
 M

where:
o = object
B = BaseClass
P = PClass
N = NClass
M = MyClass

Inside MyClass().method(n), I dispatch to either NClass.method() or
PClass.method() depending on the value of the argument n. The correct
class is called, but then the *other* class method is called as well.
E.g. this is what I expect:

MyClass().method(2)
->  calls PClass.method
   -> calls BaseClass.method

but this is what I get:

MyClass().method(2)
->  calls PClass method
   ->  calls NClass.method
       -> calls BaseClass.method


and similarly for negative arguments, swapping PClass and NClass.

First off, is this the expected behaviour? I seems strange to me, can
somebody explain why it is the nominally correct behaviour?

Pretty darn sure it's expected. super() follows the MRO, which in this
case is M, P, N, B, o; thus, the super() of an M instance from a P
method is indeed N.
This would normally be useful as one typically wants to call the
"parent" methods of all ancestor classes for which the method is
defined, rather than picking-and-choosing as MyClass.method() does. To
put it another way, if super() didn't act like it does, other people
would complain about N.method() getting skipped (e.g. "What then was
the point of my subclassing N if its methods don't get called?").

Remember that super() is sorta misnamed; Dylan's equivalent is named
"next-method" (Dylan is where Python cribbed much of its multiple
inheritance system from).

Cheers,
Chris

 

[Michele Simionato]

I don't understand why I'm getting the following behaviour when using
super() with multiple inheritance.

super is working as intended. If you do not want cooperative methods,
don't use super
and call directly the superclass. I usually recommend avoiding
multiple inheritance
altogether. You may want to read "Things to know about super":

http://www.artima.com/weblogs/viewpost.jsp?thread=236275
http://www.artima.com/weblogs/viewpost.jsp?thread=236278
http://www.artima.com/weblogs/viewpost.jsp?thread=237121

 

[Steven D'Aprano]

Pretty darn sure it's expected. super() follows the MRO, which in this
case is M, P, N, B, o; thus, the super() of an M instance from a P
method is indeed N.

I was expecting super() to look at the class, not the instance. So if I
call super(PClass, self), I expected that since PClass does not inherit
from NClass, nothing should be inherited from NClass.

This would normally be useful as one typically wants to call the
"parent" methods of all ancestor classes for which the method is
defined, rather than picking-and-choosing as MyClass.method() does.

Gotcha. I was expecting super() to call the first matching ancestor
method, not all of them.

 

[Steven D'Aprano]

super is working as intended. If you do not want cooperative methods,
don't use super and call directly the superclass.

Gotcha.

Is there a standard name for what I'm trying to do, versus what super()
does? I assume the term for what super() does is "cooperative multiple
inheritance". What should I call what I'm doing? "Non-cooperative MI"
perhaps?

I usually recommend avoiding multiple inheritance altogether.

In my case, PClass and NClass are actually private classes, and it seemed
like a nice way to avoid having to fill MyClass with slightly-different
versions of each method to deal with slight variations in the arguments.
I'm aiming for some sort of polymorphic inheritance: in a method, if the
argument meets some condition, inherit from PClass, if it meets another
condition inherit from NClass, and so on. Is there are standard name for
this idea?

You may want to read "Things to know about super":

http://www.artima.com/weblogs/viewpost.jsp?thread=236275
http://www.artima.com/weblogs/viewpost.jsp?thread=236278
http://www.artima.com/weblogs/viewpost.jsp?thread=237121

Nice, thank you. It will take me a while to digest all that, this is my
first attempt at deliberate multiple inheritance, and obviously my
expectations were completely different.

 

[Michele Simionato]

Thanks for these articles. Any chance they will appear in a single
location, so we don't need three separate URLs for them?

Good idea. I have collect them in PDF form here:
http://www.phyast.pitt.edu/~micheles/python/super.pdf

Also, one article passes on this recommendation:

    use super consistently, and document that you use it, as it is part
    of the external interface for your class, like it or not.

(this is a literal citation from "super considered harmful" by James
Knight)

    So, if use classes coming from a library in a multiple inheritance
    situation, you must know if the classes were intended to be
    cooperative (using super) or not. Library author should always
    document their usage of super.

I can see lots of ways this fact could be documented, none of them
terribly concise. It's a simple “yes/no” fact about a class, after all;
what's the best way of succinctly conveying this information in a world
where the problem is barely known, let alone the solution?

Difficult question. You know that in an ideal world I would just
throw
away multiple inheritance, it is just not worth the complication.
In the real world, this is how a developer can document his hierarchy
in one-line:

class Base(object):
"This hierarchy use super. See http://fuhm.net/super-harmful"

or

class Base(object):
"This hierarchy does not use super. See http://fuhm.net/super-harmful"

That's all that is needed to third party code wanting to inherit from
the hierarchy.

 

[Gabriel Genellina]

En Sat, 26 Sep 2009 01:48:08 -0300, Steven D'Aprano

I'm aiming for some sort of polymorphic inheritance: in a method, if the
argument meets some condition, inherit from PClass, if it meets another
condition inherit from NClass, and so on. Is there are standard name for
this idea?

That doesn't look like an inheritance relationship ("is a").
If you say "from now on, act as a PClass, until further notice" and later
"now, act as a NClass" you're using the Strategy pattern.

If you decide at every invocation which method to call, it's a dispatcher;
you may use a dictionary to map each alternative to the function to be
invoked. If it only depends on the type of the argument, there is a hidden
gem in pkgutil (simplegeneric) that can help you.

 

[Chris Rebert]

I was expecting super() to look at the class, not the instance.

To be pedantic, it looks at the type of instance, taking into account
the class passed to it.

Gotcha. I was expecting super() to call the first matching ancestor
method, not all of them.

Well, it doesn't call all of them, it's just that each method
implementation usually calls super() itself; thus, in concert, in
normal usage, all the methods end up getting called.

Cheers,
Chris

 

[Michele Simionato]

If you decide at every invocation which method to call, it's a dispatcher;  
you may use a dictionary to map each alternative to the function to be  
invoked. If it only depends on the type of the argument, there is a hidden  
gem in pkgutil (simplegeneric) that can help you.

And here is an example:

http://www.artima.com/weblogs/viewpost.jsp?thread=237764

(a lot of use case for multiple inheritance are better solved with
multimethods).

 

[Dieter Maurer]

...
You know that in an ideal world I would just
throw
away multiple inheritance, it is just not worth the complication.

I am a fan of multiple inheritance: it lets the compliler/language runtime
do stupid tasks (implementing delegations), I would otherwise have to do
explicitely. True, there may be complications - but often, they can be
avoided....