Builtn super() function. How to use it with multiple inheritance? Andwhy should I use it at all?

[Jean-Michel Pichavant]

I'm not sure I understand your point here. If you call super() from a
method that doesn't exist in any superclass, then you are correct, it
won't call a method in a superclass, and will raise AttributeError.

But in the more sensible case that you only call super() when there is
actually something to inherit, then of course it calls the method in a
superclass. It certainly doesn't call methods from arbitrary unrelated
classes, only those which are in the MRO. That is, superclasses.

If Z is below A in the hierarchy, then we have no difficulty in
identifying Z as a subclass of A, and likewise we should have no problem
with identifying A as *a* (and not "the") superclass of Z, no matter how
distant they are or how tangled the DIG between them.

The exception would be if you could have loops in the class hierarchy, in
which case the concepts of super- and sub-classes breaks down completely.
But even if some other languages allowed that, Python doesn't, so we're
safe.

Quoting Michele's article (I think he's still hanging around this list)

"Readers familiar will single inheritance languages, such as Java or
Smalltalk, will have a clear concept of superclass in mind. This
concept, however, has /no useful meaning/ in Python or in other multiple
inheritance languages".


the complete article on super:
http://www.artima.com/weblogs/viewpost.jsp?thread=236275

super is a strange name for something that makes no sense in python
(talking about superclass here). It doesn't take away the usefulness of
super, but it's a proof that you can ruin a good feature with a bad name.

JM

 

[Raymond Hettinger]

EXCEPT for the last one, which must NOT call super!

The posted example happens to work because object has
a default __init__ method that does nothing. But this
is not generally true of other methods, which means you
need a "terminating" class at the end of the MRO whose
methods don't call super.

That is an important point and it is what Guido does in his examples:
http://www.python.org/download/releases/2.2.3/descrintro/#cooperation

The design options are:

* if overriding a method provided by object() such as __init__(),
__getattribute__() or __setattr__(), then you should call super() in
each class that overrides or extends those methods.

* if you know the whole class structure in advance, you call super()
in every class except the last one -- that is what Guido does in the
save() example.

* if you don't know the whole class structure in advance, then you
can't be sure which is that last class in the mro with the target
method, so you need to wrap the super() call in a try / except
AttributeError

Raymond

 

[Gregory Ewing]

A
/ \
C B
\ /
D
/ \
E F

Yes, a super call might jog left from
C to B, but only when being called from one of the lower classes D-F.
That's still an upwards call relative to the originator, not sidewards.

But it's not an upward call relative to the class mentioned
in the super() call, which is why I say it's misleading.

 

[Steven D'Aprano]

But it's not an upward call relative to the class mentioned in the
super() call, which is why I say it's misleading.

Which class would that be?

I think I'm going to need an example that demonstrates what you mean,
because I can't make heads or tails of it. Are you suggesting that a call
to super(C, self).method() from within C might call B.method(self)?

 

[Steven D'Aprano]

Quoting Michele's article (I think he's still hanging around this list)

"Readers familiar will single inheritance languages, such as Java or
Smalltalk, will have a clear concept of superclass in mind. This
concept, however, has /no useful meaning/ in Python or in other multiple
inheritance languages".

I have read Michelle Simionato's articles on super in Python. He has
taught me a lot. But on that specific matter, I think he is wrong.

Of course, he is right to say that the concept of *the* superclass is
meaningless in a MI language like Python. If MyClass inherits method
spam() from class A, and inherits method ham() from class B, which is
"the" superclass of MyClass?

But Michelle is wrong to conclude that the problem lies with the concept
of *superclass*. The problem lies with the idea that there is ONE
superclass. By dismissing the entire concept, he is throwing out the baby
with the bathwater.

The useful, and I would argue *correct*, concept of superclass is very
simple. It is a reflection of subclass: if Z is a subclass of A, then A
is a superclass of Z. This follows e.g. superset and subset. We don't
have any problem understanding that a class can have many subclasses. Why
the resistance to the idea that a class can have many superclasses?

Even in a single inheritance language, if we had a class hierarchy

A -> B -> C -> ... -> Y -> Z

it makes perfect sense to describe *all* of A-Y as superclasses of Z,
just as we describe all of B-Z as subclasses of A.

 

[Ian Kelly]

That is indeed confusing to some people, especially those who refuse to
to accept the notion that "superclass" means the same as "next in MRO",
maintaining instead that superclass refers to one of the base classes,
their bases, etc. -- IMO a defensible position.

super might have better been called next_in_mro, next_method, or
next_class, except those are harder to type, and way less catchy than
"super".  The Dylan and CLOS operator that super is most closely based
on is called (call-)next-method.

I have to chime in and agree that the name "super" is problematic.
I'm reading this thread with a sense of alarm because I apparently
never read the super() documentation too closely (why would I? "Oh,
it just accesses an attribute from a superclass. Moving on.") and
have been writing code for the past four years under the impression
that super() will always refer to a superclass of the current class.
Fortunately, I don't use multiple inheritance often, and when I do I
prefer to write the superclasses explicitly (perhaps because of the
same misconception), so I probably haven't really abused it terribly.

On a tangent, is it just me, or is the super() documentation
incorrect, or at least unclear? Quoting from the first two
paragraphs:

super(type[, object-or-type])

Return a proxy object that delegates method calls to a parent or
sibling class of type. This is useful for accessing inherited methods
that have been overridden in a class. The search order is same as that
used by getattr() except that the type itself is skipped.

The __mro__ attribute of the type lists the method resolution
search order used by both getattr() and super(). The attribute is
dynamic and can change whenever the inheritance hierarchy is updated.

In the first paragraph, "type" refers to "type", the first parameter
in the function signature. In the second paragraph, "type" refers to
the type instance or the type of the object passed as the second
argument. If it also referred to the first parameter, then super()
would always access a superclass as I initially thought; I wonder if
this might have been partially responsible for my confusion.

Cheers,
Ian

 

[Steven D'Aprano]

I have to chime in and agree that the name "super" is problematic. I'm
reading this thread with a sense of alarm because I apparently never
read the super() documentation too closely (why would I? "Oh, it just
accesses an attribute from a superclass. Moving on.") and have been
writing code for the past four years under the impression that super()
will always refer to a superclass of the current class.

In Python 2.x, super() doesn't know what the current class is. You have
to explicitly tell it. If you tell it a lie, surprising things will
happen.

Assuming you accurately tell it the current class, can you give an
example where super() doesn't refer to a superclass of the current class?



[...]

On a tangent, is it just me, or is the super() documentation incorrect,
or at least unclear? Quoting from the first two paragraphs:

Yes, it's a bit unclear, because it's a complex function for dealing with
a complicated situation. But if you have single inheritance, it's simple.
Anywhere you would write

class C(B):
def method(self, *args):
B.method(*args)

you can write

class C(B):
def method(self, *args):
super(C, self).method(*args)


and it will Just Work.

super(type[, object-or-type])

Return a proxy object that delegates method calls to a parent or
sibling class of type.

I think that the bit about sibling class refers to super(type, type2)
calls, rather than the super(type, instance) calls which I've been
discussing. I've never needed, and don't understand, the two type version
of super(), so I can't comment on it.

 

[Brian Victor]

Which class would that be?

I think I'm going to need an example that demonstrates what you mean,
because I can't make heads or tails of it. Are you suggesting that a call
to super(C, self).method() from within C might call B.method(self)?

Yes, it would.

class A(object):
def test_mro(self):
print "In A"

class B(A):
def test_mro(self):
print "In B"
super(B, self).test_mro()
raise Exception()

class C(A):
def test_mro(self):
print "In C"
super(C, self).test_mro()

class D(C, B):
def test_mro(self):
print "In D"
super(D, self).test_mro()

D().test_mro()

Notice the exception being raised in B. This results in the following traceback:

Traceback (most recent call last):
File "mro.py", line 21, in <module>
D().test_mro()
File "mro.py", line 19, in test_mro
super(D, self).test_mro()
File "mro.py", line 14, in test_mro
super(C, self).test_mro()
File "mro.py", line 9, in test_mro
raise Exception()
Exception

Since the idea of super() as I understand it is to make sure every class
in an object's hierarchy gets its method called, there's really no way
to implement super() in a way that didn't involve a non-superclass being
called by some class's super() call.

 

[Ian Kelly]

In Python 2.x, super() doesn't know what the current class is. You have
to explicitly tell it. If you tell it a lie, surprising things will
happen.

Assuming you accurately tell it the current class, can you give an
example where super() doesn't refer to a superclass of the current class?

Brian gave a good example, so I refer you to that.

Yes, it's a bit unclear, because it's a complex function for dealing with
a complicated situation. But if you have single inheritance, it's simple.
Anywhere you would write

class C(B):
   def method(self, *args):
       B.method(*args)

you can write

class C(B):
   def method(self, *args):
       super(C, self).method(*args)


and it will Just Work.

Until somebody else comes along and creates class E that inherits from
both C and D and also uses super(), and now suddenly the super() call
in C becomes equivalent to "D.method(*args)" instead for instances of
E, potentially with unexpected results. Or worse, E.method
incorrectly calls both C.method and D.method explicitly, and now
D.method gets invoked twice, once implicitly from C.method, and once
explicitly from E.method.

I realize that it is the responsibility of the person writing class E
to make sure they're working with class C correctly, but in order for
them to do that, ultimately every method in class C should be
documented as to whether it calls super() or not, and if so whether it
is designed only for single inheritance or with the MRO in mind. That
really shouldn't be necessary, and in my current view from a
maintenance perspective, best practice is to only use super() in the
multiple-inheritance scenarios it was specifically designed for

super(type[, object-or-type])

    Return a proxy object that delegates method calls to a parent or
sibling class of type.

I think that the bit about sibling class refers to super(type, type2)
calls, rather than the super(type, instance) calls which I've been
discussing. I've never needed, and don't understand, the two type version
of super(), so I can't comment on it.

I understand the two-type version works the same way, but is intended
for class methods, like so:

class A(object):
@classmethod
def f(cls, *args):
print "A.f"

class B(A):
@classmethod
def f(cls, *args):
print "B.f"
return super(B, cls).f(*args)

class C(A):
@classmethod
def f(cls, *args):
print "C.f"
return super(C, cls).f(*args)

class D(B, C):
@classmethod
def f(cls, *args):
print "D.f"
return super(D, cls).f(*args)

Cheers,
Ian

 

[Gregory Ewing]

Assuming you accurately tell it the current class, can you give an
example where super() doesn't refer to a superclass of the current class?

I think we're all confusing each other in this discussion
by not being clear enough about what we mean by the "current
class".

In a call super(C, self), there are two possible things it
could mean: the class C, or the type of self.

Obviously it will return some class in the mro of self,
but as Brian Victor's example demonstrates, it *doesn't*
necessarily return a class in the mro of C.

 

[Gregory Ewing]

super(type[, object-or-type])

...

The __mro__ attribute of the type lists the method resolution
search order used by both getattr() and super(). The attribute is
dynamic and can change whenever the inheritance hierarchy is updated.

That explanation does seem to be rather confusing. It would
certainly confuse *me* if I didn't already know what super()
did.

+1 on getting this part of the docs re-written to be several
times clearer.

 

[Carl Banks]

Brian gave a good example, so I refer you to that.









Until somebody else comes along and creates class E that inherits from
both C and D and also uses super(), and now suddenly the super() call
in C becomes equivalent to "D.method(*args)" instead for instances of
E, potentially with unexpected results.  Or worse, E.method
incorrectly calls both C.method and D.method explicitly, and now
D.method gets invoked twice, once implicitly from C.method, and once
explicitly from E.method.

I realize that it is the responsibility of the person writing class E
to make sure they're working with class C correctly, but in order for
them to do that, ultimately every method in class C should be
documented as to whether it calls super() or not, and if so whether it
is designed only for single inheritance or with the MRO in mind.  That
really shouldn't be necessary,

I was with you up to here.

It's *manifestly* necessary. One cannot inherit from a class reliably
without knowing how it's implemented. Knowing the interface is not
enough. This is not limited to multiple inheritance; it happens with
single inheritance as well, just less often. It's not limited to
Python, this is true of Java, C++, and every other OOP language I know
of.

Take list. Say you want a singly inherit from list to implement some
kind of observer that emits a signal every time an item is set. How
do you do it? You can't tell by looking at the interface. Do you
only have to override __setattr__? Or do you also have to override
other methods (insert, append, etc)? That depends on whether insert
et al. call __setattr__ or operate at a lower level. You can't tell
simply by looking at the interface. You have to have knowledge of the
internals of list.

It's the same problem with multiple inheritance, only exacerbated.
The issues with super is a new dimension, but it's the same problem:
you have to know what you're inheriting from.

This is a fundamental issue with all inheritance.

The problem, in my mind, is that people use inheritance for stuff that
doesn't need it.

class MyFoo(SomebodyElsesFoo):
def my_tacked_on_method(self):
self.call_this_method()
self.then_call_that_method()

Unnecessary and risky. Do this:

my_tacked_on_function(foo):
foo.call_this_method()
foo.then_call_that_method()

The only reason to inherit in a situation like this is if you need
polymorphism (you want to pass your MyFoo to a function that operates
on objects that define a my_tacked_on_method).

A lot Michele Simionato's objections to inheritance are based on the
idea that you don't necessarily know what you're inheriting from. I
say, if you don't know what you're inheriting from, you shouldn't be
inheriting. I say this as a strong proponent of inheritance and
multiple inheritance in particular. I believe it's the best way in
general for objects of different types to share large portions of
their behavior. But it has its place.

When you have a class you that don't anything about the implementation
of, that is NOT the place for inheritance. In that case you should
either use composition, or learn what it is you're inheriting from and
take responsibility if the owner changes it. Or just don't try to
share behavior.

My view of inheritance is a lot like the orthodox Catholic view of
sex. To Catholics, sex between married people is a wonderful, holy
thing. To me, inheriting from classes you know the implementation of
is a wonderful, holy thing. But, sex outside of marriage, and
inheriting from classes you don't know, is deadly sin and you will
burn in Hell.

and in my current view from a
maintenance perspective, best practice is to only use super() in the
multiple-inheritance scenarios it was specifically designed for

I use super() exclusively for all types of inheritance, unless I'm
inheriting from a class I know uses old form (which is none of mine,
and I only rarely inherit from classes I don't own).


Carl Banks

 

[Carl Banks]

When you have a class you that don't anything about the implementation
of, that is NOT the place for inheritance.

And, just to be clear, if the class is explicity documented as being
subclassable and the documentation states the proper procedure for
doing so, that counts as "knowing about the implementation"--even if
you don't know the exact implementation, you do have sufficient
knowledge.

I just don't want someone following up saying, "Well I think it's ok
to inherit from a class you know nothing about if it's documented
properly." I agree, and that is knowing something about it.


Carl Banks

 

[Steven D'Aprano]

Yes, it would.

[snip example]

Right, now I see what you mean. I don't have a problem with that
behaviour, it is the correct behaviour, and you are making the call from
D in the first place, so it *must* call B at some point.

If you initiate the call from C instead:
In C
In A

then B is not in C's MRO, and does not get called. This is the case I was
referring to.

I admit it did surprise me the first time I saw the example you gave.
Hell, it *confused* me at first, until I realised that self inside the
method isn't necessarily an instance of C, but could be an instance of a
subclass of C (in this case, D), and then it was "Well duh, how obvious!"
That's no different from a classic-style call to a superclass:

# inside class C(B):
B.method(self, *args)

Inside B, method() gets called with an instance of C as self.

Bringing it back to super(), since super(C, self) gets a D instance as
argument, the MRO it looks at is D's MRO, and it all just works. I think
the docs could make that a little more clear, but then, given how non-
cooperative inheritance works exactly the same way, there's a good
argument for saying the docs don't need to be changed at all. If the
behaviour is confusing for super(), then it's confusing without super()
too.

See Guido's tutorial on cooperative multitasking for more detail:

http://www.python.org/download/releases/2.2.3/descrintro/#cooperation

Since the idea of super() as I understand it is to make sure every class
in an object's hierarchy gets its method called, there's really no way
to implement super() in a way that didn't involve a non-superclass being
called by some class's super() call.

You're reading the super() signature wrong.

super(C, self)

shouldn't be interpreted as "call C's superclasses from self". It means
"starting just after C in the MRO, call self.__class__'s superclasses".

super() never calls a non-superclass. If it did, it would be a huge bug,
and inheritance would break.

 

[Gregory Ewing]

super(C, self)

shouldn't be interpreted as "call C's superclasses from self". It means
"starting just after C in the MRO, call self.__class__'s superclasses".

My contention is that nobody has any chance of guessing what
it does based on the name "super". Superness doesn't come into
it at all, except in the trivial sense that whatever class it
picks will be in the mro of the second argument, which doesn't
narrow it down much.

So the word "super" here is little better than meaningless noise.

If it were something like next_method(C, self) then one could
claim with some justification that it stands for "call the next
method found after C in the mro of self".

 

[Michele Simionato]

I have read Michelle Simionato's articles on super in Python.

One "l" please! I am a man! ;-)

But Michelle is wrong to conclude that the problem lies with the concept
of *superclass*. The problem lies with the idea that there is ONE
superclass. By dismissing the entire concept, he is throwing out the baby
with the bathwater.

I am actually more radical than that. From
http://www.artima.com/weblogs/viewpost.jsp?thread=237121:

"""
In this series I have argued that super is tricky; I think nobody can
dispute that. However the existence of dark corners is not a
compelling argument against a language construct: after all, they are
rare and there is an easy solution to their obscurity, i.e.
documenting them. This is what I have being doing all along. On the
other hand, one may wonder if all super warts aren't hints of some
serious problem underlying. It may well be that the problem is not
with super, nor with cooperative methods: the problem may be with
multiple inheritance itself.
"""

Then I spend thousands of words in the "Mixin considered harmful"
series and in other articles arguing against
multiple inheritance and cooperative methods. They are just bad design
IMO. Of course we are in the range of opinions, this is a tricky
subject: many smart people agree with me and many others equally smart
disagree.
Still I believe that super is a red herring and that you should really
start thinking: what advantages did
multiple inheritance *really* bring into my code? Could have I done
without? And what would have happen?
These are the relevant question, not the exact meaning of super in
hairy hierarchies.

M. Simionato

 

[Paul Rubin]

I am actually more radical than that. From
http://www.artima.com/weblogs/viewpost.jsp?thread=237121:
In this series I have argued that super is tricky; I think nobody can...

When I look at that URL, I see a Java stack dump:

java.lang.RuntimeException: com.jivesoftware.forum.ForumThreadNotFoundException: ID -1 is not valid
at com.artima.jivecoupled.skins.weblogs.ViewPostPage.process(ViewPostPage.java:112)
...

Seems appropriate.

 

[Jean-Michel Pichavant]

When I look at that URL, I see a Java stack dump:

java.lang.RuntimeException: com.jivesoftware.forum.ForumThreadNotFoundException: ID -1 is not valid
at com.artima.jivecoupled.skins.weblogs.ViewPostPage.process(ViewPostPage.java:112)
...

Seems appropriate.

remove the ending double dot.

JM

 

[Steven D'Aprano]

One "l" please! I am a man! ;-)

My apologies. You'd think I would know the difference between Michele and
Michelle


[snip]

Still I believe that super is a red herring and that you should really
start thinking: what advantages did
multiple inheritance *really* bring into my code? Could have I done
without? And what would have happen?
These are the relevant question, not the exact meaning of super in hairy
hierarchies.

Yes, these are very good points. It's always worth looking at
alternatives to subclassing in the first place. Far too often it's not
just the first idiom people think of, but the *only* idiom they think of.

 

[Mark Lawrence]

remove the ending double dot.

JM

aka the colon.

Cheers.

Mark Lawrence