Python's super() considered super!

[Stefan Behnel]

Steven D'Aprano, 27.05.2011 18:06:

Why?

I think Sturla is referring to the "compile time" bit. CPython cannot know
that the builtin super() will be called at runtime, even if it sees a
"super()" function call.

CPython doesn't evaluate the super call at compile time, it only keeps a
reference to the surrounding class in the code object of the method. So
super() without arguments basically inherits the class argument from the
context the method was found in at compile time. This has two quirks:

1) Copying a method from one class to another keeps the original context.
So the class argument to super() is basically fixed at compile time,
regardless of the class the method will be executed on at runtime.

2) The class is only kept as a reference when CPython sees a function call
that looks like "super" at compile time, which isn't much more than a
heuristic.

The PEP doesn't mention the first issue, but it is actually explicit about
the second issue:

http://www.python.org/dev/peps/pep-3135/

"""
While super is not a reserved word, the parser recognizes the use of super
in a method definition and only passes in the __class__ cell when this is
found. Thus, calling a global alias of super without arguments will not
necessarily work.
"""

And the prove:
... def test(self): print('TEST')
...... def meth(self): _super().test()
...Traceback (most recent call last):
SystemError: super(): __class__ cell not found

I assume this is done in order to reduce the chance of accidentally keeping
a class object alive for eternity, only because a method was originally
defined therein that inherits the class reference in its code object. So
it's a tradeoff between memory overhead and usability issues.

While I think that the tradeoff is generally ok, I agree with Sturla that a
keyword would have been the correct solution, whereas this is a clear
"works only in the common cases" approach.

Stefan

 

[Ethan Furman]

I suspect the larger issue is that Multiple Inheritance is complex, but
folks don't appreciate that. Ask anyone about meta-classes and their
eyes bug-out, but MI? Simple! NOT.

On the other hand, perhaps the docs should declare that the built-in
objects are not designed for MI, so that that, at least, would be one
less bug waiting to happen.

~Ethan~

 

[sturlamolden]

I think Sturla is referring to the "compile time" bit. CPython cannot know
that the builtin super() will be called at runtime, even if it sees a
"super()" function call.

Yes. And opposite: CPython cannot know that builtin super() is not
called,
even if it does not see the name 'super'. I can easily make foo()
alias super().

In both cases, the cure is a keyword -- or make sure that __class__
is always defined.

If super is to be I keyword, we could argue that self and cls should
be
keywords as well, and methods should always be bound. That speaks in
favour
of a super() function. But then it should always be evaluated at run-
time,
without any magic from the parser.

Magic functions belong in Perl, not Python.

Sturla

 

[Ryan Kelly]

I really don't like the Python 2 syntax of super, as it violates
the DRY principle: Why do I need to write super(type(self),self)
when super() will do? Assuming that 'self' will always be named
'self' in my code, I tend to patch __builtins__.super like this:

import sys
def super():
self = sys._getframe().f_back.f_locals['self']
return __builtins__.super(type(self),self)

This way the nice Python 3.x syntax can be used in Python 2.x.

Oh dear, you haven't thought this one through.

...snip...
... infinite recursion follows ...

Oops. There's a reason why Python 2 requires you to be explicit about
the class; you simply cannot work it out automatically at run time.
Python 3 fixes this by working it out at compile time, but for Python 2
there is no way around it.

Oh? There's not much that can't be done at runtime if you're willing to
work hard enough. Let me propose the following awful awful hack:


import sys

_builtin_super = __builtins__.super

_sentinel = object()

def _auto_super(typ=_sentinel,type_or_obj=_sentinel):
"""Automagically call correct super() at runtime"""
# Infer the correct call if used without arguments.
if typ is _sentinel:
# We'll need to do some frame hacking.
f = sys._getframe(1)
# Get the first positional argument of the function.
type_or_obj = f.f_locals[f.f_code.co_varnames[0]]
# Get the MRO for investigation
try:
mro = type_or_obj.__mro__
except AttributeError:
try:
mro = type_or_obj.__class__.__mro__
except AttributeError:
raise RuntimeError("super() used with old-style class")
# Now, find the class owning the currently-executing method.
for typ in mro:
for meth in typ.__dict__.itervalues():
if not isinstance(meth,type(_auto_super)):
continue
if meth.func_code is f.f_code:
# Aha! Found you.
break
else:
continue
break
else:
raise RuntimeError("super() called outside a method")
# Now just dispatch to builtin super.
if type_or_obj is not _sentinel:
return _builtin_super(typ,type_or_obj)
return _builtin_super(typ)


Now, try is with the following:

class Base(object):
def hello(self,msg):
print "hello", msg

class Sub1(Base):
def hello(self,msg):
print "gunna say it"
super().hello(msg)

class Sub2(Base):
def hello(self,msg):
print "yes I am"
super().hello(msg)

class Diamond(Sub1,Sub2):
def hello(self,msg):
print "here we go..."
super().hello(msg)

d = Diamond()
d.hello("autosuper!")


And you get the expected output:

here we go...
gunna say it
yes I am
hello autosuper!


There may well be some cases where this breaks down, but it seems to do
the right thing in simple cases.


Not that I'm recommending anyone use this, of course...




Ryan


--
Ryan Kelly
http://www.rfk.id.au | This message is digitally signed. Please visit
(e-mail address removed) | http://www.rfk.id.au/ramblings/gpg/ for details


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAk3gOoEACgkQfI5S64uP50pJIgCgne7bgIBGxOIBhXXmrjrwrbEU
S/oAn3cVvzDO7D2QOKy5jpy5N7jsBSRG
=FYYp
-----END PGP SIGNATURE-----

 

[sturlamolden]

Why? The fault is not that super is a function, or that you monkey-
patched it, or that you used a private function to do that monkey-
patching. The fault was that you made a common, but silly, mistake when
reasoning about type(self) inside a class.

That was indeed a silly mistake, but not what I am talking about.
See Stefan's reponse.


Sturla

 

[Thomas Rachel]

Am 28.05.2011 01:57 schrieb sturlamolden:

Yes. And opposite: CPython cannot know that builtin super() is not
called,
even if it does not see the name 'super'. I can easily make foo()
alias super().

Another alternative would have been to make use of __xxx magic.

If every class had an "automatically available" attribute, e. g.
__<classname>_classname which thus could be accessed via __classname
from inside, keeping the 2.x syntax would have been the best, using
super(__classname, self).

In both cases, the cure is a keyword -- or make sure that __class__
is always defined.

If super is to be I keyword, we could argue that self and cls should
be keywords as well, and methods should always be bound. That speaks in
favour of a super() function. But then it should always be evaluated at run-
time, without any magic from the parser.

Magic functions belong in Perl, not Python.

ACK.


Thomas