Confused with classmethods

[Diez B. Roggisch]

I understand that this is a very peculiar use of

classmethods but is this error intentional?
Or did I completely missed the point somewhere?

A little bit: classmethods are defined in a class context.

def foo(cls):
print cls

class A:
foo = classmethod(foo)

The error you observe seems to be a result of your "abuse" of classmethod
outside a class scope.

 

[Diez B. Roggisch]

Not necessarily:

def foo(cls):
print cls
f=classmethod(foo)

class A: pass

A.f = f
a=A()
a.f()

Ahhh, yes, I see the minor difference - I didn't recognize it before.

This works. Anyway, the confusion starts from the documentation of
classmethod(). Since classmethod is a *function* of builtins we can
invoke it as such from wherever we want.

Sure. AFAIK it is actually a descriptor. From what I understand, when
calling a descriptor on an instance it will invoke it's __get__ method and
pass the instance as first argument. So classmethod just gets the
__class__ from that argument and passes it to its callable together with
the rest of the arguments.

But this magic seems only to work if the descriptor has been fetched from a
classes __dict__, not the instance __dict__. Hmm. Nice.

Moreover the documentation
sais that if the first argument is an instance, its class will be
used for the classmethod. OTOH, "class scope" is not a real thing in
python. It is just some code which is executed and then we get its
locals and use it on Class(localsDict, BasesTuple, ClassName) to make
a new class, it seems. So we can create a classmethod in any scope
and then just attach it to a class's dictionary.

I'd still call the code executed inside a class statement block a "scope" -
for example in a def-statement the scope is the current frame of execution,
so

def foo():
bar = "baz"

makes the bar part of the frames local variables. Scopes just exchange or
stack the dicts for name lookup.

 

[Steve Holden]

Absolutely. In fact here is another interesting case which can lead to
nice recipies:

class B:
print "Making class B!"
if blah:
def f(self): print self
else
def f(self): raise StopIteration
print "My locals are:",locals(),"and I will make a class from them"



Well, actually, to put things right, in python there are only *two*
real scopes: global scope and local scope.

Not strictly true. There's builtins right at the back, and nowadays
inside a nested function you have a nested set of local scopes.

sholden@bigboy ~
$ cat test88.py
#
# Namespace testing
#
x = 33
y = 44

def f1():
def f2():
y = 33
print "f2", x, y
x = 22
print "f1", x, y
f2()
print "f1", x, y
print "main", x, y
f1()
print "main", x, y


sholden@bigboy ~
$ python test88.py
main 33 44
f1 22 44
f2 22 33
f1 22 44
main 33 44

sholden@bigboy ~

I remember that when I was a newbie I was confused by this.
The thing is that there is no nesting of scopes in reality.
Really?

Just to help other newbies avoid the confusion, I believe it would
be better to say, from the start, that:

- There definitelly is no nesting of scopes for if/for/while/try
blocks.
Correct.

- There is no nesting of scopes when nested functions reference
stuff from the enclosing function. It looks like there is but
there isn't because then we should be able to say:

exec "some code" in locals(), cellvars(), globals()

which we can't and proves that referencing variables from the enclosing
function is indeed a questionable feature. It all happens because the
parser detects that there are variables with that name in the enclosing
functions and creates special cellvars...

I believe you are coinfusing restrictions on exec syntax with
restrictions on namespace nexting.

- No nesting of scopes for classes because we *have* to use 'self'
(which is a good thing IMHO).

regards
Steve

 

[Ruud de Jong]

jfj schreef:Note that this is not really related to classmethods. A similar
"problem" exists if you want to use an ordinary function as a method:
print "argument: " + str(self)

pass

<function g at 0x00B9F070>

Here you see that none of the class magic has been performed; a.x
is *not* a method, but an ordinary function. It has absolutely no
relation to / knowledge of the class that a belongs to.
Calling a.x() leads to an exception:

Traceback (most recent call last):
File "<pyshell#53>", line 1, in -toplevel-
a.x()
TypeError: g() takes exactly 1 argument (0 given)

And of course, if you supply an argument, everything works fine:
argument: 1

To get the method behaviour the function needs to be associated
with the class:

The mechanism is basically as follows (with thanks to Alex Martelli's
execellent Python in a Nutshell, Chapter 5). When a.x is being resolved,
first a.__dict__['x'] is tried. In the first case x was indeed defined
in a.__dict__, and its value was returned (the function also known as
g). No class magic at all. The same happened with your
classmethod foo: you get a bare classmethod, and apparently these
are not callable.
In the second case, a.__dict__ did not have a key 'x'. The lookup then
continues in a's class: A.__dict__['x']. Again it finds the function.
But when a class __dict__ lookup returns a function, the result is
first wrapped into a bound or unbound method object (depending on
whether you arrived there via a.x or A.x).

So far for *how* it works. As to *why* it works like this, I don't know
for sure. But my guess is that the reasoning was something as follows:
if you define a function (regular or something special like a
classmethod) only for an instance of a class, you obviously don't
want to use it in a class context: it is -by definition- invisible to
the class, or to other instances of the same class.
One possible use case would be to store a callback function.
And in that case you definitely don't want the class magic to happen
when you reference the function.

Just my 2 cents

Regards,

Ruud

 

[jfj]

Hi.

Suppose this:

########################

def foo (x):
print x

f = classmethod (foo)

class A: pass

a=A()
a.f = f

a.f()
# TypeError: 'classmethod' object is not callable!

#######################

I understand that this is a very peculiar use of
classmethods but is this error intentional?
Or did I completely missed the point somewhere?

j.

 

[jfj]

A little bit: classmethods are defined in a class context.

def foo(cls):
print cls

class A:
foo = classmethod(foo)

The error you observe seems to be a result of your "abuse" of classmethod
outside a class scope.

Not necessarily:

def foo(cls):
print cls
f=classmethod(foo)

class A: pass

A.f = f
a=A()
a.f()


This works. Anyway, the confusion starts from the documentation of
classmethod(). Since classmethod is a *function* of builtins we can
invoke it as such from wherever we want. Moreover the documentation
sais that if the first argument is an instance, its class will be
used for the classmethod. OTOH, "class scope" is not a real thing in
python. It is just some code which is executed and then we get its
locals and use it on Class(localsDict, BasesTuple, ClassName) to make
a new class, it seems. So we can create a classmethod in any scope
and then just attach it to a class's dictionary.


jf

 

[jfj]

I'd still call the code executed inside a class statement block a "scope" -
for example in a def-statement the scope is the current frame of execution,
so

def foo():
bar = "baz"

Absolutely. In fact here is another interesting case which can lead to
nice recipies:

class B:
print "Making class B!"
if blah:
def f(self): print self
else
def f(self): raise StopIteration
print "My locals are:",locals(),"and I will make a class from them"

makes the bar part of the frames local variables. Scopes just exchange or
stack the dicts for name lookup.

Well, actually, to put things right, in python there are only *two*
real scopes: global scope and local scope.

I remember that when I was a newbie I was confused by this.
The thing is that there is no nesting of scopes in reality.

Just to help other newbies avoid the confusion, I believe it would
be better to say, from the start, that:

- There definitelly is no nesting of scopes for if/for/while/try
blocks.

- There is no nesting of scopes when nested functions reference
stuff from the enclosing function. It looks like there is but
there isn't because then we should be able to say:

exec "some code" in locals(), cellvars(), globals()

which we can't and proves that referencing variables from the enclosing
function is indeed a questionable feature. It all happens because the
parser detects that there are variables with that name in the enclosing
functions and creates special cellvars...

- No nesting of scopes for classes because we *have* to use 'self'
(which is a good thing IMHO).


jf

 

[jfj]

So far for *how* it works. As to *why* it works like this, I don't know
for sure. But my guess is that the reasoning was something as follows:
if you define a function (regular or something special like a
classmethod) only for an instance of a class, you obviously don't
want to use it in a class context: it is -by definition- invisible to
the class, or to other instances of the same class.
One possible use case would be to store a callback function.
And in that case you definitely don't want the class magic to happen
when you reference the function.

Yep. Got it. Indeed the reason seems to be a valid optimization:
-in 99% of the cases you request something from an instance it is a
plain old variable
-in 99% of the cases you request something from a class it's a
function.

So it would be a waste of time to check for the conversion when
something exists in the __dict__ of the instance, indeed.


OTOH, I'm talking about the "concept of python" and not CPython
implementation, and that's why I have these questions


Thanks,

jfj

 

[Ruud de Jong]

jfj schreef:

Yep. Got it. Indeed the reason seems to be a valid optimization:
-in 99% of the cases you request something from an instance it is a
plain old variable
-in 99% of the cases you request something from a class it's a
function.

So it would be a waste of time to check for the conversion when
something exists in the __dict__ of the instance, indeed.


OTOH, I'm talking about the "concept of python" and not CPython
implementation, and that's why I have these questions


Thanks,

jfj

I love this type of discussion -- it really forces me to do research
on my assumptions. But in this case I don't believe that this is
specific to CPython. From the Python Reference Manual, Section 3.2,
"The Standard Type Hierarchy", in the part about callable types,
user-defined methods:

<quote>
Note that the transformation from function object to (unbound or bound)
method object happens each time the attribute is retrieved from the
class or instance. In some cases, a fruitful optimization is to assign
the attribute to a local variable and call that local variable. Also
notice that this transformation only happens for user-defined functions;
other callable objects (and all non-callable objects) are retrieved
without transformation. It is also important to note that user-defined
functions which are attributes of a class instance are not converted to
bound methods; this only happens when the function is an attribute of
the class.
</quote>

So it seems that this mechanism is indeed fully part of the python
concept, not an optimization specific for CPython implementation.
When you define a function as an instance attribute (instead of
a class attribute), then none of the class magic happens.
When you access that attribute, you get the bare function, not a method.

 

[Bengt Richter]

Hi.

Suppose this:

########################

def foo (x):
print x

f = classmethod (foo)

class A: pass

a=A()
a.f = f

a.f()
# TypeError: 'classmethod' object is not callable!

#######################

I understand that this is a very peculiar use of
classmethods but is this error intentional?
Or did I completely missed the point somewhere?

Yes and yes ;-)

classmethod creates an an object that wraps the function you pass it.
The classmethod object has a __get__ method that gets control instead of the __get__
method of a function, when the object is found by way of attribute name
search looking in a new style class or base class. However, you created a 'classic'
class (A) instance a=A(), and the rules for looking up attributes on
classic class instances are different from the rules for newstyle class instances,
(though the difference does not come into play for your particular example ;-)
<classmethod object at 0x02E81374>

getattr(type(a), 'f') is checked first, but nothing is found. In that case,
a.f is simply retrieved with no special effects.

If you want to force the 'special effect' you could do it like:
<bound method classobj.foo of <class __main__.A at 0x02EE792C>>

Now the result of that _is_ callable:
__main__.A

The normal way is to make the method callable accessible as an attribute of the class:
<classmethod object at 0x02E81374>
The object is still retrieved from the instance (and would be for newstyle instance as well,
though you can create a descriptor other than classmethod than can prevent this).
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'classmethod' object is not callable

It's not callable, because to become a callable, its __get__ method has to be called,
which will return the actual callable -- which will not be the function itself, but
a callable that holds a reference to both the function and the thing it's "bound" to
to form a bound method of one kind or another.

If we delete the instance attribute, the search for 'f' will go to the instance's class:
<bound method classobj.foo of <class __main__.A at 0x02EE792C>>

and the automated magic happens, and what is returned _is_ callable: __main__.A

If you want to delve into it, read
http://www.python.org/2.2.2/descrintro.html
and associated info.

If you wanted to make plain foo a bound method of A, you could do it manually:
<bound method A.foo of <__main__.A instance at 0x02EF142C>>

which you can call, and it will print the first argument (named x instead of self in this case):<__main__.A instance at 0x02EF142C>

If you pass None instead of the instance, you get an unbound method:<unbound method A.foo>

Which you can call with an instance: <__main__.A instance at 0x02EF18CC>

If you don't pass the type, some things won't work fully, but the usual will <bound method ?.foo of <__main__.A instance at 0x02EF142C>>

As can be seen, the original function is in there in the bound method:
<function foo at 0x02EE8B54>

And you can call it as such if you like: hi, via plain function

Descriptors are a big deal in the new python. They are at the root
of much of the magic.

Regards,
Bengt Richter