questions (& answers) about object, type, builtin types, class,metaclass and __getattribute__

[Amirouche B.]

I'm learning a bit of python internals lately and I'm trying to figure
out the
relationship between type, objects, class, callables and
__getattribute__ resolution.

While understanding Python mechanics/concepts, I'm trying to figure
how it
translates in CPython. This post is Python centric. Questions about
the
implementation of this concepts might be the subject of a future post
[1].

I will proceed this way: I will write statements about each subject.
Don't hesitate
to pick one and provide more information or better phrasing, or
explaining why it's
not true.

Be aware that I'm considering only new-style class.


A) type vs object
-----------------

1) object is the base object, it has no bases : len(object.__bases__)
== 0
2) every object in python inherit object :
any_object_except_object.__bases__[-1] is object
3) object's type is type : object.__class__ is type
4) type parent object is object : type.__bases__ == (object,)


B) type vs metaclass
--------------------

1) type is the first metaclass ?
2) type is its own metaclass : type(type) is type ?
3) object's metaclass is type ?
4) other metaclasses *MUST* inherit type ?
5) type(any_object) == last_metaclass_..., which is, most of the time,
type ?


C) type vs class
----------------

1) Type is the metaclass of most classes
2) The class statement::

class MyClass(object):
attribute = 1

def method(self):
pass

translates to::

MyClass = type('MyClass', (object,), {'attribute': 1, 'method':
def method: pass })

3) Instantiation of any class ``MyClass(*args, **kwargs)`` translates
to::

type(MyClass).__call__(MyClass, *args, **kwargs)

This is due to __getattribute__ algorithm (see E)

4) It's in type.__call__ that happens calls to __new__ and __init__

5) 3) => classes are instance of type

6) Since type.__call__ is used to instantiate instance of instance of
type
(rephrased: __call__ is used to instantiate classes) where is the
code which
is executed when we write ``type(myobject)`` or ``type('MyClass',
bases, attributes)``
__getattribute__ resolution algorithm (see E) tells me that it
should be type.__call__
but type.__call__ is already used to class instatiation.


C') class vs class instances aka. objects
-----------------------------------------

1) A class type is a metaclass : issubclass(type(MyClass), type),
MyClass.__class__ == type(MyClass)
2) An object type is a class : most of the time
isinstance(type(my_object), type)
generally issubclass(type(type(my_object)), type)


D) builtin types
----------------

1) builtin types are their own metaclass ?
2) why function builtin type can not be subclassed ?
3) how does builtin function type relate to callable objects ?
4) int(1) is the same as int.__call__(1), since type(int) is type,
shouldn't int(1)
translates to type.__call__(int, 1) ?


E) __getattribute__
-------------------

1) ``my_object.attribute`` always translates to
``my_object.__getattribute__('attribute')``
2) Is the following algorithm describing __getattribute__ correct [2],
beware that I've
added a comment:

a) If attrname is a special (i.e. Python-provided) attribute for
objectname,
return it. # what does it mean to be Python-provided ?

b ) Check objectname.__class__.__dict__ for attrname. If it exists
and is a
data-descriptor, return the descriptor result. Search all bases of
objectname.__class__
for the same case.

c) Check objectname.__dict__ for attrname, and return if found.

d) If it is a class and a descriptor exists in it or its bases,
return the
descriptor result.

d) Check objectname.__class__.__dict__ for attrname. If it exists
and is a
non-data descriptor, return the descriptor result. If it
exists, and is
not a descriptor, just return it. If it exists and is a data
descriptor,
we shouldn't be here because we would have returned at point 2.
Search all
bases of objectname.__class__ for same case.

e) Raise AttributeError


Thanks in advance,

Regards,


Amirouche

[1] or maybe you can point me to the right direction before I ask
stupid questions.
I'm a bit familiar with Jython code, which seems to be easier than
PyPy and CPython
to read even if there is some magic due to the fact that Jython use
some JVM API
that hides somewhat how it works.
[2] this is a rewritten version of
http://www.cafepy.com/article/pytho...tes_and_methods.html#attribute-search-summary

 

[Steven D'Aprano]

A) type vs object

Correct, but for reference, a more direct test is:

object.__bases__ == ()

(no need for len).

2) every object in python inherit object :
any_object_except_object.__bases__[-1] is object

Excluding old-style objects, I believe you are correct.

3) object's type is type : object.__class__ is type
4) type parent object is object : type.__bases__ == (object,)

The relationship between type and object is somewhat special, and needs to
be bootstrapped by the CPython virtual machine.

Arbitrary types (classes) inherit from object. That means the type is a
subclass of object:
True


What's less obvious is that types are themselves objects, and therefore are
instances of object:
True


Since classes are objects, they have a type, namely ``type``.


This includes ``type`` itself:

* type is an instance of object
* object is an instance of type
* type is a subclass of object
* but object is NOT a subclass of type

B) type vs metaclass

Excluding old-style classes, yes, all custom classes (those you create with
the class statement) have a default metaclass of type.

2) type is its own metaclass : type(type) is type ?

Yes. Another bit of bootstrapping that the compiler does.

True


3) object's metaclass is type ?
Yes.


4) other metaclasses *MUST* inherit type ?

No. Metaclasses can be anything that mimics type.

.... class X(object):
.... pass
.... return X
........ __metaclass__ = meta
.... a = 1
....<class '__main__.X'>


They don't even need to return a type/class. Like decorators, they can
return anything.

.... return "spam"
........ __metaclass__ = meta
....

'spam'



5) type(any_object) == last_metaclass_..., which is, most of the time,
type ?

I'm not sure what you mean by "last_metaclass". But no. The type of an
object is its class:

type(42) => int
type("spam") => str
type(1.23) => float

However, the type of a class is *usually* type.

C) type vs class
----------------

1) Type is the metaclass of most classes
Yes.


2) The class statement::

class MyClass(object):
attribute = 1

def method(self):
pass

translates to::

MyClass = type('MyClass', (object,), {'attribute': 1, 'method':
def method: pass })

Except that the syntax won't work, the idea is broadly correct.

3) Instantiation of any class ``MyClass(*args, **kwargs)`` translates
to::

type(MyClass).__call__(MyClass, *args, **kwargs)

Like any function call, MyClass(...) becomes

type(MyClass).__call__(self, ...)

with self=MyClass. Since type(MyClass) is usually ``type``, that gives:

type.__call__(MyClass, ...)

This is due to __getattribute__ algorithm (see E)

4) It's in type.__call__ that happens calls to __new__ and __init__

If type were written in pure Python, it would probably look something like
this:

class Type(object):
# ...
# other methods
# ...
def __call__(cls, *args, **kwargs):
instance = cls.__new__(cls, *args, **kwargs)
if isinstance(instance, cls):
instance.__init__(*args, **kwargs)
return instance

But see further on, for more complication.

Note that __new__ is special-cased as a staticmethod, hence it needs the
first argument to be passed directly.

5) 3) => classes are instance of type

6) Since type.__call__ is used to instantiate instance of instance of
type
(rephrased: __call__ is used to instantiate classes) where is the
code which
is executed when we write ``type(myobject)`` or ``type('MyClass',
bases, attributes)``

You would need to check the C implementation of type, but if I were doing
this in pure Python, I'd have something like this:

class Type(object):
def __call__(self, *args, **kwargs):
if self is Type:
if kwargs:
raise TypeError('unexpected keyword arguments')
# calling type(...) directly
if len(args) == 1:
# Single argument call, like type(x)
return x.__class__
else:
# Like type(name, bases, dict)
name, bases, dict = *args
cls = Type.__new__(Type, name, bases, dict)
if isinstance(cls, Type):
cls.__init__(name, bases, dict)
else:
# called from MyClass(...)
# which becomes type(MyClass).__call__(MyClass, ...)
# self here equals MyClass, which is an instance of type but
# not type itself
instance = self.__new__(self, *args, **kwargs)
if isinstance(instance, self):
instance.__init__(*args, **kwargs)
return instance
def __new__(cls, *args):
# magic to actually create a new type object



Note that this may not be how ``type`` actually does it. See the source
code, and I hope you have better luck reading it than I did!

http://hg.python.org/cpython/file/c8e73a89150e/Objects/typeobject.c

 

[Stephen Hansen]

A) type vs object
-----------------

1) object is the base object, it has no bases : len(object.__bases__)
== 0
2) every object in python inherit object :
any_object_except_object.__bases__[-1] is object

Not exactly. Python has two somewhat different object models, "old style
classes" and "new style classes", with slightly different behavior and
internal structure.

class Foo: pass

is an "old-style class", dated back to Python's ancient past. This all
relates to the fact that 'type' and 'class' used to be two pretty
different things, where one was something you mostly did only in C, and
one was something you did (mostly) only in Python. They are largely the
same now.

3) object's type is type : object.__class__ is type
4) type parent object is object : type.__bases__ == (object,)

Saying "type" and "parent" and the like for new-style classes is
something of a misnomer. For "type" and "object", these things aren't
constructed like this.

What you have here is technically true if you go poke at it in the
interpreter, but it doesn't really /mean/ anything because its not how
these objects came to be and is circular and a bit confusing. These
fundamental objects are created special.

B) type vs metaclass

Type is the basic, default "metaclass", yes. A metaclass is a callable
that constructs class objects.

2) type is its own metaclass : type(type) is type ?

Only in a purely theoretical way. It doesn't actually mean anything;
moreover, type(something) is NOT how you determine somethings metaclass.
Its how you determine somethings type.

The two concepts may be very distinct. Lots of things don't have
metaclasses.

3) object's metaclass is type ?

Again, only theoretically.

4) other metaclasses *MUST* inherit type ?

Absolutely not. Any callable can be a metaclasss. Despite its name, it
doesn't have to be itself a class or anything. Just something you can
call with er, 3 (I forget exactly) arguments, and which returns a
constructed class object.

5) type(any_object) == last_metaclass_..., which is, most of the time,
type ?

Not necessarily at all. In fact, there is no way I'm aware of to
determine if a metaclass was involved in a classes construction unless
said metaclass wants to provide such a mechanism.

Metaclasses are kind of a hack. They are a way to hook into the class
construction that's normally done and do something, anything you want,
(even hijack the whole procedure and NOT construct a class at all, but
play a song if you want) before its all finished.

For example, this is a metaclass I've used:

PageTypes = {}

class _PageRegistration(type):
def __new__(cls, name, bases, dct):
klass = type.__new__(cls, name, bases, dct)
typename = name[:-9].lower()
if not typename:
typename = None

PageTypes[typename] = klass
klass.Type = typename

return klass

class QueuePage(sc.SizedPanel):
__metaclass__ = _PageRegistration

Note, the fact that my _PageRegistration metaclass inherits is itself a
class which inherits from type is just one convenient way to write
metaclasses. It could as simply have been just a function.

Metaclasses are somewhat poorly named in that they are really, "creation
hooks".

C) type vs class

Yes and no. Yes, in that most classes are created using the default
mechanism inside CPython. The class body is executed in a scope, the
resulting dictionary is bound to a new class object, bases and the like
are set, and such.

No in that it really just, IIUC, skips the whole "metaclass" part of the
process because this 'default mechanism' doesn't need to call out into
other code to do its job. At least, I think-- May be wrong here,
metaclasses are something of a dark voodoo and I'm not 100% entirely
familiar with the internal workings of CPython.

But functionally, a metaclass is the chunk of code responsible for the
actual physical construction of the class object.

2) The class statement::

class MyClass(object):
attribute = 1

def method(self):
pass

translates to::

MyClass = type('MyClass', (object,), {'attribute': 1, 'method':
def method: pass })

Translates to, I don't know about that. Is functionally equivalent, yes.
It is more or less what happens.

3) Instantiation of any class ``MyClass(*args, **kwargs)`` translates
to::

type(MyClass).__call__(MyClass, *args, **kwargs)

This is due to __getattribute__ algorithm (see E)
4) It's in type.__call__ that happens calls to __new__ and __init__

Again, "translates to" is suggesting "this is what happens when you do
X", which I don't know if is strictly true. CPython inside may be
optimizing this whole process. Especially when it comes to "magic
methods", __x__ and the like -- CPython rarely uses __get*_ for those.
It just calls the methods directly on the class object.

What you're doing here is functionally equivalent to what CPython does,
but it may take a more direct route to get there. CPython has the
advantage of not being Python and having direct access to internals.

5) 3) => classes are instance of type

6) Since type.__call__ is used to instantiate instance of instance of
type
(rephrased: __call__ is used to instantiate classes) where is the
code which
is executed when we write ``type(myobject)`` or ``type('MyClass',
bases, attributes)``
__getattribute__ resolution algorithm (see E) tells me that it
should be type.__call__
but type.__call__ is already used to class instatiation.

Python callables can have more then one argument, and more then one
behavior, and can choose to do more then one thing based on the number
of arguments.

They don't have overloading, but they can do it themselves.

You've already proven that, and you already know that -- you're sorta
over-complicating this

You said:

MyClass = type('MyClass', (object,), {'attribute': 1, 'method': def
method: pass })

Type.__call__ does both.

C') class vs class instances aka. objects

An old-style classe's type is classobj. A new-style class's type is
type. It may or may not be a "metaclass". type(foo) is not defined as
'the metaclass of foo', but 'the type of foo'. It often is the same
thing, since most of the time type(myclass) is "type" itself and "type"
is the default "metaclass". But mixing up the two concepts isn't really
very helpful.

2) An object type is a class : most of the time
isinstance(type(my_object), type)
generally issubclass(type(type(my_object)), type)
Generally.

D) builtin types

Builtin types are constructed by the interpreter, and may or may not
involve themselves with the class/type hierarchy. Even when they do,
they may only do so partially.

2) why function builtin type can not be subclassed ?

Because functions are not classes. They're discrete things of their own.
That a function object has a __class__ attribute is an example of a
builtin partially involving itself in the class/type hierarchy, largely
just for introspection purposes-- since functions are first-class
citizens, you need to be able to pass it around and test to determine
what one thing or another actually is.

Now, functions ARE indeed PyObjects, the internal CPython representation
of an object, of which all things are. But that's not really the same
thing as ultimately inheriting from and involving itself in "object" and
types/classes.

3) how does builtin function type relate to callable objects ?

It doesn't, really.

4) int(1) is the same as int.__call__(1), since type(int) is type,
shouldn't int(1)
translates to type.__call__(int, 1) ?

You're overly focused on __call__, for one thing. For another, no, I'm
not sure why you think one should "translate" to another (again: just
because you've got the basic relationship of the meta-parts of Python
down pat, doesn't mean Python internals actually goes through all those
actual steps when doing things).

int(1) translates into:

int.__call__(1)
int_inst = int.__new__(1)

More or less. Now, "int" is itself a "class" (type), so it was, itself,
constructed by a call to type (well, "type"'s internal C version of
itself, Py_TYPE).

type, the metaclass or class-constructor version, is only called to
create /classes/, at class-creation-time. After "int" itself exists, its
no longer involved.

int has its own __new__ that's used to construct actual instances of
itself. type is now out of the picture.

E) __getattribute__

No. __getattribute__ is a mechanism to hook into the attribute fetching
mechanism. (As is __getattr__). It is NOT always invoked.

Especially in the case of __*__ attributes, which by and large bypass
such hooks, as the CPython internals is calling those functions directly
on the class instances themselves.

2) Is the following algorithm describing __getattribute__ correct

This is broadly incorrect because it implies that __getattribute__ is an
internal protocol that Python uses for attribute-resolution, which is
simply untrue. Its a method you may define on new style classes which,
if present, is called when an attribute is requested from an object (but
NOT in the case of __*__ methods, usually, which are obtained internally
by a direct struct access, i.e., mytype->tp_new gets mytype.__new__).

If no such attribute exists, it goes along to do its default
attribute-resolution process, including the descriptor protocol and dict
checking and the like.

__getattribute__ is an optional hook that you can define which allows a
Python class to /bypass/ the normal mechanism for normal (non-magic)
attributes.

If you're asking what the normal mechanism is, its broadly:

- Check to see if the object's base-classes have a descriptor of the
attributes name. If so, call that.
- Check to see if the object's instance dict has an attribute of the
name. If so, return that.
- Check to see if the object's base-classes have an attribute of the name.

More or less. I think. I'm probably leaving something out there.

--

Stephen Hansen
... Also: Ixokai
... Mail: me+list/python (AT) ixokai (DOT) io
... Blog: http://meh.ixokai.io/


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iQEcBAEBAgAGBQJOUni4AAoJEKcbwptVWx/lZ9IH/RNO25T/HnXs3ly9wzHHvSG1
2Ms4kQTVeynGUT7ER5ZTgH6/rXRMEd6PCk03KMAkHSNFlhpGYUKsLnvwnxfq1gBH
4FczuL9du6doqlHkdvLw42Bi0tOzmUYHqJwMQajZZLu15LXa1UPpKROqR661DI1/
FSQQiqE/CAbS3KyPxID9o18RGIBQQICYZBkBWwDRhBidWEOQ54ktovAsBhCMuzg7
O/Ph9Fowj4no3VyvDaCJDscj2AN9UT8iRMMbDtZlb7QeSpo5AWZ4HiYRwIHjkZXB
XlTXn2eZCHZ5TX71SHSlj20bFnCXmdDrWCsNS6eGFtbltlS/W7iFR/+ZNoJEnCw=
=I4LK
-----END PGP SIGNATURE-----

 

[Chris Angelico]

Not exactly. Python has two somewhat different object models, "old style
classes" and "new style classes", with slightly different behavior and
internal structure.

  class Foo: pass

is an "old-style class", dated back to Python's ancient past. This all
relates to the fact that 'type' and 'class' used to be two pretty
different things, where one was something you mostly did only in C, and
one was something you did (mostly) only in Python. They are largely the
same now.

And "now" includes everything in Python 3, where a class implicitly
derives from object if no other subclassing is given.

ChrisA

 

[Amirouche B.]

The relationship between type and object is somewhat special, and needs to
be bootstrapped by the CPython virtual machine.

Since you are talking about CPython, I'm wondering how it is
bootstraped since you can easly reference PyType in PyObject that part
is not hard.

Yes. Another bit of bootstrapping that the compiler does.

self reference is easy same as referencing PyType from PyObject and
PyObject from PyType.

I'm not sure what you mean by "last_metaclass". But no. The type of an
object is its class:

see this code for example proove my point:

class meta_a(type):
def __new__(cls, *args, **kwargs):
return type.__new__(cls, *args, **kwargs) # see (¤)


class meta_b(meta_a):
def __new___(cls, *args, **kwargs):
return meta_a.__new__(cls, *args, **kwargs) # same as above


class ClassWithTypeMetaA(object):
__metaclass__ = meta_a


class ClassWithTypeMetaB(object):
__metaclass__ = meta_b


type(ClassWithTypeMetaA) == meta_a
type(ClassWithTypeMetaB) == meta_b


[¤] super call doesn't work here, anyone can say why ?

Regards,

Amirouche

 

[Amirouche B.]

Saying "type" and "parent" and the like for new-style classes is
something of a misnomer. For "type" and "object", these things aren't
constructed like this.

What you have here is technically true if you go poke at it in the
interpreter, but it doesn't really /mean/ anything because its not how
these objects came to be and is circular and a bit confusing. These
fundamental objects are created special.

The code snippet is here to illustrate how it is visible in the
interpreter. But
you are right.

Only in a purely theoretical way. It doesn't actually mean anything;
moreover, type(something) is NOT how you determine somethings metaclass.
Its how you determine somethings type.

see the answer to Steven D'Aprano. type(class_object) ==
a_meta_class_object

The two concepts may be very distinct. Lots of things don't have
metaclasses.

All object in new style class have a metaclass at least type.

Again, only theoretically.

and again the famous "bootstrapping" make it like type created object.
From the
outside world of the Python implementation object looks like a type
instance.

5) type(any_object) == last_metaclass_..., which is, most of the time,
type ?

Not necessarily at all. In fact, there is no way I'm aware of to
determine if a metaclass was involved in a classes construction unless
said metaclass wants to provide such a mechanism.

Metaclasses are kind of a hack. They are a way to hook into the class
construction that's normally done and do something, anything you want,
(even hijack the whole procedure and NOT construct a class at all, but
play a song if you want) before its all finished.

For example, this is a metaclass I've used:

PageTypes = {}

class _PageRegistration(type):
def __new__(cls, name, bases, dct):
klass = type.__new__(cls, name, bases, dct)
typename = name[:-9].lower()
if not typename:
typename = None

PageTypes[typename] = klass
klass.Type = typename

return klass

class QueuePage(sc.SizedPanel):
__metaclass__ = _PageRegistration

Note, the fact that my _PageRegistration metaclass inherits is itself a
class which inherits from type is just one convenient way to write
metaclasses. It could as simply have been just a function.

Metaclasses are somewhat poorly named in that they are really, "creation
hooks".

It the same issue in django, views are only function, until you need
complex
behavior and you want a "namespace" to put everything in it. IMO
that's why class
based views exists for complex cases. That said being able to declare
a metaclass
only as a functions is neat.

Yes and no. Yes, in that most classes are created using the default
mechanism inside CPython. The class body is executed in a scope, the
resulting dictionary is bound to a new class object, bases and the like
are set, and such.

No in that it really just, IIUC, skips the whole "metaclass" part of the
process because this 'default mechanism' doesn't need to call out into
other code to do its job. At least, I think-- May be wrong here,
metaclasses are something of a dark voodoo and I'm not 100% entirely
familiar with the internal workings of CPython.

But functionally, a metaclass is the chunk of code responsible for the
actual physical construction of the class object.

For me it takes some variables, namely ``bases``, ``class_dict`` and
``configuration class_name`` and do something with it, probably
creating a
class_object which behaviour is parametred with the context. I did not
know
Python before new-style class, so probably for most people explainning
that
metaclasses are a creation hook is easier for them...

Again, "translates to" is suggesting "this is what happens when you do
X", which I don't know if is strictly true. CPython inside may be
optimizing this whole process.
Especially when it comes to "magic
methods", __x__ and the like -- CPython rarely uses __get*_ for those.
It just calls the methods directly on the class object.

IIUC the code of Jython tells me what I've written. If the first part
of the algorithm
is "lookup for special methods" (what you seem to say) then we both
agree that we
agree, isn't it ?

Moreover I'm not looking in this part to understand how CPython works
internally,
but how Python works. Since I'm most proeffencient in Python I
translate it to
Python.

*Translates* means "it's a shortcut for".

Python callables can have more then one argument, and more then one
behavior, and can choose to do more then one thing based on the number
of arguments.

They don't have overloading, but they can do it themselves.

You've already proven that, and you already know that -- you're sorta
over-complicating this
You said:

MyClass = type('MyClass', (object,), {'attribute': 1, 'method': def
method: pass })

Type.__call__ does both.

Actually I missed that :/

Builtin types are constructed by the interpreter, and may or may not
involve themselves with the class/type hierarchy. Even when they do,
they may only do so partially.

That's the part I don't understand. How can they possibly work
differently. They
surrely make their bussiness in the infrastructure/algorithm already
in place in
type and object to but I cant spot the right piece of code that will
explain it.

Because functions are not classes. They're discrete things of their own.
That a function object has a __class__ attribute is an example of a
builtin partially involving itself in the class/type hierarchy, largely
just for introspection purposes-- since functions are first-class
citizens, you need to be able to pass it around and test to determine
what one thing or another actually is.

Now, functions ARE indeed PyObjects, the internal CPython representation
of an object, of which all things are. But that's not really the same
thing as ultimately inheriting from and involving itself in "object" and
types/classes.

Coming with general idea about what are builtin type would make it
easier to
understand function objects.

What I can tell is that function objects behave like the instances of
a class, and
the class takes (at least) runnable code as arguments.

No. __getattribute__ is a mechanism to hook into the attribute fetching
mechanism. (As is __getattr__). It is NOT always invoked.

Especially in the case of __*__ attributes, which by and large bypass
such hooks, as the CPython internals is calling those functions directly
on the class instances themselves.

see my answer in the other response