Bug? If not, how to work around it?

[Gonçalo Rodrigues]

Hi,

Consider the following class:
.... def __init__(self, obj):
.... self.__obj = obj
.... def __getattr__(self, name):
.... return getattr(self.__obj, name)
....

Now:

a = Test([])
a.__iter__

iter(a)

Traceback (most recent call last):

Is this a bug? If not, how to code Test such that iter sees the
__iter__ of the underlying object?

With my best regards,
G. Rodrigues

P.S: Tested with the latest 2.3 on win2k.

 

[Mark Day]

... def __init__(self, obj):
... self.__obj = obj
... def __getattr__(self, name):
... return getattr(self.__obj, name)
...

Now:

a = Test([])
a.__iter__

iter(a)

Traceback (most recent call last):

Is this a bug? If not, how to code Test such that iter sees the
__iter__ of the underlying object?

I'm guessing that iter() is looking for an __iter__ attribute without
going through __getattr__ to find it. So, I tried adding the following
method to the Test class:

def __iter__(self):
return self.__obj.__iter__

but that returned the following error:

TypeError: iter() returned non-iterator of type 'method-wrapper'

I changed the __iter__ method to the following, and it seems to do what
you want:

def __iter__(self):
return iter(self.__obj)

As to whether this is a bug, I don't know.

-Mark

 

[Aahz]

... def __init__(self, obj):
... self.__obj = obj
... def __getattr__(self, name):
... return getattr(self.__obj, name)
...

a = Test([])
a.__iter__

iter(a)

Traceback (most recent call last):

Is this a bug? If not, how to code Test such that iter sees the
__iter__ of the underlying object?

As Mark guessed, iter() goes directly to the attribute rather than using
the __getattr__ machinery of the class. However, you can intercept it
using a metaclass, but that requires a bit of fancy footwork to reach
down into the instance to get self.__obj.

 

[Gonçalo Rodrigues]

If you don't want to do what works,
why ask us to bother answering?

Because it may not work. That is, the use case I have in mind serves
as proxy for an object that may or may not have __iter__. IOW if I
stick __iter__ I have to code a second class, etc... etc... Oh well...

TJR

With my best regards,
G. Rodrigues

 

[Gonçalo Rodrigues]

class Test(object):

... def __init__(self, obj):
... self.__obj = obj
... def __getattr__(self, name):
... return getattr(self.__obj, name)
...

a = Test([])
a.__iter__

iter(a)

Traceback (most recent call last):

Is this a bug? If not, how to code Test such that iter sees the
__iter__ of the underlying object?

As Mark guessed, iter() goes directly to the attribute rather than using
the __getattr__ machinery of the class. However, you can intercept it
using a metaclass, but that requires a bit of fancy footwork to reach
down into the instance to get self.__obj.

Actually I came across this because of a little metaclass I was
coding. And my head hurts already as it is...

So, is it a bug? In other words, should I open a bug report?

With my best regards,
G. Rodrigues

 

[Steven Taschuk]

Quoth Mark Day:
[...]

I'm guessing that iter() is looking for an __iter__ attribute without
going through __getattr__ to find it. [...]

Right. This is normal for special methods when invoked by special
notations (e.g., __len__ invoked by way of len(), __add__ invoked
by way of +, etc.).

[...] So, I tried adding the following
method to the Test class:

def __iter__(self):
return self.__obj.__iter__

Well, you have to actually call __iter__ to get the iterator:

def __iter__(self):
return self.__obj.__iter__()

(This makes iter(testobj) not quite the same as
iter(testobj.__obj) for new-style instances, mind you.)

 

[Gonçalo Rodrigues]

Because it may not work. That is, the use case I have in mind serves
as proxy for an object that may or may not have __iter__. IOW if I
stick __iter__ I have to code a second class, etc... etc... Oh well...

Replying to myself:

Just to make things clearer, having iter(self.__obj) as in

def __iter__(self):
return iter(self.__obj)

Would mean that my proxy class would become an iterable. And that is
not what I want because *other* parts in my code check that an object
is in iterable by looking for __iter__. Suffice it to say that fake
iterables, although not the worst thing in the world, are not
desirable. Maybe this means that I have to approach things in another
way. Humpf.

With my best regards,
G. Rodrigues

 

[Aahz]

So, is it a bug? In other words, should I open a bug report?

I'd say it's not a bug. It's a natural consequence of the way new-style
classes are intended to work. OTOH, opening a bug report may result in
improvement of the documentation -- if you do choose a bug report, I'd
suggest taking that tack.

 

[Alex Martelli]

I would open a bug report; at the best this is a documentation bug,

I disagree that it's a documentation bug.

since
the fact that ___getattr__ is skipped by special methods is never

The language reference says:
""" A class can implement certain operations ... by defining methods
with special names. """

This is VERY explicit -- "__getattr__ is skipped" is false (the
__getattr__ a *metaclass* could define would indeed be used as
equivalent to the class defining methods) -- the _class_'s __getattr__
if any is irrelevant because it's used on class *instances* (as
the language reference also says a few pages later) and thus it's
not the *CLASS* that's defining "methods with special names" here.

If anything, a mention might be warranted that, for compatibility,
a bug is retained in old-style classes, whereby, in addition to the
class's ability to define methods with special names, any old-style
class INSTANCE may also define them (e.g. through __getattr__) in
contrast with what the docs have just said about the general case.

I do suspect that the peculiarities of old-style classes are not
well documented today, more generally than just this tidbit, alas.


Alex

 

[Gonçalo Rodrigues]

As I understand your clarification, you want the proxy to actually be
iterable (via delegation) when given as an arg to iter(), but to not
look iterable to other code (call it test_iterable()). And your
test_iterable() *currently* looks only for the presence/absence of the
very thing needed to make iter() work.

To look an iterable by delegation *if* the underlying object is, yes.

It is definitely not a Python bug that this does not work.

It finally dawned on me why this is so: it's the way new classes work.
In a sloppy language: a __getattr__ hook is like an instance
attribute, but the iter machinery goes after the class not the
instance so it misses __getattr__.

So my test_iterable (in your nomenclature) is wrongly coded because it
gives false positives.

[Ideas snipped]

Thanks for the input. But since something has to change anyway, the
best idea seems to be to stick __iter__ in the proxy class after all
and change the other code that was counting on testing iterableness by
looking __iter__. This will involve some changes, but it's the "least
worse of all evils."

With my best regards,
G. Rodrigues

 

[Michele Simionato]

"__getattr__ is skipped" is false (the
__getattr__ a *metaclass* could define would indeed be used as
equivalent to the class defining methods) -- the _class_'s __getattr__
if any is irrelevant because it's used on class *instances* (as
the language reference also says a few pages later) and thus it's
not the *CLASS* that's defining "methods with special names" here.

Alex, did you look at the thread I mentioned? Here is the problem
with __getattr__ in metaclasses for special methods (quoting from
that thread).

"""
defining __len__ on the class does not work:

class M(type):
def __getattr__(self,name):
if name=='__len__': return lambda self:0

class F: __metaclass__=M
f.__len__() # AttributeError: 'F' object has no attribute '__len__'

As you see, the problem is that len(x) is not calling x.__len__(),
nor x.__class__.__len__(x); I really would like to know how ``len``
(or ``str``, ``repr``, etc.) work.
"""

If you remember, Bjorn Pettersen reported this (or something similar)
months ago. I am saying that this behaviour is not documented, at
least AFAIK.


Michele

 

[Bengt Richter]

class Test(object):
... def __init__(self, obj):
... self.__obj = obj
... def __getattr__(self, name):
... return getattr(self.__obj, name)
...
a = Test([])
a.__iter__
<method-wrapper object at 0x0112CF30>
iter(a)
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
TypeError: iteration over non-sequence


Is this a bug? If not, how to code Test such that iter sees the
__iter__ of the underlying object?

As Mark guessed, iter() goes directly to the attribute rather than using
the __getattr__ machinery of the class. However, you can intercept it
using a metaclass, but that requires a bit of fancy footwork to reach
down into the instance to get self.__obj.

Actually I came across this because of a little metaclass I was
coding. And my head hurts already as it is...

So, is it a bug? In other words, should I open a bug report?

I don't think it's a bug, but there's a lot to explore ;-)
Here is something I've been hacking at to do some exploring. Maybe you will find it interesting:

====< for_grodrigues.py >=============================================
class Test(object):
def __init__(self, *args, **kw): pass
def __new__(cls, obj):
# helper to capture method in closure for method delegator
def mk_meth_deleg(obj, meth, name):
def meth_deleg(arg0, *args, **kw):
if isinstance(arg0, type):
return meth(arg0, *args, **kw)
else:
return meth(obj, *args, **kw)
return meth_deleg

# get the objects's class dict to build a new class
proxy_cdict = obj.__class__.__dict__.copy()
for name, meth in proxy_cdict.items():
if( name.startswith('__') and
callable(meth) and
#not name.startswith('__getattr') and
not name == '__init__'
):
m = mk_meth_deleg(obj, meth, name)
proxy_cdict[name] = m

# customize proxy class as desired here
def __getitem__(self, i):
if i==2: return '[Test-modified %r]'%obj
else: return obj
proxy_cdict['__getitem__'] = __getitem__

#derive from object's class if possible for stuff we haven't caught??
try:
type('',(obj.__class__,), {})
except TypeError:
proxy_base = object
else:
proxy_base = obj.__class__
proxy_class = type(
obj.__class__.__name__+'_proxy',
(proxy_base,),
proxy_cdict)
proxy_obj = proxy_class()
proxy_obj.instattr = 'proxy_obj instance attribute set by Test'
return proxy_obj
======================================================================
Sample interaction:

>>> from for_grodrigues import Test
>>> a = Test([1,2,3])
>>> a [1, 2, 3]
>>> type(a)

>>> a.__iter__

>>> iter(a)

>>> a.__iter__()

>>> ait = iter(a)
>>> ait.next() 1
>>> for i in a: print i,

...
1 2 3

Note that the __iter__ bypassed the doctored __getitem__ of our proxy object, vs:

>>> for i in range(3): print a,

...
1 2 [Test-modified 3]

>>> b = Test(None)
>>> type(b)

>>> b None
>>> b[2]

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "for_grodrigues.py", line 26, in __getitem__
if i==2: return '[Test-modified %r]'%obj
TypeError: unsubscriptable object

Now we try to iterate over the string, but it doesn't provide __iter__ apparently,
since we can see that it wound up using __getitem__:
...
a b [Test-modified 'c'] d

vs, again the list:

>>> a [1, 2, 3]
>>> type(a)

>>> for i in a: print i,

...
1 2 3

>>> c 'abcd'
>>> c[::-1]

'dcba'
That doesn't use __getitem__ either.

By inheriting from the object's class, we pick up base class methods:
... def afoo(self): return 'afoo()'
... ... def bfoo(self): return 'bfoo()'
... <class '__main__.B_proxy'>

There are interesting issues of proxy object attribute space vs the real object vs the
proxy class methods vs the real object's methods, vs inheritance for it, etc.
and you could alter the priorities.

You can mess with it yourself, and maybe eliminate some hacking cruft ;-)

Regards,
Bengt Richter

 

[Bengt Richter]

Alex, did you look at the thread I mentioned? Here is the problem
with __getattr__ in metaclasses for special methods (quoting from
that thread).

"""
defining __len__ on the class does not work:

class M(type):
def __getattr__(self,name):
if name=='__len__': return lambda self:0

class F: __metaclass__=M

f.__len__() # AttributeError: 'F' object has no attribute '__len__'

As you see, the problem is that len(x) is not calling x.__len__(),
nor x.__class__.__len__(x); I really would like to know how ``len``
(or ``str``, ``repr``, etc.) work.
"""

Not very tested, but maybe you have to write it something like this:
... def __new__(cls, name, bases, cdict):
... cdict['__len__'] = lambda self:0
... return type.__new__(cls, name, bases, cdict)
... 0
True

Or am I missing the point? ;-/

If you remember, Bjorn Pettersen reported this (or something similar)
months ago. I am saying that this behaviour is not documented, at
least AFAIK.

Regards,
Bengt Richter

 

[Michele Simionato]

Not very tested, but maybe you have to write it something like this:
... def __new__(cls, name, bases, cdict):
... cdict['__len__'] = lambda self:0
... return type.__new__(cls, name, bases, cdict)
...

0

True

Or am I missing the point? ;-/

Regards,
Bengt Richter

Yes, you are missing the point, indeed

The point is that if you define a fake special method via __getattr__, it
works if it is called as F.__len___(f), but it DOES not work if it is
called as len(f). I suspect Gonzalo's problem is the same with iter.

Built-in like iter, len, etc. look directly for special methods __iter__,
__len__ etc., *without* checking if they are defined by __getattr__. I
am not saying that this is necessarely a bug, I am saying that this is
not documented and that at least three persons have been beaten by this
issue in the last few months. Not me personally, hence the reason why I didn't
submit a bug report, but this time (if Gonzalo is not going to do that), I
will submit the report, unless Alex is going to prove that this is
already documented

Michele

 

[Michele Simionato]

It finally dawned on me why this is so: it's the way new classes work.
In a sloppy language: a __getattr__ hook is like an instance
attribute, but the iter machinery goes after the class not the
instance so it misses __getattr__.

No, iter does not work even if you define __getattr__ at the metaclass
level, i.e. iter(c) is not type(c).__iter__(c) if "__iter__" is defined
via __getattr__ :
def __getattr__(cls,name):
if name=='__iter__': return lambda self:iter([])Traceback (most recent call last):
File "<pyshell#10>", line 1, in -toplevel-
iter(c)
TypeError: iteration over non-sequence<listiterator object at 0x00A3DB30>

having-brought-more-confusion-in-your-head-yours-sincerely,

Michele

 

[Alex Martelli]

Michele Simionato wrote:
...

The point is that if you define a fake special method via __getattr__, it
works if it is called as F.__len___(f), but it DOES not work if it is
called as len(f). I suspect Gonzalo's problem is the same with iter.

You're right. Special methods are looked up in the 'slots' (tp_iter and
the like), for the purpose of Python operations and built-ins (iter() and
the like), and __getattr__, being dynamic, doesn't affect those slots.

E.g.:

def fakeiter(x):
return iter('ciao')

class witer(type):
def __getattr__(cls, name):
if name == '__iter__':
cls.__iter__ = fakeiter
return cls.__iter__
raise AttributeError, name

__metaclass__ = witer

class X: pass

try:
for x in iter(X()):
print x,
print
except TypeError:
print 'TypeError, as expected'
else:
print "No TypeError (?!)"

try:
for x in X().__iter__(): print x,
print
except AttributeError:
print 'AttributeError, as expected'
else:
print "No AttributeError (?!)"

for x in X.__iter__(X()): print x,
print

try:
for x in iter(X()):
print x,
print
except TypeError:
print 'TypeError, alas'
else:
print "No TypeError (good!)"

Once X.__iter__ IS finally accessed, it gets installed in the
class object, thereby affecting the slots and making iter happy.

But until that happens, the class's tp_iter is null, iter()
does not cause a __getattr__ call, and X().__iter__, being an
access on the instance, doesn't go up to the metaclass either.

Built-in like iter, len, etc. look directly for special methods __iter__,
__len__ etc., *without* checking if they are defined by __getattr__. I
am not saying that this is necessarely a bug, I am saying that this is
not documented and that at least three persons have been beaten by this

I agree. The way the Nutshell puts it is "in the new-style object
model, implicit use of special methods always relies on the class-level
binding of the special method, if any" -- and this is correct and
complete, even though probably too terse for anything except perhaps
a Language Reference (which IS allowed to be very terse as long as
it's correct and complete). __getattr__ is not a BINDING of the
special method, though it may be considered a DEFINITION of it, which
is why the current phrase in the Language Reference is not 100% correct
and complete -- only 99.44%, and I agree that the remaining 0.56%
_is_ a delicate defect in the documentation.

It would probably be more helpful to most readers if a section on
new-style classes, one on old-style classes, and/or one on special
methods, expanded on this issue just a wee little bit. Alas, the
Language Reference being meant for language-lawyers only, it's hard
to argue that it should deign to clarify things, as long as they
_are_ expressed in ways that are correct and complete.

issue in the last few months. Not me personally, hence the reason why I
didn't submit a bug report, but this time (if Gonzalo is not going to do
that), I will submit the report, unless Alex is going to prove that this
is already documented

It's not _correctly and completely_ documented, I have to agree -- the
docs use the potentialy-slightly-ambiguous verb 'define' rather than the
more specific one 'bind'.


Alex

 

[Gonçalo Rodrigues]

On 8 Aug 2003 04:23:48 -0700, (e-mail address removed) (Michele Simionato) wrote:

[text snipped]

Not me personally, hence the reason why I didn't
submit a bug report, but this time (if Gonzalo is not going to do that), I
will submit the report, unless Alex is going to prove that this is
already documented

I'm gonna do it this weekend. I'll post a kind of abstract of what I
learned from this thread.

With my best regards,
G. Rodrigues