Is __mul__ sufficient for operator '*'?

[Muhammad Alkarouri]

Hi everyone,

I was having a go at a simple implementation of Maybe in Python when I
stumbled on a case where x.__mul__(y) is defined while x*y is not.

The class defining x is:

class Maybe(object):
def __init__(self, obj):
self.o = obj
def __repr__(self):
return 'Maybe(%s)' % object.__getattribute__(self, "o")
def __getattribute__(self, name):
try:
o = object.__getattribute__(self, "o")
r = getattr(o,name)
if callable(r):
f = lambda *x:Maybe(r(*x))
return f
else:
return Maybe(r)
except:
return Maybe(None)

The code exercising this class is:

Traceback (most recent call last):
File "<pyshell#83>", line 1, in <module>
x*7
TypeError: unsupported operand type(s) for *: 'Maybe' and 'int'

The farthest I can go in this is that I presume that __mul__ (as
called by operator *) is supposed to be a bound method while I am
returning a lambda function. Is this correct? And How can I make the
implementation support such operators?

Cheers,

Muhammad Alkarouri

 

[Gary Herron]

Hi everyone,

I was having a go at a simple implementation of Maybe in Python when I
stumbled on a case where x.__mul__(y) is defined while x*y is not.

The class defining x is:

class Maybe(object):
def __init__(self, obj):
self.o = obj
def __repr__(self):
return 'Maybe(%s)' % object.__getattribute__(self, "o")
def __getattribute__(self, name):
try:
o = object.__getattribute__(self, "o")
r = getattr(o,name)
if callable(r):
f = lambda *x:Maybe(r(*x))
return f
else:
return Maybe(r)
except:
return Maybe(None)

The code exercising this class is:



Traceback (most recent call last):
File "<pyshell#83>", line 1, in <module>
x*7
TypeError: unsupported operand type(s) for *: 'Maybe' and 'int'

The product 7*x will execute __mul__, but to catch x*7, you need to
define __rmul__ (the 'r' stands for reverse or some such).

However, in fact, you have *not* defined __mul__ in your class. Your
__getattribute__ is catching __mul__ as an undefined reference, and
doing something with it -- not sure what though.

As proof, continue testing: x.__mul__ and x.__rmul__ both return
values (lambdas defined within __getattribute__) and neither x*7 or 7*x
work.

If you want both x* and 7*x to be defined, try
def __mul__(self,r):
...
def __rmul__(self,r):
....

Or the operation is commutative, perhaps you can get away with reusing
__mul__ for both.
def __mul__(self,r):
...
__rmul__ = __mull__


Gary Herron

 

[Mick Krippendorf]

Traceback (most recent call last):
File "<pyshell#83>", line 1, in <module>
x*7
TypeError: unsupported operand type(s) for *: 'Maybe' and 'int'

The farthest I can go in this is that I presume that __mul__ (as
called by operator *) is supposed to be a bound method while I am
returning a lambda function. Is this correct? And How can I make the
implementation support such operators?

It does not so much depend on the function being bound, but on the fact
that Python expects Maybe.__mul__ to be present already when it tries to
execute '*'. You could always add this, though:

def __mul__(self, other):
return Maybe.__getattribute__(self, "__mul__")(other)

but of course you'd have to do it for every operator you want to use on
your Maybe objects. Your use of __getattribute__ OTH suggests you're
trying to avoid exactly that. I'd rather go for a monadic implementation
with unit, bind and lift operations, e.g:

----8<--------8<--------8<--------8<--------8<----

class Monad(object):
# unit:
def __init__(self, value):
self.value = value
# bind:
def __rshift__(self, function):
return function(self.__class__)(self.value)
def __str__(self):
return "%s(%s)" % (self.__class__.__name__, self.value)

def lift(f):
def lift_unit(m):
def lift_bind(x):
return m(f(x))
return lift_bind
return lift_unit

class idem(Monad):
def __call__(self, value):
return value

class Nothing(object):
def __rshift__(self, function):
return Nothing
def __str__(self):
return "Nothing"
Nothing = Nothing()

class Maybe(Monad):
def __new__(cls, value=Nothing):
if value is Nothing:
return Nothing
return super(Monad, cls).__new__(cls)

if __name__ == "__main__":

x = Maybe(9)
print x >> lift(lambda v: v * 7)
print x >> lift(lambda v: v * 7) >> idem

y = Maybe(Nothing)
print y >> lift(lambda v: v * 7)
print y >> lift(lambda v: v * 7) >> idem

----8<--------8<--------8<--------8<--------8<----

While I can see how this monadic stuff is usefull in Haskell et al.,
I'm still not sure how to apply it to Python. And for the impenetrable
mathematical language in which Monads are usually presented, well...


HTH,
Mick.

 

[Gabriel Genellina]

En Mon, 19 Oct 2009 21:31:44 -0300, Muhammad Alkarouri

I was having a go at a simple implementation of Maybe in Python when I
stumbled on a case where x.__mul__(y) is defined while x*y is not.

__special__ methods are searched in the type, not in the instance
directly. x*y looks for type(x).__mul__ (among other things)

 

[Mick Krippendorf]

__special__ methods are searched in the type, not in the instance
directly. x*y looks for type(x).__mul__ (among other things)

So I thought too, but:

class meta(type):
def __mul__(*args):
return 123

class boo(object):
__metaclass__ = meta

print boo.__mul__

b = boo()
print b * 7

also explodes. Or am I misinterpreting the word "type" here?


Mick.

 

[Gabriel Genellina]

Gabriel Genellina schrieb:

So I thought too, but:

class meta(type):
def __mul__(*args):
return 123

class boo(object):
__metaclass__ = meta

print boo.__mul__

b = boo()
print b * 7

also explodes. Or am I misinterpreting the word "type" here?

This is by design; see
http://docs.python.org/reference/datamodel.html#special-method-lookup-for-new-style-classes

Methods defined on the meta-type aren't considered methods of the type;
otherwise, every object would have the methods defined in `type` itself,
because this is the metatype of every other object.

When I said "x*y looks for type(x).__mul__" the search for '__mul__' is
done in type(x) itself, and all its base types along the MRO - NOT on the
metatype, and not using getattr. There is no way to express this exact
search in Python code (that I know of), but it's more or less like
searching for '__mul__' in dir(type(x)).

In particular, __mul__ is stored in two slots (a slot is a field in a
structure holding function pointers: nb_multiply in the tp_as_number
structure, *and* sq_repeat in tp_as_sequence); defining or assigning to
__mul__ "magically" updates those internal pointers. When executing x*y,
__mul__ is retrieved directly from those pointers -- it is *not* searched
by name.

In short, you have to define the __mul__ method on the type itself or any
of its bases.

 

[Mick Krippendorf]

http://docs.python.org/reference/datamodel.html#special-method-lookup-for-new-style-classes

Ok. That explains a lot. And your explanation tells the rest. Thank you.

In short, you have to define the __mul__ method on the type itself or
any of its bases.

I found this, where Raymond Hettinger shows how this could be changed:
http://mail.python.org/pipermail/python-bugs-list/2005-December/031439.html

Mick.

 

[Mick Krippendorf]

I was having a go at a simple implementation of Maybe in Python when I
stumbled on a case where x.__mul__(y) is defined while x*y is not.

class Maybe(object):
def __init__(self, obj):
self.o = obj
def __repr__(self):
return 'Maybe(%s)' % object.__getattribute__(self, "o")
def __getattribute__(self, name):
try:
o = object.__getattribute__(self, "o")
r = getattr(o,name)
if callable(r):
f = lambda *x:Maybe(r(*x))
return f
else:
return Maybe(r)
except:
return Maybe(None)


Traceback (most recent call last):
File "<pyshell#83>", line 1, in <module>
x*7
TypeError: unsupported operand type(s) for *: 'Maybe' and 'int'

Here's how I'd do it. It will not win a beauty contest any time soon,
but at least it's a workaround:

----8<--------8<--------8<--------8<--------8<--------8<--------8<----

def meta(lift, lifted, not_lifted=[]):
not_lifted = list(not_lifted) + object.__dict__.keys()
class MetaMaybe(type):
def __new__(meta, mcls, bases, dct):
dct.update(
(name, lift(name))
for name in set(lifted) - set(not_lifted)
)
return type(mcls, bases, dct)
return MetaMaybe

class Nothing(object):
__metaclass__ = meta(lambda name: lambda self, *a, **k: self, (
"__add__", "__sub__", "__mul__", "__div__", "__truediv__",
"__floordiv__", "__divmod__", "__radd__", "__rsub__",
"__rmul__", "__rdiv__", "__rtruediv__", "__rfloordiv__",
"__rdivmod__", "__rshift__", "__lshift__", "__call__",
# and so on, for every special method that Nothing knows
))
def __new__(cls, value=None):
try: # singleton
return cls.value
except AttributeError:
cls.value = super(Nothing, cls).__new__(cls)
return cls.value
def __str__(self):
return "Nothing"
__repr__ = __str__

Nothing = Nothing()

def just(vcls):
def lifter(name):
attr = getattr(vcls, name)
def lifted(self, *ms):
try:
return self.lift(attr)(self, *ms)
except:
return Nothing
return lifted
class Just(object):
__metaclass__ = meta(lifter, vcls.__dict__.keys())
def __new__(cls, value):
if value in (Nothing, NotImplemented):
return Nothing
return super(Just, cls).__new__(cls)
def __init__(self, value):
self.value = value
def __str__(self):
return "Just(%s)" % self.value
@classmethod
def lift(c, f):
return lambda *ms:c(f(*(m.value for m in ms)))
return Just

from collections import defaultdict

class TypeDict(defaultdict):
def __missing__(self, key):
if self.default_factory is None:
raise KeyError(key)
return self.default_factory(key)

class Maybe(object):
typemap = TypeDict(just)
def __new__(cls, value):
return Maybe.typemap[value.__class__](value)

def foo(x, y):
return x * 2 + y * 3

if __name__ == "__main__":

print Maybe(Nothing)
print Maybe(1) / Maybe(0)
print Maybe(10.) * Maybe(5) / Maybe(2) ** Maybe(3)
print Maybe(foo)(Maybe(6), Maybe(10))
print Maybe("hello").upper()
print Maybe("hello").startswith(Maybe("h"))
print getattr(Maybe("hello"), "startswith")(Maybe("h"))
print Maybe(foo)(Maybe("hello! "), Maybe("what? "))
print Maybe(foo)(Maybe("hello! "), Nothing)

----8<--------8<--------8<--------8<--------8<--------8<--------8<----

I haven't tested it very thoroughly, so it's quite possible there are
lots of bugs in it, but it is only intended as a demo.

As Gabriel Genellina pointed out, the search for special methods is done
in the type, so we have to put our own versions there, during type
creation in the metaclass' __new__ method. The above code does this for
all methods of the wrapped object's type, not just special ones, and
"lifts" them to expect Maybe objects instead of "normal" objects. They
also wrap their return values into Maybe objects.

Maybe is an algebraic data type. The call Maybe(some_value) returns
either Nothing, if some_value happens to be Nothing, or else an object
of type Just that wraps some_value. More precisely, there is not one
type Just, but as many as types of Just-wrapped objects (Just<int>,
Just<string>, ...). Just is therefore a kind of parameterized type. It's
similar to inheriting from a C++ template parameter type:

template <class T>
class MyType : public T {...}

but not quite, since in C++ the inherited member functions' signatures
are unchanged, whereas in the above code "inherited" methods are changed
to expect and return Maybe objects.

Some things don't work, though, e.g. slicing. But this could be
implemented via specialized lifting functions for the __XXXitem__
methods. One thing that does work though, is that ordinary functions can
be wrapped as Maybe objects which then "do the same thing" on other
Maybe objects that the normal functions do on normal objects, like in
the foo-example. So it's quite close to a monadic version, in that it
"lifts" objects and functions from one type space into another one, the
Maybe space. But compared to a real Monads it's much more pythonic, IMO.


HTH,
Mick.