Annoying behaviour of the != operator

[Jordan Rastrick]

Can anybody please give me a decent justification for this:

class A(object):
def __init__(self, a):
self.a = a

def __eq__(self, other):
return self.a == other.a

s = A(3)
t = A(3)
True

I just spent a long, long time tracking down a bug in a program that
results from this behaviour.

Surely the != operator should, if no __ne__ method is present for
either object, check to see if an __eq__ method is defined, and if so,
return its negation?

Actually, that brings me to a wider question - why does __ne__ exist at
all? Surely its completely inconsistent and unnessecary to have
seperate equals and not equals methods on an object? a != b should just
be a short way of writing not (a == b). The fact the two can give a
different answer seems to me to be utterly unintuitive and a massive
pitfall for beginners (such as myself).

 

[Dave Benjamin]

Surely the != operator should, if no __ne__ method is present for
either object, check to see if an __eq__ method is defined, and if so,
return its negation?

Actually, that brings me to a wider question - why does __ne__ exist at
all? Surely its completely inconsistent and unnessecary to have
seperate equals and not equals methods on an object? a != b should just
be a short way of writing not (a == b). The fact the two can give a
different answer seems to me to be utterly unintuitive and a massive
pitfall for beginners (such as myself).

I agree that it's confusing. I didn't even understand this behavior
myself until I went and looked it up:

http://docs.python.org/ref/customization.html

The rationale for this behavior is in PEP 207 -- Rich Comparisons:

http://python.fyxm.net/peps/pep-0207.html

Personally, I implement the __cmp__ method when I want my objects to be
comparable, so I've never run into this problem.

Dave

 

[Fredrik Lundh]

I just spent a long, long time tracking down a bug in a program that
results from this behaviour.

Surely the != operator should, if no __ne__ method is present for
either object, check to see if an __eq__ method is defined, and if so,
return its negation?

Actually, that brings me to a wider question - why does __ne__ exist at
all? Surely its completely inconsistent and unnessecary to have
seperate equals and not equals methods on an object? a != b should just
be a short way of writing not (a == b). The fact the two can give a
different answer seems to me to be utterly unintuitive and a massive
pitfall for beginners (such as myself).

surely reading the documentation would be a great way to avoid
pitfalls?

http://docs.python.org/ref/customization.html

__lt__, __le__ (etc)

New in version 2.1. These are the so-called "rich comparison" methods,
and are called for comparison operators in preference to __cmp__() below.

/.../

There are no implied relationships among the comparison operators. The
truth of x==y does not imply that x!=y is false. Accordingly, when defining
__eq__, one should also define __ne__ so that the operators will behave
as expected.

/.../

__cmp__

Called by comparison operations if rich comparison (see above) is not
defined. Should return a negative integer if self < other, zero if self
== other, a positive integer if self > other. /.../

for a number of situations where __ne__ cannot be derived from __eq__,
see:

http://www.python.org/peps/pep-0207.html

</F>0

 

[Mahesh]

No, why should Python assume that if you use != without supplying a
__ne__ that this is what you want? Without direction it will compare
the two objects which is the default behavior.

So, s != t is True because the ids of the two objects are different.
The same applies to, for example s > t and s < t. Do you want Python to
be smart and deduce that you want to compare one variable within the
object if you don't create __gt__ and __lt__? I do not want Python to
do that.

Regards,
M

 

[Dan Bishop]

No, why should Python assume that if you use != without supplying a
__ne__ that this is what you want?

Because every single time I've used __ne__, that *is* what I want.

Without direction it will compare
the two objects which is the default behavior.

It's also the default behavior that x == y and x != y are mutally
exclusive.

 

[Jordan Rastrick]

Well, I don't really want the objects to be comparable. In fact, to
quote that PEP you linked:

An additional motivation is that frequently, types don't have a
natural ordering, but still need to be compared for equality.
Currently such a type *must* implement comparison and thus define
an arbitrary ordering, just so that equality can be tested.

I don't want to order the objects. I just want to be able to say if one
is equal to the other.

Here's the justification given:

The == and != operators are not assumed to be each other's
complement (e.g. IEEE 754 floating point numbers do not satisfy
this). It is up to the type to implement this if desired.
Similar for < and >=, or > and <=; there are lots of examples
where these assumptions aren't true (e.g. tabnanny).

Well, never, ever use equality or inequality operations with floating
point numbers anyway, in any language, as they are notoriously
unreliable due to the inherent inaccuracy of floating point. Thats
another pitfall, I'll grant, but its a pretty well known one to anyone
with programming experience. So I don't think thats a major use case.

Are there any other reasonable examples people can give where it makes
sense for != and == not to be each other's complement?

Besides, the argument given doesn't justfiy the approach taken. Even if
you want to allow objects, in rare cases, to override __ne__ in a way
thats inconsistent with __eq__, fine, do so. But thats no reason for
Python not to fall back on __eq__ in the cases where __ne__ is not
defined, rather than reverting to object identity comparison (at least
I assume thats what its doing)

To draw a parallel:

In Java, the compareTo method in a class is allowed to be inconsistent
with the equals method, but those who write such code are advised to
heavily advertise this fact in their documentation and source, so as
not to catch unwary users off guard.

In Python, the *default* behaviour is for the equals and not equals
operations to disagree if the __eq__ method happens to be overriden
(which it definitely should be in a great number of cases).

Surely this isn't the right way to do things.

How many classes out there that have the following redundant,
cluttering piece of code?

def __ne__(self, other):
return not self.__eq__(other)

Worse still, how many classes don't have this code, but should, and are
therefore harbouring highly confusing potential bugs?

Unless someone can explain some sort of problem that arises from having
!= take advantage of a __eq__ method where present, I'd suggest that it
should do so in Python 2.5.

I'd be surprised if such a change broke so much as a single line of
existing Python code.

 

[Peter Hansen]

for a number of situations where __ne__ cannot be derived from __eq__,
see:

http://www.python.org/peps/pep-0207.html

That "number" being "one"?

I can see only one comment that seems to describe that situation, where
it refers to "IEEE 754 floating point numbers do not satisfy [== being
the complement of !=]".

(Though that may be justification enough for the feature...)

-Peter

 

[Rocco Moretti]

Unless someone can explain some sort of problem that arises from having
!= take advantage of a __eq__ method where present, I'd suggest that it
should do so in Python 2.5.

If you're serious about this proposal, please formalize it in a PEP.

Things to specify:

How extensive are these changes? Is it just !=, or if __eq__ is not
defined, will it be equal to "not __ne__()"? __lt__/__gt__?
__le__/__ge__? Do you simulate __le__ with '__lt__ or __eq__'? How about
__lt__ with '__le__ and __ne__'?

I'd be surprised if such a change broke so much as a single line of
existing Python code.

Well, since it currently raises an error when __ne__ is not defined, I'd
say your right on that account. The only corner case is when people
would rely on 'a != b' to be implicitly translated to 'b != a'. If 'a !=
b' gets translated to 'not a == b', this may change semantics if the two
are not equivalent.

 

[Jordan Rastrick]

Just because a behaviour is documented, doesn't mean its not counter
intutitive, potentially confusing, and unnessecary.

I have spent a fair amount of time reading the Python docs. I have not
yet memorised them. I may have read this particular section of the
reference manual, or I may have not, I can't remember. This is the
first time I've had cause to override the __eq__ operator, and so the
first time I've encountered the behaviour in a concrete setting, which
I feel is a far more effective way to learn than to read all the
documentation cover to cover.

Where are the 'number of situations' where __ne__ cannot be derived
from __eq__? Is it just the floating point one? I must admit, I've
missed any others.

And no, I don't think I really want Python to assume anything much -
explicit is better than implicit after all. If I don't override __eq__,
or __gt__, or __cmp__, i certainly don't expect Python to infer the
behaviour I want.

But I explicitly provided a method to test equality. And look at the
plain english meaning of the term "Not equals" I think its pretty
reasonable

As far as __gt__, __lt__, __ge__, and the rest go, the answer is
simple. Anyone defining things the 'obvious' way would override
__cmp__, and sidestep the whole issues. Those who want the less obvious
behaviour of can provide it explicitly.

But as PEP207 says, you shouldn't have to override cmp if you don't
want to provide an arbitrary ordering. And its the people doing the
non-obvious thing, having __ne__ and __eq__ inconsistent, who should
have to code it and state it explicitly.

If want an orderable object, I'll just define __cmp__. If I want an
equal-comparable object, I should just be able to define __eq__. If I
want anything fancy, I can go ahead and explicitly write methods for
__gt__, __ne__, etc.

 

[John Roth]

Well, I don't really want the objects to be comparable. In fact, to
quote that PEP you linked:

An additional motivation is that frequently, types don't have a
natural ordering, but still need to be compared for equality.
Currently such a type *must* implement comparison and thus define
an arbitrary ordering, just so that equality can be tested.

I don't want to order the objects. I just want to be able to say if one
is equal to the other.

Here's the justification given:

The == and != operators are not assumed to be each other's
complement (e.g. IEEE 754 floating point numbers do not satisfy
this). It is up to the type to implement this if desired.
Similar for < and >=, or > and <=; there are lots of examples
where these assumptions aren't true (e.g. tabnanny).

Well, never, ever use equality or inequality operations with floating
point numbers anyway, in any language, as they are notoriously
unreliable due to the inherent inaccuracy of floating point. Thats
another pitfall, I'll grant, but its a pretty well known one to anyone
with programming experience. So I don't think thats a major use case.

Two floating point numbers are equal if they are within epsilon of each
other. Epsilon, of course, depends on the application. I do floating
point compares quite nicely in PyFit, with intuitive results. It simply
requires you to know what you're doing rather than blindly following
the herd over the cliff.

That is, however, not the reason for the statement. The reason for the
statement is that the feature was motivated by the numerics package,
which has a number of special requirements, floating point exceptional
values only being one. Read the PEP.

John Roth

 

[Jordan Rastrick]

I'd suggest the only nessecary change is, if objects a,b both define
__eq__ and not __ne__, then a != b should return not (a == b)

If a class defines __ne__ but not __eq__, well that seems pretty
perverse to me. I don't especially care one way or another how thats
resolved to be honest.

The ordering comparisons (__lt__, __ge__ etc) are fine as they are I'd
say, since they only come up in the rare cases where __cmp__ isn't
sufficient.

I'll wait for a bit more discussion on here before starting a PEP - as
I've said I'm only a beginner, and a more experience Pyonista may yet
give a perfectly good argument in favour of the current behaviour.

 

[Robert Kern]

Are there any other reasonable examples people can give where it makes
sense for != and == not to be each other's complement?

__eq__ and __ne__ implement *rich* comparisons. They don't have to
return only True or False.

In [1]:import Numeric

In [2]:a = Numeric.array([1, 2, 3, 4, 5])

In [3]:b = Numeric.array([1, 0, 4, 0, 5])

In [4]:a == b
Out[4]:array([1, 0, 0, 0, 1],'b')

In [5]:a != b
Out[5]:array([0, 1, 1, 1, 0],'b')

In [6]a != b) == (not (a == b))
Out[6]:array([1, 0, 0, 0, 1],'b')

In [7]:not (a == b)
Out[7]:False

In [8]:not (a != b)
Out[8]:False

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Matt Warden]

Jordan,

But I explicitly provided a method to test equality. And look at the
plain english meaning of the term "Not equals" I think its pretty
reasonable

Indeed. Furthermore, it seems quite silly that these would be different:

a != b
not (a == b)

To be fair, though, other languages have peculiarities with equation.
Consider this Java code:

String s1 = "a";
String s2 = "a";
String s3 = new String("a");
String s4 = new String("a");

s1 == s2; // true
s1.equals(s2); // true

s1 == s3; // false
s1.equals(s3); // true

s3 == s4; // false
s3.equals(s4); // true

Doesn't make it any less silly, though.

--
Matt Warden
Miami University
Oxford, OH, USA
http://mattwarden.com


This email proudly and graciously contributes to entropy.

 

[George Sakkis]

I'd suggest the only nessecary change is, if objects a,b both define
__eq__ and not __ne__, then a != b should return not (a == b)

If a class defines __ne__ but not __eq__, well that seems pretty
perverse to me. I don't especially care one way or another how thats
resolved to be honest.

The ordering comparisons (__lt__, __ge__ etc) are fine as they are I'd
say, since they only come up in the rare cases where __cmp__ isn't
sufficient.

I'll wait for a bit more discussion on here before starting a PEP - as
I've said I'm only a beginner, and a more experience Pyonista may yet
give a perfectly good argument in favour of the current behaviour.

I've been bitten by this particular wart more than once, so I wrote a
metaclass for automating the addition of the missing rich comparisons
with the expected semantics: == and != are complementary, and given
either of them and one of <, >, <=, >=, the other three are defined.
You can check it out at
http://rafb.net/paste/results/ymLfVo81.html. Corrections and comments
are welcome.

Regards,
George

 

[Jordan Rastrick]

Well, I'll admit I haven't ever used the Numeric module, but since
PEP207 was submitted and accepted, with Numeric as apparently one of
its main motivations, I'm going to assume that the pros and cons for
having == and ilk return things other than True or False have already
been discussed at length and that argument settled. (I suppose theres a
reason why Numeric arrays weren't just given the same behaviour as
builtin lists, and then simple non-special named methods to do the
'rich' comparisons.)

But again, it seems like a pretty rare and marginal use case, compared
to simply wanting to see if some object a is equal to (in a non object
identity sense) object b.

The current situation seems to be essentially use __cmp__ for normal
cases, and use the rich operations, __eq__, __gt__, __ne__, and rest,
only in the rare cases. Also, if you define one of them, make sure you
define all of them.

Theres no room for the case of objects where the == and != operators
should return a simple True or False, and are always each others
complement, but <, >= and the rest give an error. I haven't written
enough Python to know for sure, but based on my experience in other
languages I'd guess this case is vastly more common than all others put
together.

I'd be prepared to bet that anyone defining just __eq__ on a class, but
none of __cmp__, __ne__, __gt__ etc, wants a != b to return the
negation of a.__eq__(b). It can't be any worse than the current case of
having == work as the method __eq__ method describes but != work by
object identity.

So far, I stand by my suggested change.

 

[Rocco Moretti]

Jordan,




Indeed. Furthermore, it seems quite silly that these would be different:

a != b
not (a == b)

It's only "silly" if one sticks to strict Boolean semantics or
implicitly assumes the law of the excluded middle
(http://en.wikipedia.org/wiki/Excluded_middle), the principle of
bivalence (http://en.wikipedia.org/wiki/Principle_of_bivalence), or the
law of noncontradiction
(http://en.wikipedia.org/wiki/Law_of_non-contradiction). Despite "law"
status, it is possible (and useful) to imagine situations where they
don't hold. (A'la non-euclidlean geometry).

The main problem is that Python is trying to stick at least three
different concepts onto the same set of operators: equivalence (are
these two objects the same?), ordering (in a sorted list, which comes
first?), and mathematical "size". This gives the wacky world where
"[(1,2), (3,4)].sort()" works, whereas "[1+2j, 3+4j].sort()" doesn't.

Luckily another related concept, identity, has already been separated
out (the 'is' operator). It would be nice (but I'm not hoding my breath)
if the complete issue gets resolved with Python 3000.

 

[Kay Schluehr]

Just because a behaviour is documented, doesn't mean its not counter
intutitive, potentially confusing, and unnessecary.

I have spent a fair amount of time reading the Python docs. I have not
yet memorised them. I may have read this particular section of the
reference manual, or I may have not, I can't remember. This is the
first time I've had cause to override the __eq__ operator, and so the
first time I've encountered the behaviour in a concrete setting, which
I feel is a far more effective way to learn than to read all the
documentation cover to cover.

Where are the 'number of situations' where __ne__ cannot be derived
from __eq__? Is it just the floating point one? I must admit, I've
missed any others.

For certain classes you can define behaviour like this:
-2

My Expr class implements __eq__ and __ne__ in the following way:

def __eq__(self,other):
self._wrapPredicate("==",other)
return self._iseq(other)

def __ne__(self,other):
self._wrapPredicate("!=",other)
return not self._iseq(other)

It would be hard to use operator overloading to create expressions if
there are a lot of implicite assumptions. On the other hand I agree
with you about the default behaviour of __ne__ and that it should be
related locally to the class that overloads __eq__ and not to some
global interpreter defined behaviour.

Kay

 

[Christopher Subich]

> I can see only one comment that seems to describe that situation,

where it refers to "IEEE 754 floating point numbers do not satisfy [==
being the complement of !=]".

>
> (Though that may be justification enough for the feature...)

To my naive eye, that possibility seems like justification for the
language to not -enforce- that (not (a == b)) == (a != b), but for the
vast majority of cases this is true. Perhaps the language should offer
the sensible default of (!=) == (not ==) if one of them but not the
other is overriden, but still allow overriding of both.

This would technically break backwards compatibilty, because it changes
default behavior, but I can't think of any good reason (from a python
newbie perspective) for the current counterintuitive behavior to be the
default. Possibly punt this to Python 3.0?

 

[Jordan Rastrick]

I'm a Maths and Philosophy undergraduate first and foremost, with
Computer Science as a tacked on third; I've studied a fair bit of logic
both informally and formally, and am familiar with things such as the
non-nessecity of the law of the excluded middle in an arbitrary
propositional calculus farmework.

I can now also see the usefulness of overriding != and == to do things
other than simple equality comparison. Kay Schueler's Expr class seems
like a particularily elegant and beautful example! (and seems to me a
much better justification for rich comparisons than the rather mundane
typying-reduction case of Numeric arrays)

So I'm not arguing Python should demand != and not(a == b) return the
same thing all the time (although I did question this in my original
post). My argument is simply one of pragmatism - cases where this is
not the case are the highly unusual ones, and so they should be the
ones forced to write seperate __eq__ and __ne__ methods, In *every*
example that's been rased so far - Floats, Numeric.array, Expr,
(hypothetically) some unusual Symbolic Logic program without the law of
excluded middle - these methods are both by nessecity independently
defined, and my suggestion would not change the status quo at all.

Mahesh raised the argument some posts back that Python should not 'just
guess' what you want. But the problem is, it *already does* - it
guesses you want object identity comparison if you haven't written
__ne__. But if __ne__ is not provided, than the negation of

a==b

is *surely* a better guess for a != b than the negation of

a is b

As always, explicit is better than implicit. But if we're going to be
implicit, lets be implicit in the way that makes the most sense. I
can't stand Perl's autoconversion from strings to integers. But how
much worse would it be if strings were auto-converted to, for example,
the sum of the ordinal value of their ascii characters?

OK, thats about as compelling as I can make the argument. If Perl
bashing won't sway Python fans over, I don't know what will

P.S. Excuse the excessive presence typos throughout all my posts, its
been a long night.

 

[Robert Kern]

Mahesh raised the argument some posts back that Python should not 'just
guess' what you want. But the problem is, it *already does* - it
guesses you want object identity comparison if you haven't written
__ne__. But if __ne__ is not provided, than the negation of

a==b

is *surely* a better guess for a != b than the negation of

a is b

The problem arises that, in the presence of rich comparisons, (a == b)
is not always a boolean value, while (a is b) is always a boolean value.
I *would* prefer that (a != b) raise an error when __ne__ isn't
provided, but such is life until 3.0.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter