Virtual functions are virtually invisible!

[rantingrick]

Hello Folks!

In my quest to uncover the many asininities contained within the
tkSimpleDialog.py module i found myself contemplating some the very
fundamental aspects of the Python language proper. And this is aspect
be... "Pythonic notation".

But before i introduce you to my latest discovery of collective
ineptitude i want to discuss some of the great pythonic notations we
already enjoy.

In the beginning, Mr. Van Rossum (in a stroke of complete brilliance i
might add!) stumbled upon the idea of syntactical notation and he
seized upon it! We all know how python uses indention to denote scopes
(and i just love this aspect about the language!) but my next favorite
notation is the leading and trailing double underscores that wrap
special method names[1] Because they just JUMP out at you whilst
reading code (and they should!). Even if they only say...

"""Hey, i am special and you should only care about me if you are an
experienced pythonista, otherwise nothing to see here, move along
lad!"""

Actually i did not realize just HOW important this notation was to my
code comprehension until i started tinkering with another language
called "Ruby" (who BTW does not use any notation for this matter).
Ruby doesn't use any special notation so all the member methods just
blend into the crowd.

Nothing jumps out and says "HEY, I'M SPECIAL, LOOK AT ME"... instead
you start to get sea sick from the homogeneous structure of it all.
You cannot quickly categorize the method hierarchy without extensive
reading and wasted time. This is a shame!

But this gushing over underscores is not the subject of my thread
today, what concerns me is the fact that virtual methods in derived
classes just blend in to the crowd. For me this presents major
problem, because i don't know what "behind the scenes" actions are
going to take place without a very personal relationship with the
underlying code base. More wasted time!

So what i am proposing is some way to differentiate methods that have
been clobbered by the derived class. And it's not the fact that they
have been clobbered that is of any concern to me, NO, what concerns me
is that i don't know what "invisible" side effects are taking place
"behind-the-scenes", much less which methods are the root of such
invisible side effects.

You know as Pyhthonista's we always frown on syntactic decorators
HOWEVER like all good things these visual cues are best taken when in
moderation. I believe in these cases syntactic decorators are not only
needed but they are warranted!

Of course the simplest solution is a doc string explaining the case to
a reader. But as we all know even pyhtonista's refuse to follow the
style guide so this will be a non starter. I think we really need some
sort of visual cue in the form of forced syntactical notation (just
like the special method underscores).

[1] http://docs.python.org/reference/datamodel.html#special-method-names

 

[Gregory Ewing]

what concerns me is the fact that virtual methods in derived
classes just blend in to the crowd.

I think we really need some
sort of visual cue in the form of forced syntactical notation (just
like the special method underscores).

If you're suggesting that it should be impossible to override
a method unless it is specially marked somehow in the base
class, I think that would be a bad idea.

One of the principles behind the design of Eiffel is that
classes should *always* be open to modification in any way
by a subclass. The reason is that the author of a class
library can't anticipate all the ways people might want to
use it. Consequently Eiffel has no equivalent of C++'s
"private" declaration -- everything is at most "protected".
It also has no equivalent of Java's "final".

I like the fact that Python doesn't have these either.

 

[rantingrick]

If you're suggesting that it should be impossible to override
a method unless it is specially marked somehow in the base
class, I think that would be a bad idea.

Sorry i did explain properly... No NOT marked in the BASE class but
marked in the DERIVED class! My concerns are from a readability
standpoint. Here's a naive example...

class Top(object):
def __init__(self):
pass
def m1(self):
"""overide"""
return True
def m2(self):
print 'm2'


def Derived(Top):
def __init__(self):
Top.__init__(self)
def <overide>m1(self):
return False

My argument is this...

"""If class "Derived" exists in another module the reader has no
idea which methods where clobbered and which were not WITHOUT being
intimate with the entire code base."""

I suggest we solve this dilemma by forcing a syntax "tag" when
declaring clobbering virtual functions. And if someone forgets to
include the tag python would throw an exception. This has the effect
of making the code MUCH easier to follow by the human readers. And it
put NO constraints on the API. Also it has the added effect of warning
unsuspecting programmers of name clashes that would otherwise not
produce an error.

One of the principles behind the design of Eiffel is that
classes should *always* be open to modification in any way
by a subclass. The reason is that the author of a class
library can't anticipate all the ways people might want to
use it. Consequently Eiffel has no equivalent of C++'s
"private" declaration -- everything is at most "protected".
It also has no equivalent of Java's "final".

Exactly! We should never put limits on what methods CAN be virtual.
However, we CAN enforce a syntax that removes ambiguity from the
equation.

 

[Chris Angelico]

 I suggest we solve this dilemma by forcing a syntax "tag" when
declaring clobbering virtual functions.

Python has other dilemmas, too. I suggest we adopt the same solution.
For instance, every statement should begin with a marker, so that we
know it isn't a comment; every variable name should be adorned, so
that we know it's not a keyword like 'if'; and every decimal integer
should begin with "0d" so that we know it isn't hex. Plus, we should
get rid of binary operators with their messy precedence tables;
everything should become function calls.

<STMT> $assign($x,$add(0d5,0d7)) </STMT> # assigns 12 (decimal) to x

This would make Python far less ambiguous. We should proceed with this
right away.

ChrisA

 

[Michael Hrivnak]

It sounds to me like you need a better IDE, better documentation,
and/or better code to work on and use. I don't understand why it's
difficult to look at a derived class as see what methods are
overridden. If you are working on the code, it is quite obvious what
methods exist in the base class. If you're not willing to get an
intimate understanding of how the base class works, you probably
shouldn't be working on the subclass. If the base class is difficult
to understand, it's probably poorly written and/or poorly documented.
Neither of these problems should be solved by adding complexity to the
language. Referencing the Zen of Python: "If the implementation is
hard to explain, it's a bad idea."

If you are just using a library but not developing it, why does it
matter what methods are overridden? As long as class "Derived"
behaves the way it is documented, who cares how it got that way or
what is going on behind the scenes? If you need to read the code to
figure out how it works, then it's just poorly documented.

Django is a great example, because it is very well documented. Most
users have little idea of what base classes are involved and what
features are overridden, because it doesn't matter when you are just
using the library. When you need to write your own subclass of a
django class, then it might matter, and you should see my first
paragraph.

And in terms of "non-starters", any "Pythonista" who isn't willing to
adhere to the style guide and document their code wouldn't work on my
team for very long, if at all. There is just no excuse for that.

Michael

 

[rantingrick]

It sounds to me like you need a better IDE, better documentation,
and/or better code to work on and use.

Yes the last two points are relevant here. However whilst IDE choice
belongs to the user, documentation and code are in the hands of the
developer; who's own selfish needs often outweigh that of the user AND
community as a whole.

 I don't understand why it's
difficult to look at a derived class as see what methods are
overridden.

Well in my simple example it is VERY easy, WHY? Here are a few reasons
why the clobbered methods are easy to spot (in both the base and
derived classes)...

* Both exists within the same view frame. No need to flip back and
forth between two windows.
* Both have only one method. The complexity increases exponentially
by the number of methods AND the length of their respective code
blocks!
* "Derived.m1" has a syntactical mark. You can clearly see which
method has been clobbered WITHOUT even bothering to look at the base
class.

The only time you SHOULD have to look at the base class is to see what
mandates the virtual method may impose. Does it require a Boolean
return? An Integer? A Float? A List? Does it modify an object? Etc,
etc?

 If you are working on the code, it is quite obvious what
methods exist in the base class.

Let me correct your statement... IF you have an intimate understanding
of the base. Heck what if the base is derived also? These things are
NOT strictly single level you know.

 If you're not willing to get an
intimate understanding of how the base class works, you probably
shouldn't be working on the subclass.  

Not true, many times you don't need an intimate understanding of the
base to wield a derived class. That's when the syntactical markers
come in handy.

If the base class is difficult
to understand, it's probably poorly written and/or poorly documented.
Neither of these problems should be solved by adding complexity to the
language.

How is adding syntactical markers to an otherwise obfuscation of
virtual method clobbering going to confuse anyone? I would say the
opposite is true. Python has used the forced convention "double-
leading-and-trailing-underscores" to mark special methods since it's
beginning. One reason for this convention is prevent name clashes
HOWEVER the most important reason (i would argue) is for readability
of source code. When Guido implemented this convention it was one of
his greatest gifts to Python and the world (only to be outdone by
forced indention!).

 Referencing the Zen of Python: "If the implementation is
hard to explain, it's a bad idea."

What if the code is "difficult" to read? Does "readability count" ring
a bell?

If you are just using a library but not developing it, why does it
matter what methods are overridden?  As long as class "Derived"
behaves the way it is documented, who cares how it got that way or
what is going on behind the scenes?  If you need to read the code to
figure out how it works, then it's just poorly documented.

Yes. It is obviously poorly documented. And all the writer would have
to do is put a little doc string there saying "clobbered virtual
method here". HOWEVER, do you know how many folks bother to ACTUALLY
do that? Huh? Do ya? None! Exactly.

Django is a great example, because it is very well documented.  Most
users have little idea of what base classes are involved and what
features are overridden, because it doesn't matter when you are just
using the library.  When you need to write your own subclass of a
django class, then it might matter, and you should see my first
paragraph.

When a method has been clobbered any reader of such code NEEDS to know
about it. WHY, well because usually clobbered methods have "magic"
going on being the scenes. Sometimes many layers of "magic".

And in terms of "non-starters", any "Pythonista" who isn't willing to
adhere to the style guide and document their code wouldn't work on my
team for very long, if at all.  There is just no excuse for that.

I agree. However as we are all aware many "great Pythonistas" refuse
to follow the style guide (*cough* freg! . The only way to correct
this problem is via a forced syntactical marker placed by the derived
class's writer. Just like with "__IDENTIFIER__"

READABILITY COUNTS!

 

[Michael Hrivnak]

I can't believe you're saying that you will create a sub-class without
taking the time to understand the base class. Seriously? That right
there is why you are seeing method overrides that aren't documented.
How can you document something you don't understand? Furthermore, how
can you have any confidence in your subclass if you don't understand
what its base class is doing? Do you write unit tests? How do you
know what to test if you don't understand the code you are
subclassing?

Suggesting that no developers document their methods? Incredible.

Clobbered methods have "magic"? Multi-layered "magic"? Perhaps when
you take the time to 1) document the base class and 2) understand that
base class before subclassing it, that "magic" will start to look like
reasonable and logical processes.

Professional developers document their code. They take the time to
understand the code they are working on. If you don't want to do
those things, that's your business, and I'm sure you can find other
people who also don't do those things. But I really don't think
you'll have much success changing the language to accommodate your
refusal to follow the most basic best practices.

Best of luck,
Michael

 

[rantingrick]

I can't believe you're saying that you will create a sub-class without
taking the time to understand the base class.

I'm NOT saying that so stop putting words in my mouth!

Seriously?  That right
there is why you are seeing method overrides that aren't documented.

You being a bit bombastic now with this *rolls-eyes*

How can you document something you don't understand?  Furthermore, how
can you have any confidence in your subclass if you don't understand
what its base class is doing?  Do you write unit tests?  How do you
know what to test if you don't understand the code you are
subclassing?

You took my simple statement of... " It is not ALWAYS necessary to
have an intimate knowledge of the base class when creating derived
classes"... and extrapolated THAT nonsense?

Okay let me show you a simple example of not needing to know the base
class intimatly. I'll use the tkSimpleDialog...

class MyDialog(tkSimpleDialog.Dialog):
def body(self, master):
#imagine i created widgets here.

NOW. Would it really be nessesary at this point to have an intimate
knowledge of the base class? Hmm?

Suggesting that no developers document their methods?  Incredible.

Very, very few document clobbered virtual methods. yes.

Clobbered methods have "magic"?  Multi-layered "magic"?

Of course! That is, in the perverted case of Clarke's third law[1]...
"Any sufficiently advanced technology is indistinguishable from
magic"... And since class inheritance can be multi-layered, um, you
get the idea.

 Perhaps when
you take the time to 1) document the base class and 2) understand that
base class before subclassing it, that "magic" will start to look like
reasonable and logical processes.

You're preaching to the choir reverend!

Professional developers document their code.  They take the time to
understand the code they are working on.  If you don't want to do
those things, that's your business, and I'm sure you can find other
people who also don't do those things.  But I really don't think
you'll have much success changing the language to accommodate your
refusal to follow the most basic best practices.

It's not me that needs to change kind sir, it is the community in
general. I document my code. I follow the python style guide. I always
create unit tests. Unfortunately no matter how well i write code i
cannot force others to do so. Heck i have posted many fixes for the
abomination called tkSimpleDialog and not only do they refuse to
upgrade the code, they refuse to even post comments!

This mandate must be handed down from the gods who reside on "Mount
REFUSE-E-OUS to RECOGNIZE-E-OUS a major PROBLEM-O-MOUS"

[1] http://en.wikipedia.org/wiki/Clarke's_three_laws

 

[Chris Angelico]

This mandate must be handed down from the gods who reside on "Mount
REFUSE-E-OUS to RECOGNIZE-E-OUS a major PROBLEM-O-MOUS"

I assume you're trying to reference Mount Olympus where the Greek gods
live, but I'm left thinking more of Mount Vesuvius... possibly not the
best reference for what you're saying.

But once again, Ranting Rick posts something saying that he is perfect
and everyone else needs to change. I can see why you keep landing in
killfiles [1].

ChrisA

[1] http://bofh.ch/bofh/bsmh2.html

 

[rantingrick]

I assume you're trying to reference Mount Olympus where the Greek gods
live, but I'm left thinking more of Mount Vesuvius... possibly not the
best reference for what you're saying.

Actually no i was purposely implying Mt. Vesuvius. You know, the
VOLCANO that erupted and left poor Pompeii in ruins? Here is some text
from the wiki verbatim:

"""Mount Vesuvius is best known for its eruption in AD 79 that led to
the burying and destruction of the Roman cities of Pompeii and
Herculaneum. They were never rebuilt, although surviving townspeople
and probably looters did undertake extensive salvage work after the
destructions."""


I modified the text "slightly" to reflect our current conundrum:

"""Guido and Pydev are best known for their absence around 2000, which
led to the burying and destruction of comp.lang.Python (and the
community as a whole) in ash clowns. They were never rebuilt, although
a few surviving "good" townspeople did undertake extensive attacks
from trolls after the destruction."""

 

[Chris Angelico]

Actually no i was purposely implying Mt. Vesuvius. You know, the
VOLCANO that erupted and left poor Pompeii in ruins? Here is some text
from the wiki verbatim:

Yes, I do know that mountain. But it doesn't have very many gods
sitting on it... maybe a magma elemental, but that's all. Anyhow, this
is quite off-topic for Python I think. (Though not off-topic for
rantingrick.)

ChrisA

 

[alex23]

i cannot force others

If only you really understood that.

 

[Fabio Zadrozny]

Sorry i did explain properly... No NOT marked in the BASE class but
marked in the DERIVED class! My concerns are from a readability
standpoint.

I also like the idea of override annotations and I've created a blog
post at: http://pydev.blogspot.com/2011/06/overrideimplements-templates-on-pydev.html
to explain how I do use it (and in a way that I think should be
standard in Python the same way it's in Java).

Cheers,

Fabio

 

[rantingrick]

I also like the idea of override annotations and I've created a blog
post at:http://pydev.blogspot.com/2011/06/overrideimplements-templates-on-pyd...
to explain how I do use it (and in a way that I think should be
standard in Python the same way it's in Java).

My hat is off to you Fabio! This is a great way to have the method
clobbering stand out. I can read this code and know EXACTLY what is
going on behind the scenes! And it also has the benefit of being very
compact as apposed to a doc-string.


From Fabio's blog verbatim:


class A(object):
def method(self):
pass

class B(A):

@overrides(A.method)
def method(self):
pass

@implements(file.write)
def write(self):
pass

 

[Dave Angel]

My hat is off to you Fabio! This is a great way to have the method
clobbering stand out. I can read this code and know EXACTLY what is
going on behind the scenes! And it also has the benefit of being very
compact as apposed to a doc-string.




class A(object):
def method(self):
pass

class B(A):

@overrides(A.method)
def method(self):
pass

@implements(file.write)
def write(self):
pass

Just one small change. Change those @ signs to #, and you've got a
deal. Then make a minor change to pylint, and you have a way to enforce it.