platform-specific overrides of functions and class methods (expandingon imputils demo code)

[lkcl]

i've just had to put something together for pyjamas-desktop which may
prove to be useful to other people, so i'm pointing people in its
general direction, for archive purposes.

the purpose behind the platform override system is to allow
implementations of a common API, in python, to share the majority of
their codebase (modules, classes, functions etc.) _but_, for _very_
specific platform implementation purposes, allow for "overrides".

example:

test.py
def test():
print "hello"

platform/testOverridingPlatformName.py:
def test():
print "this is not kansas any more"

code is here:
http://pyjamas.svn.sourceforge.net/viewvc/pyjamas/trunk/pyjd/

modules to pay attention to: importers.py (a modified version of Demos/
imputils/importers.py) and modcompile.py. setup/init function is very
simple - see __init__py.in last few lines.

on reading modcompile.py you may be forgiven for going "wtf????" but
basically PlatformParser loads the python source; turns it into an
AST; then also the "platform-specific" version is loaded and turned
into an AST; then, the two ASTs are handed to the "merge" function
which does a top-level "replace" of functions and a top-level
"replace" of class methods.

_theen_ the resultant "merged" AST is handed to a module which was
derived from compiler/pycodegen.py - none of the code in there can
cope with being handed an already-compiled AST so it was necessary to
make a class that did. looking around on the internet i find quite a
few people asking about how to do this, so ... take a look at
modcompile.Module, and how it's used. very simple.

i've disabled saving and detection of .pyc files for now, because the
job of loading code from .pyc should really be done by PlatformParser
(which understands the concept of... duh, a platform). the _last_
thing that you want to happen is to run one platform's code, generate
some .pyc files, then change a config file to run a different back-end
platform, for example (which is possible with pyjamas-desktop) and end
up running the _wrong_ platform-specific code.

for those people wondering, "why in god's green earth would anyone
want to _do_ such a thing???" it's quite simple: the alternative is a
complete dog's dinner, in pyjamas:

def do_some_DOM_manipulation(document):
# heeeere we go...
if platform == 'hulahop' # this is for XULrunner
do_some_stuff()
elif platform == 'pywebkitgtk':
do_something_different()
elif platform == 'mshtml' # for pywin32 and comtypes....
do_something_dreadful()
else:
do_the_default_here()

now imagine that across an API with ... four hundred functions.

i just... couldn't bring myself to do that. not when the pyjs (python-
to-javascript) compiler _already_ solved this problem (thanks to james
tauber) by deploying the AST merge concept.

all i did was merge that with the imputils demo code so that, rather
than at compile-time the pyjs compiler goes and generates five
platform-specific versions of the same application, pyjamas-desktop at
_run_ time can do exactly the same thing.

but - it's a generic enough idea to be of value elsewhere - for
example, all the loooovely code in e.g. the setup distutils? all
those looovely "if os.platform == 'win32'", and "if sys.this = 'posix'
" could be replaced with platform-specific overrides that got merged
at run-time, thus _dramatically_ simplifying the look and the
useability of the code.

enjoy.

l.

 

[lkcl]

thought that people might like to know: i found also that imputil, the
standard python module, was lacking the necessary complexity in being
a substitute for the standard __import__ function.

the additions required were very simple:

# note the addition of level=-1 which is ignored

def _import_hook(self, fqname, globals=None, locals=None,
fromlist=None,
level=-1):

.....

# Grrr, some people "import os.path" or do "from os.path
import ..."
if len(parts) == 2 and hasattr(top_module, parts[1]):
if fromlist:
return getattr(top_module, parts[1])
else:
return top_module

# assume that the module has already been imported,
# walk from top_module to find it.
mod = top_module

for k in parts[1:]:
if not hasattr(mod, k):
print "no mod", mod, k, parts
mod = None
break
mod = getattr(mod, k)

if mod:
return mod

# If the importer does not exist, then we have to bail. A
missing
# importer means that something else imported the module, and
we have
# no knowledge of how to get sub-modules out of the thing.
raise ImportError, 'No module named ' + fqname

it's the module-walking that's been added: going from the "top"
module, looking for an attribute of the split-parts. this code makes
the assumption that if top module already exists, and it has a sub-
module, and that has a sub-sub-module, and that has a sub-sub-sub-
module etc. then it is perfectly reasonable to return that pre-
existing, already-imported [sub-]+ module as the return result.

the reason why this had to be added is because comtypes.gen.{some
module} auto-generator relies on it, and calls __import__ *direct*.

if somebody would like to add this to the python bugtracker, as a
contribution, that would be great. alternatively, you might like to
have a word with the python developers to get them to remove the
censorship on my contributions.

l.

 

[Aahz]

if somebody would like to add this to the python bugtracker, as a
contribution, that would be great. alternatively, you might like to
have a word with the python developers to get them to remove the
censorship on my contributions.

Excuse me? What censorship?
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"Given that C++ has pointers and typecasts, it's really hard to have a
serious conversation about type safety with a C++ programmer and keep a
straight face. It's kind of like having a guy who juggles chainsaws
wearing body armor arguing with a guy who juggles rubber chickens wearing
a T-shirt about who's in more danger." --Roy Smith

 

[lkcl]

Excuse me? What censorship?

i was ordered to cease contributing to python. the cause was the
provision of a port of python to run under mingw32, which passed all
but eight of the regression tests.

l.

 

[Terry Reedy]

i was ordered to cease contributing to python.

As I remember from the public discussion, you were asked to cease using
the bugtracker as a progress blog ...

the cause was the
provision of a port of python to run under mingw32, which passed all
but eight of the regression tests.

and wait until you had a complete patch to submit. There is also a real
question of whether such ports should be folded into the main CPython
code base, as some have been, or maintained as separate projects, as
other have been.

tjr

 

[Aahz]

As I remember from the public discussion, you were asked to cease using
the bugtracker as a progress blog ...


and wait until you had a complete patch to submit.

That matches my memory as well. Luke, your contributions would
definitely be welcome, but contributions must follow some process.

 

[lkcl]

lkclwrote:

As I remember from the public discussion, you were asked to cease using
the bugtracker as a progress blog ...

i used the bugtracker as a substitute for svn access, so that other
people who may be interested could follow, pick up and contribute.

to describe valuable contributions as "a prrrogress blllllooggggg"
makes it sound like the work is at the same level as much of the
mindless drivel found on e.g. wordpress.

if that's how it was being viewed, then it's no wonder there were
complaints.

at the time that i was ordered to cease contributing, the work done
passed all but 8-12 of the standard python regression tests. out of
several thousand.

and wait until you had a complete patch to submit.

1) i've stopped. it's declared complete. there. it can now be
added. does that help?

2) you know as well as i do that there is never going to be a
"complete" patch. no-one in the world is a "perfect" coder. work is
_always_ ongoing. there is _always_ going to be "a better patch".

3) "wait until there is a complete patch" flies in the face of the
basic tenets of both commercial _and_ free software development. you
_never_ "wait until the work is completed". release early, release
often, remember?

There is also a real
question of whether such ports should be folded into the main CPython
code base, as some have been, or maintained as separate projects, as
other have been.

the mingw32 port comprises two (necessary) features: 1) cross-
compiling support, which the mingw32 port uses to create python.exe
(from a posix system) 2) actual support for mingw32-compiled python,
including proper support on ILP32 systems, checking support for
features properly using autoconf tests not "#ifdef WIN32" etc. etc.

cross-compiling of cpython is tricky as hell, starting from pgen and
going from there, to create python.exe. to be able to run that - in
order to compile the modules and run the regression tests is _even
more_ tricky. but - it's been done.

roumen and the other contributors to the cross-compile support have
consistently had their contributions considered to be worthless,
basically, for nearly five years, now. _despite_ clear evidence
indicating a need, and clear requests for cross-compiling to be
included in the standard python distribution.

for example, from the gentoo team, who have a bitch of a job cross-
compiling python for embedded-mips systems.


then there's the issue of continuing to rely on the proprietary
microsoft compiler toolchain. you _know_ that's going to cause
problems, ultimately, with an important free software project such as
python.

so to throw mud in the face of people willing to think ahead and put
in the work to get you [PSF / python-devs] out of a potentially
difficult [political/strategic] situation is just ... i don't
understand it. i even solved the issue of compiling python under
mingw32 using assemblies, so that it could be linked against MSVCRT80,
91 etc.

these kinds of multiple interdependent technical issues aren't the
sorts of things you "put into a single patch and then stop" - you keep
on at it, picking at one part of the pattern, moving to the next
issue, pick some more, solve that, and keep moving.

so - these are the kinds of things i received [not exact words]:
"please stop submitting continual patches, we think it's about the
same level of value as some of the trash we see on wordpress.com".
"please stop doing work because we think it's worthless". "we don't
really like and don't really want to maintain the win32 port of python
anyway, so why are you bothering to work on it?"

do you notice a theme, here? there isn't anything which is
particularly encouraging. or places any value on the work being
done. or indicates a willingness to extend free software.

this theme was so completely contrary to what i was expecting
[openness, trust and respect] that i just... went "does not compute"
and continued coding. i don't mean that in a flippant way - i really
_mean_, my mind went "this message [from e.g. martin] doesn't match my
expectations of a free software project developer/contributor. what's
the next technical challenge i have to solve?" note the discontinuity
in the internal dialog between those two sentences. there was no
_judging_ of the message received; no "pffh. that idiot. let's spend
some time and energy _actively_ dismissing the person and the
message". it was _literally_: "does not compute. equals zero. next
issue on list: do coding". it's a pathological approach, i know - but
it's one that solves technical issues with extreme [some say alarming]
speed.

anyway. it was an interesting experience. and you can make use of
someone with such capabilities, and _not_ get stressed, and work out
how to make use of them -or you can throw their work in their face,
and move the goalposts around, close all their bugreports (making up
reasons where possible) - all to get them to "go away". "stop
increasing my stress levels".

overall, from the experience, i did get the overall impression that
the core developers are slightly overwhelmed, and don't have time to
focus on anthing other than what's already on "the roadmap". i.e.
"unscheduled" major free software contributions - such as ports to new
systems - are viewed as a burden rather than a responsibility that
should, without question and without hesitation, be incorporated and
encouraged, in order to increase the reach of python and in order to
build a stronger, larger python community.

one of the arguments utilised to discourage the work on the mingw32
port was that "it would have to be supported - by us. officially."
such statements are self-fulfilling, and increase the "burden of
responsibility" and the stress.

a very sensible person told me that stress is where the mind has an
internal picture (expectations / viewpoint), makes a comparison with
the outside world, and the discrepancy is so large that the mind
CANNOT COPE.... and seeks to place BLAME on the EXTERNAL world.

if people are willing, there are _always_ ways in which contributions
can be accepted, such that free software can progress. it's just that
in the [extremely rapid] development cycle of the mingw32 cross-
compiled port of python, i didn't meet the "expectations" of the
python developers. there's nothing i can do about peoples' _internal_
expectations, and i'm really sorry, but i don't have the social-
interaction-capacity to _understand_ peoples' expectations [i don't
get enough practice]. so, i "make do" by assuming that free software
people are always going to be receptive, enthusiastic, open and
encouraging, and that, when they encounter an experience that
indicates that they might fall short of the "ideal" free software
project, they'll move sharp-ish to correct the issue encountered.

my limitations are that if they're _not_ any or all of those things,
or if those corrections _aren't_ made, i simply do the mental
equivalent of "syntax error. decrease priority of message. pop(next
task on stack)". i don't _forget_ the incoming message - it just
gets... a lower priority, and thus appears to have been ignored. "too
much energy will be consumed processing this discrepancy between _my_
ideal expectations of how free software contributors should
communicate, and work on free software, and this message. that's a
distraction from spending energy on getting free software work done.
decision equals easy: work on free software".

that's what happened to you, martin. it looked like i was ignoring
you. it also didn't help that i also was sufficiently busy focussing
approx 11 hours non-stop a day for several weeks that some of the
things you asked me to stop doing, so as to reduce your stress levels,
i _literally_ didn't see them.

martin and guido: you asked me to apologise, some months ago. to do
so requires that i express clearly what it was that i was apologising
_for_, as what i'd already apologised for simply wasn't enough and
wasn't clear enough, (and my previous apologies had several areas
where people assumed the worst, and assumed that i was trying to
disguise ego trips as apologies). that's taken nearly a year to go
by, for me to subconsciously work on what to say, and work through it,
above.

so - with the context above: you can understand and appreciate this:
that i'm sorry that i have different expectations of how free software
projects should operate, and for not knowing what the right thing to
do is when my expectations aren't met. as in - i _really_ don't
know. i seem to have this strange belief that if you solve the
technical challenges, people will work out how to move the goalposts
_towards_ incorporating the solutions, rather than away -
_irrespective_ of the "social" issues, placing "social interaction"
issues at absolute rock-bottom priority against achieving the
technical goal. i'm sorry that this ethos and the beliefs and
expectations that i have cause people distress.

l.