bytecode non-backcompatibility

[Maurice LING]

You're right - and byte codes *as they exist now* aren't an acceptable
middle ground, either. The problem isn't that the interpreter might
change (that can be defeated by distributing the interpreter with the
bytecode files), but that the byte code can be decompiled to get your
Python source back. See <URL:
http://www.crazy-compilers.com/decompyle/ > for an example.

I know that bytecodes can be decompiled back to python source. Similarly
Java class files can be decompiled back to Java source files quite
easily. IANAL, however, I do think that reverse engineering can be seen
as a purposeful act in the eyes of law. I remembered that EU Council
Directive 9/250/ECC does explicits the allowable scope of decompilation
(for interoperability, preserving expression of ideas, achieving
performance of objectives).

For companies, I think it is touchy to ask permission to release the
source codes even though they may be alright with releasing the same
Java JAR file into the community.

We do know that regardless of whatever methods of compilation there may
be, if some genius is willing to spend sufficient time to crack a
program. Even if he has to read the binary codes, it will still break.
But that is a wilfull act rather than just looking at the source codes
when it is distributed with it. It's just like picking up your money
from the ground if you had dropped it as compared to pickpocketing you
as compared to hold you at knife point and robbing you...... Same end
result but it is viewed differently in the eyes of law and morals.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Now I understand that Python bytecodes are only dealing with pure python
source codes. However, the same question lies, how can it (set of
bytecodes) be made stable, like Java bytecodes, which are pretty stable?
Perhaps a better question will be, what techniques Java VM designers
use that enables Java class files (and JAR files) to be stable and
usable across versions that is lacking in Python?

The Java class file format is not any more stable than the Python .pyc
format - I think every major release of the JVM has changed details
in the class file format.

The difference is that the newer JVMs accept old class files, by
means of separate loader code for each class file version that was
ever released (and very possibly also support for class file versions
that were never released).

The same approach would be possible in Python, but nobody has
contributed code to do so. It is unlikely that future Python
versions will provide such compatibility with the current byte
code format, either, unless somebody steps forward and volunteers
to maintain that compatibility.

Maintaining this backwards compatibiltiy is a tedious and boring
task, given the many much easier alternatives, so that volunteers are
unlikely to jump in. Sun manages to provide the compatibility by
paying somebody to actually maintain it.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

technicalities are wrong but situation remains unchanged.

For C modules, it is very likely that new versions of Python
will continue to break the ABI, by changing the layout of
structures.

The most straight-forward way to deal with it as a sysadmin
or user is to install multiple versions of Python on a single
machine. If Fink considers python2.4 as a replacement for
python2.3, then this is a flaw in Fink. In Debian, there is
a python package, which currently depends on python2.3.
Sometime in the future, it will depend on python2.4. Users
which update will then get python2.4, however, python2.3
will remain installed and usable, with all the extension
modules that were installed for it.

Regards,
Martin

 

[Terry Reedy]

From a technical perspective, I can accept that .pyc files are private
and temporary. To a certain extend, it does helps development cycle.
Every time I amend my source codes, I just run it without having to
consider or needing to re-compile the source files.

From a user perspective, source code is the run-time program.

The idea of having to release the program or library as source files does
ring alarms in many executives in corporate world.

I understand that people use Python while resisting its purpose and design.
But I also understand that it is *their* responsibilty to hide their code,
which may possibly mean not using Python, or which may mean developing
proprietary methods to translate to something designed to *not* be
readable. For all we know, some of the developers have been paid to do
exactly that -- and not talk about it.

Python is *designed* for human readability. That is one of its big
features! The same also seems somewhat true for CPython's bytecodes,
especially when disassembled with the dis module that comes with the
interpreter. You even get all the object names included in the code
object.

Terry J. Reedy

 

[Maurice LING]

For C modules, it is very likely that new versions of Python
will continue to break the ABI, by changing the layout of
structures.

The most straight-forward way to deal with it as a sysadmin
or user is to install multiple versions of Python on a single
machine. If Fink considers python2.4 as a replacement for
python2.3, then this is a flaw in Fink. In Debian, there is
a python package, which currently depends on python2.3.
Sometime in the future, it will depend on python2.4. Users
which update will then get python2.4, however, python2.3
will remain installed and usable, with all the extension
modules that were installed for it.

Regards,
Martin

Fink does not consider python2.4 to be a replacement for python2.3. In
fact, you can install python2.2, 2.3 and 2.4 in the same machine with
Fink. It will maintain 3 sets of libraries as /sw/lib/python2.2,
/sw/lib/python2.3 and /sw/lib/python2.4. The chore is that when say Fink
installs python2.4, all the libraries in /sw/lib/python2.3/site-packages
have to be re-installed into /sw/lib/python2.4/site-packages, one by
one. There is no simple way of doing that... which makes any system
admin needing to re-install 50 3rd party libraries into
/sw/lib/python2.4/site-packages a big task, as well as satisfying the
necessary dependencies.

So if C extension API (or whatever that is really called) is stable, the
system admin can just copy all of /sw/lib/python2.3/site-packages into
/sw/lib/python2.4/site-packages and it should work. From what you've
said, it seems that this isn't possible. So my alternative solution is
that PyPI have a mechanism to maintain what had been installed in the
site-package directory and to download the libraries and install into
the new site-package directory...

What do you think?

Cheers
maurice

 

[Terry Reedy]

So my alternative solution is that PyPI have a mechanism to maintain what
had been installed in the site-package directory and to download the
libraries and install into the new site-package directory...

What do you think?

I doubt anyone disputes that upgrades are more hassle than we would like.
My main point was that freezing CPython's technology is not the solution.
Any other upgrade helper that you can contribute will be welcome.

Terry J. Reedy

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

So if C extension API (or whatever that is really called) is stable, the
system admin can just copy all of /sw/lib/python2.3/site-packages into
/sw/lib/python2.4/site-packages and it should work.

It would be counter-productive to make it stable, as this would render
certain desirable changes impossible. Backwards compatibility on C API
(source) level is a goal, binary compatibility within a 2.x release
is a goal, binary compatibility across 2.x releases is not.

[automatically install a set of packages]

What do you think?

That would certainly be possible. Contributions are welcome.

Regards,
Martin

 

[Maurice LING]

I doubt anyone disputes that upgrades are more hassle than we would like.
My main point was that freezing CPython's technology is not the solution.
Any other upgrade helper that you can contribute will be welcome.

Terry J. Reedy

So there is no way of releasing a close-source application written in
Python?

Forget about reverse engineering or decompiling, things like EU Council
Directive, Berne Convention , and legislation of individual countries
are to be used and argued for handling what decompilation is allowed or
not allowed. It does make a difference between looking at the codes
when it is given and performing an act of decompilation to look at the
codes. As mentioned in one of my postings, it may be analogous to the
difference between you dropping some cash on the ground and I take it,
and I performing the act of pickpocketing and take your money.

I am not advocating close source in any sense and I believe the choice
of open or close source distribution is individual's and corporate's rights.

Cheers
Maurice

 

[Maurice LING]

[automatically install a set of packages]

What do you think?

That would certainly be possible. Contributions are welcome.

Regards,
Martin

I've emailed to catelog-sig mailing list and is still waiting to hear
something. Currently, I have no idea of the structure of PyPI. I hope I
can hear from them soon and generate some starting points...

Maurice

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

So there is no way of releasing a close-source application written in
Python?

I fail to see the relationship between the statement and your question.

Again, whether or not the C API is frozen has nothing to do with
whether or not the bytecode format is frozen. So any implication
from not freezing the C API to not being able to ship byte-code
only is invalid.

As for only shipping byte code: sure, that is certainly possible,
and people do that all the time.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

I've emailed to catelog-sig mailing list and is still waiting to hear
something. Currently, I have no idea of the structure of PyPI. I hope I
can hear from them soon and generate some starting points...

Posting questions is not the only way to find answers. The source code
of PyPI is available: sf.net/projects/pypi.

Regards,
Martin

 

[Maurice LING]

Posting questions is not the only way to find answers. The source code
of PyPI is available: sf.net/projects/pypi.

Regards,
Martin

I've browsed the source codes of PyPI in sf.net, nothing pops up as very
useful to me... I reckoned that sitting in here or hoping for replies
from catelog-sig isn't going to help much.

On that, I've prototyped a very simple proof-of-concept of what I have
in mind, regarding a Fink-like tool for Python's 3rd party libraries
management. Please bear in mind that this is a proof-of-concept
prototype, really bare bones. It can be found in 'centipyde' module in
sf.net/projects/ib-dwb. In case the CVS hadn't updated by the time
someone reads this, the directory layout is this:

.../centipyde
.../centipyde/centipyde.py
.../centipyde/pgkinfo
.../centipyde/pgkinfo/ply15.info


ply15.info contains the following text (pretty much modelled against Fink):

package=ply15
maintainer=.
dependencies=.
downloadurl=http://systems.cs.uchicago.edu/ply/ply-1.5.tar.gz
prebuildscript=tar zxvf ply-1.5.tar.gz
sourcedir=ply-1.5
buildscript=python setup.py build
installscript=sudo python setup.py install


centipyde.py is the following:
=====================================================

"""
Author: Maurice H.T. Ling <[email protected]>
Copyright (c) 2005 Maurice H.T. Ling
Date created : 28th April 2005
"""

PKGINFOPATH = 'pkginfo'
INSTALL_LOG = 'install.log'

import os
import string
import sys

def install_package(package_name):
f = open(os.getcwd() + os.sep + PKGINFOPATH + os.sep + package_name
+ '.info', 'r')
install_info = {}
for line in f.readlines():
line = string.split(line, '=')
if line[1][-1] == os.linesep:
install_info[line[0]] = string.strip(line[1][:-1])
else: install_info[line[0]] = string.strip(line[1])
f.close()
print "Package Installation Information: " + str(install_info)

os.system('curl -O ' + str(install_info['downloadurl']))

preinstall = []
preinstall = string.split(install_info['prebuildscript'], ';')
for cmd in preinstall: os.system(cmd)

cwd = os.getcwd()
print cwd
os.chdir(os.path.join(os.getcwd(), install_info['sourcedir']))
print os.getcwd()

buildscript = []
buildscript = string.split(install_info['buildscript'], ';')
for cmd in buildscript: os.system(cmd)

installscript = []
installscript = string.split(install_info['installscript'], ';')
for cmd in installscript: os.system(cmd)


if sys.argv[1] == 'install':
install_package(sys.argv[2])

=====================================================

When I run "python centipyde.py install ply15", PLY1.5 gets downloaded
from David Beazley's website, uncompressed and installed into the
site-package.

All comments and code donations are welcomed.

Cheers
Maurice

 

[Mike Meyer]

If you are installing packages into your home directory, then sudo is
not needed. But if you are installing it for everybody's use, then it
is necessary. Fink runs using superuser privileges.

No, sudo isn't necessary. It isn't provided by
default for all Unix installations, so Python would have to add a
dependency on it, which would be a bad thing.

sudo is sufficient. Other means are also sufficient. It would be wrong
for Python to assume a specific Unix security model (i.e. - "sudo")
for installations.

<mike

 

[Mike Meyer]

So if C extension API (or whatever that is really called) is stable,
the system admin can just copy all of /sw/lib/python2.3/site-packages
into /sw/lib/python2.4/site-packages and it should work. From what
you've said, it seems that this isn't possible.

Correct. This isn't possible. It's not clear it's *desirable*,
either. For one thing, code can fail to port across versions, so
things will break at random (instead of all at once, I admit). For
another, I dislike the idea of a module sitting unupdate for long
periods of time (decades, say). Updating Python provides a good excuse
to update all the python modules, etc.

Now, it would be nice if Python provided an easy way to do the
update. The FreeBSD packaging system happens to provide a way to
automate this process, but it's still not exactly easy.

So my alternative
solution is that PyPI have a mechanism to maintain what had been
installed in the site-package directory and to download the libraries
and install into the new site-package directory...

PyPI is the wrong place to maintain a record of installed
software. Distutils is the software that does the installing (ok, on
most modules, anyway), and is the software that should record what
packages are installed.

Neither distutils nor PyPI has a "download" functionality. For that
matter, PyPI doesn't have a hard requirement on listing where a module
comes from. I'd say distutils is the software that should provide it,
so I can say:

python setup.py install --depends

and get all the dependencies. But that's not at all clear. PyPI will
have to cooperate by providing URLs to packages, instead of to pages.

<mike