bytecode non-backcompatibility

[Maurice LING]

Hi,

I've been using Python for about 2 years now, for my honours project and
now my postgrad project. I must say that I am loving it more and more
now. From my knowledge, Python bytecodes are not back-compatible. I must
say that my technical background isn't strong enough but is there any
good reason for not being back-compatible in bytecodes?

My problem is not about pure python modules or libraries but the problem
is with 3rd party libraries with C bindings (not python pure). It means
that with every upgrade of python, I have to reinstall all my 3rd party
libraries which can be quite a bit of work...

I do hope this problem will be sorted out some day.

Cheers
Maurice

 

[Robert Kern]

Hi,

I've been using Python for about 2 years now, for my honours project and
now my postgrad project. I must say that I am loving it more and more
now. From my knowledge, Python bytecodes are not back-compatible. I must
say that my technical background isn't strong enough but is there any
good reason for not being back-compatible in bytecodes?
>
My problem is not about pure python modules or libraries but the problem
is with 3rd party libraries with C bindings (not python pure).

Then this has nothing to do with bytecode incompatibility. Only
pure-Python modules get compiled to bytecode. You mean binary
compatibility of the C API.

If you're going to have significant improvements in the core
interpreter, you have to break binary compatibility from time to time.

It means
that with every upgrade of python, I have to reinstall all my 3rd party
libraries which can be quite a bit of work...

I do hope this problem will be sorted out some day.

This problem is mitigated somewhat by the fact that the devs keep binary
compatibility within a major revision series (e.g. 2.3.0, 2.3.1, 2.3.2,
....).

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Fuzzyman]

Hi,

I've been using Python for about 2 years now, for my honours project and
now my postgrad project. I must say that I am loving it more and more

now. From my knowledge, Python bytecodes are not back-compatible. I must
say that my technical background isn't strong enough but is there any

good reason for not being back-compatible in bytecodes?

It *shouldn't* be a problem for pure python modules. The interpreter
will recognise that the bytecode has the wrong 'magic number' and
recompile.

Seeing as a new major version of python will probably be installed in a
new directory at the very least you will have to copy the modules
across. If they have install files (or a setup.py) wouldn't it be
better to use that *anyway* ?

My problem is not about pure python modules or libraries but the problem
is with 3rd party libraries with C bindings (not python pure). It means
that with every upgrade of python, I have to reinstall all my 3rd party
libraries which can be quite a bit of work...

It is a nuisance. It's more of a nuisance when third part modules with
'c' components are compiled for the new version of python (a windows
specific problem).

I did find that the last upgrade forced me to evaluate which extensions
I actually used. The answer was 'a few less than I thought'. It became
a good opportunity to spring clean my 'site-packages' folder.

Best Regards,

Fuzzy
http://www.voidspace.org.uk/python

 

[Terry Reedy]

I think the OP is confusing three different CPython implementation
features -- bytecodes, marshal format, and C API, all of which change
pretty slowly -- and a MS Windows/C (dis)feature.

CPython bytecodes only concern code written in Python but are mostly not a
concern because recompilation is automatic when needed. The exception is
code that directly mucks around with bytecodes, especially the intentially
semi-undocumented numerical codes.

Code objects also include marshaled object values. The code object for
'haha = 987654321' must contain representations for the constants 'haha'
and '987654321' as well as the bytecode for the assignment. So even if
bytecodes remain the same, marshal can change (as it did for 2.4, I
believe, and as it will for floats in 2.5 to fix a bug) and trigger auto
recompiles.

C extensions interact with the interpreter via function calls that
constitute the C API. As much as possible, the developers consciously
avoid changes that break old code or even old binaries. Guido has claimed
that for Linux, there are extension binaries compiled for 2.0 that still
work with 2.4.

However, for MS Windows/C, there is a 'feature' with respect to DLLs that
requires recompilation of extensions to work with a new version of the
Python DLL, even if the C API is unchanged (or so people who understand
this have said). So the OPs complaint about having to get and install new
extension binaries for each Python version might well be directed to MS.

This does not, of course, negate the idea that it would be nice if the
update process were somehow make easier. Indeed, the more-batteries
included distributions from ActiveState and Enthought specifically aim at
this.

Terry J. Reedy

 

[Maurice LING]

It is a nuisance. It's more of a nuisance when third part modules with

'c' components are compiled for the new version of python (a windows
specific problem).

I did find that the last upgrade forced me to evaluate which extensions
I actually used. The answer was 'a few less than I thought'. It became
a good opportunity to spring clean my 'site-packages' folder.

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?

I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine the
trouble if I do a "fink selfupdate" and "fink update-all" and finds that
my Python is replaced by a newer version, say Python 2.4. Then all my
3rd party libraries in /sw/lib/python2.3/site-packages are un-usable...
And I will have to re-download and re-install all the libraries again.

Yes, I do agree that it is a nice time to clean out "site-packages" but
imagine the work of a system admin who has to re-install 50 packages...
I do believe that we can use some help here, if possible... Perhaps
someone can enlighten us on the technicalities of "so" files and/or C
bindings in Python...

Now I understand that Python bytecodes are only dealing with pure python
source codes. However, the same question lies, how can it (set of
bytecodes) be made stable, like Java bytecodes, which are pretty stable?
Perhaps a better question will be, what techniques Java VM designers
use that enables Java class files (and JAR files) to be stable and
usable across versions that is lacking in Python?

Thanks.

Cheers
Maurice

 

[Dennis Lee Bieber]

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?

Linux equivalent of a Windows DLL -- "shared object" file I
think... a binary object file that is dynamically loaded/linked at run
time. The application binding just knows the name/offset into the object
for each function call.

I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine the
trouble if I do a "fink selfupdate" and "fink update-all" and finds that
my Python is replaced by a newer version, say Python 2.4. Then all my
3rd party libraries in /sw/lib/python2.3/site-packages are un-usable...
And I will have to re-download and re-install all the libraries again.

If Fink stuffs a Python 2.4 in your /sw/lib/python2.3/ directory
-- blame Fink! (whatever Fink is...) A Python 2.4 install would be in a
..../python2.4/... directory, and your 2.3 should be untouched. The
/search path/ might have been changed to find 2.4 first, and /that/
might be a problem.

--

 

[Mike Meyer]

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?

Shared object files. They're executable code that can be linked into a
program at runtime. Like DLL's on Windows.

I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine
the trouble if I do a "fink selfupdate" and "fink update-all" and
finds that my Python is replaced by a newer version, say Python
2.4. Then all my 3rd party libraries in
/sw/lib/python2.3/site-packages are un-usable... And I will have to
re-download and re-install all the libraries again.

Yeah, it's pain. It would be nice if setup.py kept a list of installed
packages that you could extract before updating, so you would know
what needed to be updated.

Last time I updated, I used the systems package handler. I asked it
for the names of all the Python packages that were installed and saved
that to a file. Then I installed the new version of Python. Then I
edited the package list to turn it into a series of package update
commands, and ran that file.

Now I understand that Python bytecodes are only dealing with pure
python source codes. However, the same question lies, how can it (set
of bytecodes) be made stable, like Java bytecodes, which are pretty
stable? Perhaps a better question will be, what techniques Java VM
designers use that enables Java class files (and JAR files) to be
stable and usable across versions that is lacking in Python?

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

<mike

 

[Maurice LING]

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

Thanks Mike,

My arguments to the developers will be:

1. Python had gone from a purely scripting language to a general purpose
programming language.

2. The current compilation scheme (compiling to bytecode as and when it
is needed) works well for scripting purposes but is less desirable in
commercial settings. Less distribution happens when it is used purely
for scripting purposes, such as system maintenance or tuning.

3. Using Python in commercial settings will usually require distribution
of resulting software and it is may or may not be desirable to
distribute source codes as well. Unless the application is frozen,
distributing source code is a must.

4. One advantage that Java platform has is that it does not require the
release of source files and still promotes platform-independence.

5. Unstable bytecodes makes updating to a newer version of Python very
tedious and risk breaking old scripts, if they uses C modules.

6. Unstable bytecodes also makes research work into Python, such as,
just-in-time compilation and just-in-time specialization unfavourable as
they may only be applicable to a specific version of Python. There is
much less chance of getting a project grant than if the same project is
applied for Java (stable bytecodes).

What other point are there?

I may be chopped by saying this but by having a stable set of bytecodes,
we may lose a means of optimization. But we may gain more from filling
the need for commerical distribution of applications writing in Python
and ease of upgrading...

At current stage, every time a new version of Python is installed in my
server or system, I have to test and ensure the needed libraries are
there... which may be a horror in corporate settings. Couple that with
messy dependencies of the libraries to be installed. I remembered the
time I was trying to install Eric 3, the dependencies makes me want to
give up... I have to install Qt, then pyQt, then something else, then
Eric3. Imagine you need 10 of those libraries... which may happen... I
am not yet masochistic enough to take pleasures in this...

I also hope some python developers are reading this thread as well......
Call this a desperate plea from some of us......

Cheers
Maurice

 

[Steve Holden]

Thanks Mike,

[... arguments about bytecode stability ...]

What other point are there?

I may be chopped by saying this but by having a stable set of bytecodes,
we may lose a means of optimization. But we may gain more from filling
the need for commerical distribution of applications writing in Python
and ease of upgrading...

At current stage, every time a new version of Python is installed in my
server or system, I have to test and ensure the needed libraries are
there... which may be a horror in corporate settings. Couple that with
messy dependencies of the libraries to be installed. I remembered the
time I was trying to install Eric 3, the dependencies makes me want to
give up... I have to install Qt, then pyQt, then something else, then
Eric3. Imagine you need 10 of those libraries... which may happen... I
am not yet masochistic enough to take pleasures in this...

I also hope some python developers are reading this thread as well......
Call this a desperate plea from some of us......

Cheers
Maurice

There's enough attention to backward compatibility that the chance of
code breakage is quite small for pure-Python modules. Given that case,
all we really need for them is a suitable way of moving them forward to
an updated distribution (though, of course, the module authors may have
brought out new versions that use more advanced language features, there
is no real compulsion to migrate to those if you seek stability).

I don't believe that the major problem is the changes to the bytecodes -
extension modules (those written in C and/or C++) don't generally use
the bytecodes anyway. The real problem is the API to the interpreter,
which again the developers retain the right to change between minor (but
not micro) versions.

regards
Steve

 

[Mike Meyer]

Thanks Mike,

My arguments to the developers will be:

1. Python had gone from a purely scripting language to a general
purpose programming language.

I think that's irrelevant. It just means that there are more
applications around that have to be dealt with when you update.

2. The current compilation scheme (compiling to bytecode as and when
it is needed) works well for scripting purposes but is less desirable
in commercial settings. Less distribution happens when it is used
purely for scripting purposes, such as system maintenance or tuning.

The solution with the current situation depends on what you mean by
"commercial settings".

3. Using Python in commercial settings will usually require
distribution of resulting software and it is may or may not be
desirable to distribute source codes as well. Unless the application
is frozen, distributing source code is a must.

People have recommended that you distribute the Python interpreter
with commercial applications. This makes installing them much easier
for the end user. It also means the byte code and the interpreter will
always match.

4. One advantage that Java platform has is that it does not require
the release of source files and still promotes platform-independence.

Java is only "platform-independent" to the degree that the jvm and
libraries you need for your application are already available. A
quick google finds applications that require jvm versions - so I
suspect that you can't always run Java code built for a new version of
the jvm on older jvms. So you have to make sure the target platform
has a recent enough implementation of the jvm installed. You may have
problems if you try using a different groups jvm as well. I haven't
looked into that in a number of years.

So you can't just ship java bytecodes to someone expecting they'll be
able to run it out of the box. They may well need to update their java
environment, or install a second one (I recall one time having four
Java products that took in total three different jvms to run. Bleah.)

This isn't really different from Python. In both cases, the end user
has to have a suitable version of the bytecode interpreter and
libraries installed. Java is a little better in that they provide
backwards compatability.

5. Unstable bytecodes makes updating to a newer version of Python very
tedious and risk breaking old scripts, if they uses C modules.

Unstable bytecodes have nothing to do with these problems.

The current CPython installation process puts the python command and
the libraries for different versions in different directories. This
allows you to have multiple versions installed so you can keep old
scripts working with the old version until you've had time to test
them. It also makes life much easier on the developers, as they can
have a development version installed on the machine at the same time
as they have a production version without breaking the old scripts. It
also means you have to reinstall all your modules on the new
installation - which is what makes the update process tedious for me.

Now, this could be mitigated by having Python libraries installed in
the same location for all versions. You could fix all the bytecode
files by running compileall.py as a script. Sometimes, a module won't
work properly on a new version of Python, and will have to be
updated. You'll have to find those by trial and error and fix them as
you find them. You'll also have to recompile all the C modules, which
will break running scripts on the old interpreter.

Under the currewnt system, old scripts that are run by the new
interpreter will break if all the modules they need aren't installed
yet. Once they're installed, the scripts should work.

If you have scripts that use C modules that aren't installed in the
standard Python search path, they may well break on the new
interpreter until you recompile them. Having both versions of the
interpreter available makes this tolerable.

The cure I proposed seems worse than disease. If you've got a better
solution, I'm sure we'd be interested in hearing it.

6. Unstable bytecodes also makes research work into Python, such as,
just-in-time compilation and just-in-time specialization unfavourable
as they may only be applicable to a specific version of Python. There
is much less chance of getting a project grant than if the same
project is applied for Java (stable bytecodes).

This doesn't seem to have stopped the Psyco project from turning out a
jit compiler. I can't speak to grant applications, though. If you're
really interested, you could apply to the PSF for a grant.

I may be chopped by saying this but by having a stable set of
bytecodes, we may lose a means of optimization. But we may gain more
from filling the need for commerical distribution of applications
writing in Python and ease of upgrading...

The commercial distribution needs can be met without going to stable
bytecodes. The ease of upgrading needs aren't met by going to stable
bytecodes.

At current stage, every time a new version of Python is installed in
my server or system, I have to test and ensure the needed libraries
are there... which may be a horror in corporate settings. Couple that
with messy dependencies of the libraries to be installed. I remembered
the time I was trying to install Eric 3, the dependencies makes me
want to give up... I have to install Qt, then pyQt, then something
else, then Eric3. Imagine you need 10 of those libraries... which may
happen... I am not yet masochistic enough to take pleasures in this...

Fixing the byte-code problem won't help with the dependency
problem. If anything, it'll make it worse, because you'll have old
libraries that were installed with previous versions of Python to
contend with. If those are acceptable, that's all well and good. But
for complex packages like eric - which depends on sip and qt and a
number of other things - the latest versions tend to rely on having
up-to-date libraries. So you install eric, and watch it fail because
some library is out of date. Update that library and repeat. The
easiest way to deal with this is to find all the libraries it needs,
and just update them all.

The dependencies problem is actually pretty easy to solve. In fact, in
many Python environments, it's already solved. On my system, if I
install Eric3 using the provided installation package, the
dependencies will be picked up automatically. Not very helpfull if
you're not on such a system, I know. The long-term solution is for
PyPI to grow to include this functionality.

Thank you,
<mike

 

[Maurice LING]

The solution with the current situation depends on what you mean by
"commercial settings".

Commerically, I will mean distributing the program without the codes.
The best way under the current method is to freeze the application. In
Java, you can just distribute the JAR files or CLASS files. I can
imagine there will be some level of touchiness when source codes are
distributed...

People have recommended that you distribute the Python interpreter
with commercial applications. This makes installing them much easier
for the end user. It also means the byte code and the interpreter will
always match.




Java is only "platform-independent" to the degree that the jvm and
libraries you need for your application are already available. A
quick google finds applications that require jvm versions - so I
suspect that you can't always run Java code built for a new version of
the jvm on older jvms. So you have to make sure the target platform
has a recent enough implementation of the jvm installed. You may have
problems if you try using a different groups jvm as well. I haven't
looked into that in a number of years.

So you can't just ship java bytecodes to someone expecting they'll be
able to run it out of the box. They may well need to update their java
environment, or install a second one (I recall one time having four
Java products that took in total three different jvms to run. Bleah.)

This isn't really different from Python. In both cases, the end user
has to have a suitable version of the bytecode interpreter and
libraries installed. Java is a little better in that they provide
backwards compatability.

I think backward compatibility is good enough. A Java JAR file compiled
under for JVM 1.2 can still run in JVM 1.4. I don't think anyone will or
can expect forward compatibility, that will require godly powers...

According to Steve Holden's reply, Python C API is only backward
compatible in minor versions but not major versions. The issue is, why
not major versions as well unless there's something exceedingly prohibitive?

Unstable bytecodes have nothing to do with these problems.

The current CPython installation process puts the python command and
the libraries for different versions in different directories. This
allows you to have multiple versions installed so you can keep old
scripts working with the old version until you've had time to test
them. It also makes life much easier on the developers, as they can
have a development version installed on the machine at the same time
as they have a production version without breaking the old scripts. It
also means you have to reinstall all your modules on the new
installation - which is what makes the update process tedious for me.

Now, this could be mitigated by having Python libraries installed in
the same location for all versions. You could fix all the bytecode
files by running compileall.py as a script. Sometimes, a module won't
work properly on a new version of Python, and will have to be
updated. You'll have to find those by trial and error and fix them as
you find them. You'll also have to recompile all the C modules, which
will break running scripts on the old interpreter.

Under the currewnt system, old scripts that are run by the new
interpreter will break if all the modules they need aren't installed
yet. Once they're installed, the scripts should work.

If you have scripts that use C modules that aren't installed in the
standard Python search path, they may well break on the new
interpreter until you recompile them. Having both versions of the
interpreter available makes this tolerable.

The cure I proposed seems worse than disease. If you've got a better
solution, I'm sure we'd be interested in hearing it.

Perhaps this is another call for Python version of CPAN (CPyAN or PYAN).
It can be modelled after Fink or Darwinports.

If I remembered correctly, Fink uses apt-get and curl.

What can be done in PYAN is to encourage all 3rd party library
developers to centralize their libraries in it, which I think all will
gladly respond. All that is needed to be deposited in PYAN is a
description file, like the simplest setup.py file. All that is needed in
this description file is
1. where to download the source codes (e.g. CVS)?
2. version number?
3. standard stuffs (optional) like authors, descriptions, copyright?

Python can then have a built-in mechanism to read the description file
and download the source codes and do the standard "sudo python setup.py
install" to install the library into site-package. The main thing this
mechamisn does is to maintain what had been installed in site-package
through it and what versions. So when newer versions of Python is
installed, there can be a script to take the old site-package log,
download the same set of libraries and install them for the new version
of Python.

Darwinports uses a very simple means to achieve this. All the
description files are in a CVS repository. The actual source codes for
the libraries may or may not reside in the CVS repository. Darwinports
system installation is little more than checking out the entire
repository of description files. When user wants to install a library,
it then downloads the library's source codes and performs the standard
"configure, make, make install" operations. Of course, the root user
will have to do a update to update to the latest version of description
files.

I think something like this can be set up for Python quite easily. I
recall some time back, there's a long discussion about setting up
Python's version of CPAN but I can't recall the contents of the
discussions.

Fixing the byte-code problem won't help with the dependency
problem. If anything, it'll make it worse, because you'll have old
libraries that were installed with previous versions of Python to
contend with. If those are acceptable, that's all well and good. But
for complex packages like eric - which depends on sip and qt and a
number of other things - the latest versions tend to rely on having
up-to-date libraries. So you install eric, and watch it fail because
some library is out of date. Update that library and repeat. The
easiest way to deal with this is to find all the libraries it needs,
and just update them all.

The dependencies problem is actually pretty easy to solve. In fact, in
many Python environments, it's already solved. On my system, if I
install Eric3 using the provided installation package, the
dependencies will be picked up automatically. Not very helpfull if
you're not on such a system, I know. The long-term solution is for
PyPI to grow to include this functionality.

I must say that I do not quite understand your system. Please enlighten me.

Thanks and cheers
Maurice

 

[Terry Reedy]

Now I understand that Python bytecodes are only dealing with pure python
source codes.

Then stop blaming (machine-independent) CPython 'bytecodes' for any
problems you have with compiled-to-machine-language C extensions, which
have nothing to do with bytecodes. Bytecodes also have nothing directly to
do with source-code compatibility.

 

[Terry Reedy]

I think backward compatibility is good enough. A Java JAR file compiled
under for JVM 1.2 can still run in JVM 1.4. I don't think anyone will or
can expect forward compatibility, that will require godly powers...

One difference between Java and Python is this: Java bytecodes are, as I
understand it, part of the Java language definition. CPython bytecodes are
intentionally not part of the language at all. Except maybe fore PyPy,
other implementations do not use them. Jython translates Python source to
Java bytecodes. Pyrex translates augmented Python source to C, to be
compiled to native machine code. Ironman translates, I presume, to .NET
common language. PyParrot (don't know if it has an official name)
translates to Parrot bytecodes. Viper translated to OCamel.

If you want an implementation with frozen bytecodes, you are free to make
one ;-)

Terry J. Reedy

 

[Maurice LING]

Then stop blaming (machine-independent) CPython 'bytecodes' for any
problems you have with compiled-to-machine-language C extensions, which
have nothing to do with bytecodes. Bytecodes also have nothing directly to
do with source-code compatibility.

technicalities are wrong but situation remains unchanged.

maurice

 

[Maurice LING]

One difference between Java and Python is this: Java bytecodes are, as I
understand it, part of the Java language definition. CPython bytecodes are
intentionally not part of the language at all. Except maybe fore PyPy,
other implementations do not use them. Jython translates Python source to
Java bytecodes. Pyrex translates augmented Python source to C, to be
compiled to native machine code. Ironman translates, I presume, to .NET
common language. PyParrot (don't know if it has an official name)
translates to Parrot bytecodes. Viper translated to OCamel.

If you want an implementation with frozen bytecodes, you are free to make
one ;-)

Terry J. Reedy

So there are currently 7 implementations or variations of the Python
language and you are suggesting I make another one? If I am to do that,
I will use CPython 2.4.1 and call it a implementation that maintains the
current set of bytecodes (newer versions can have more bytecodes and
deprecated bytecodes are still left there for backward compatibility)
and C API interface. Till now I still do not know what is so exceedingly
prohibitive to do that? Why should I create such a direct fork?
Personally I do not have the resources to maintain this fork.

The Pacman (binary executable) that I played on a 286 a decade ago still
works well on my brother's pentium III system, so from the point of a
user, it is backward compatible. The same can't be said for .pyc files
or .so files (I may be wrong with .so files here).

What I do have resources (time and energy) for is to work with the
maintainers of PyPI to implement the package maintenance system I've
described......

maurice

 

[Terry Reedy]

So there are currently 7 implementations or variations of the Python
language and you are suggesting I make another one?

Perhaps you missed the winkie ;-)

The Pacman (binary executable) that I played on a 286 a decade ago still
works well on my brother's pentium III system, so from the point of a
user, it is backward compatible.

How well does it work on a P4 WinXP system? If it does, you're lucky. I
have games that will not run in spite of the compatibility settings being
set. And so I will keep a Win95 machine and a Win98 machine for as long as
they run.

The same can't be said for .pyc files

The same *can* be said for some decade-old .py files. Certainly, most
everything written for 2.0 onward still works. The same will also be true
for .pyc files as long as you run them with their corresponding binary and
as long as the binary stills run.

However, .pyc files are private, temporary caches created by a CPython
interpreter for its own use and tied to its current implementation
technology. They are not intended to be a public distribution form and
indeed cannot be a means to run Python programs on other interpreters.
Guido has stated that he want the freedom to change or even replace parts
of the internal implementation as he sees fit. (2.5 will probably get a new
compiler, with the difference mostly transparent to users.)

One of the principles of OOP is separation of interface from
implementation. User of a class should only use the public interface and
not depend on private implementation. One reason is so implementation can
be changed even radically as long as interface is kept constant. The
Python language is interface. CPython bytecodes are implementation.

What I do have resources (time and energy) for is to work with the
maintainers of PyPI to implement the package maintenance system I've
described......

Good. I agree that we need more along that line.

Terry J. Reedy

 

[Maurice LING]

The same *can* be said for some decade-old .py files. Certainly, most

everything written for 2.0 onward still works. The same will also be true
for .pyc files as long as you run them with their corresponding binary and
as long as the binary stills run.

However, .pyc files are private, temporary caches created by a CPython
interpreter for its own use and tied to its current implementation
technology. They are not intended to be a public distribution form and
indeed cannot be a means to run Python programs on other interpreters.
Guido has stated that he want the freedom to change or even replace parts
of the internal implementation as he sees fit. (2.5 will probably get a new
compiler, with the difference mostly transparent to users.)

One of the principles of OOP is separation of interface from
implementation. User of a class should only use the public interface and
not depend on private implementation. One reason is so implementation can
be changed even radically as long as interface is kept constant. The
Python language is interface. CPython bytecodes are implementation.

From a technical perspective, I can accept that .pyc files are private
and temporary. To a certain extend, it does helps development cycle.
Every time I amend my source codes, I just run it without having to
consider or needing to re-compile the source files.

The idea of having to release the program or library as source files
does ring alarms in many executives in corporate world. Less freezing
the modules (which renders it platform dependent) or using Jython (which
is not possible when C extensions are involved), there is no middle
grounds in CPython for distribution without source codes.

Every now and then, there will be new threads in this list about people
asking the means and possibilities of releasing a module/program without
having to release the source code (there's another new thread on it
today, "Can .py be compiled?")...

Good. I agree that we need more along that line.

I've posted my idea on catelog-sig mailing list. Hope it get picked up...

Cheers
Maurice

 

[Mike Meyer]

Perhaps this is another call for Python version of CPAN (CPyAN or
PYAN). It can be modelled after Fink or Darwinports.

If I remembered correctly, Fink uses apt-get and curl.

What can be done in PYAN is to encourage all 3rd party library
developers to centralize their libraries in it, which I think all will
gladly respond. All that is needed to be deposited in PYAN is a
description file, like the simplest setup.py file. All that is needed
in this description file is
1. where to download the source codes (e.g. CVS)?
2. version number?
3. standard stuffs (optional) like authors, descriptions, copyright?

To copy all of what CPAN does, you need to know what other modules it
depends on, so you can automatically download those as well.

Python can then have a built-in mechanism to read the description file
and download the source codes and do the standard "sudo python
setup.py install" to install the library into site-package.

I don't like this - it would make Python depend on sudo being
available. I'd rather it not do that, and let each systems
administrator issue the command according to *their* security policy.

I think something like this can be set up for Python quite easily. I
recall some time back, there's a long discussion about setting up
Python's version of CPAN but I can't recall the contents of the
discussions.

It's not clear it's easy. It's *very* clear it won't happen until
someone steps up to do it. The latter is a problem.

I must say that I do not quite understand your system. Please enlighten me.

FreeBSD has a packaging system (like fink on the Mac) called either
"Ports" (if you build from source) or "Packages" (if you install
pre-compiled binaries). I use Ports, so I'm going to talk about
them. But Packages are built from Ports, and all the information in
the port winds up in the package, so you get the same functionality
with a slightly different command set.

Basically, each port has a list of all the other ports it depends
on. When you install a port, the system will check to see if each of
the dependent ports is installed - usually by checking if a file it
should install exists - and if it doesn't, will install the dependent
port. The dependencies of that port are recursively checked. So if I
were to install the Eric3 port, it would note that I need the
qscintilla, py-sip and py-qt (ports of python modules are all named
"py-<something>"). py-sip and py-qt are already installed, so it would
skip thos. qscintilla isn't, so that would get installed. qscintilla
doesn't have any dependencies, so that would stop there. After
qscintilla is installed, eric3 would be installed.

All of this is recorded in the Packages database, and I can query that
to find out what python modules are installed, which module installed
a specific file, etc. I can also use that to cleanly remove the
module, which the python setup.py doesn't do for me.

I'm not familiar with fink, but I suspect it offers much the same
functionality. rpm's and apt-get offer that functionality for many
Linux systems. Solaris has it's own package system as well.

Basically, for common Unix platforms, the dependency problems you ran
into with Eric3 should be solved by the platforms package system,
assuming that Eric3 is available as a package.

<mike

 

[Mike Meyer]

The idea of having to release the program or library as source files
does ring alarms in many executives in corporate world. Less freezing
the modules (which renders it platform dependent) or using Jython
(which is not possible when C extensions are involved), there is no
middle grounds in CPython for distribution without source codes.

You're right - and byte codes *as they exist now* aren't an acceptable
middle ground, either. The problem isn't that the interpreter might
change (that can be defeated by distributing the interpreter with the
bytecode files), but that the byte code can be decompiled to get your
Python source back. See <URL:
http://www.crazy-compilers.com/decompyle/ > for an example.

Selling byte codes to someone who's worried about shipping source is
selling them snake oil. I think it's unprofessional, at best.

<mike

 

[Maurice LING]

What can be done in PYAN is to encourage all 3rd party library

To copy all of what CPAN does, you need to know what other modules it
depends on, so you can automatically download those as well.

Yes, dependencies are necessary to be declared. The maintainers of each
package should list the dependencies as well to make it work.

I don't like this - it would make Python depend on sudo being
available. I'd rather it not do that, and let each systems
administrator issue the command according to *their* security policy.

If you are installing packages into your home directory, then sudo is
not needed. But if you are installing it for everybody's use, then it is
necessary. Fink runs using superuser privileges.

It's not clear it's easy. It's *very* clear it won't happen until
someone steps up to do it. The latter is a problem.

I've posted my method in catelog-sig list. And is willing to offer some
time on it... if that helps... But to get it working, I think it will
need 5-10x more time than I can put into it...

Cheers
maurice