Multiple versions of Python coexisting in the same OS

[News123]

Well, but a script converted with py2exe can't really ever assume that
there is a Python interpreter, at all.

true


However, why I thought about this is, that
I write sometimes python code, which tries to call other python files.

later on for distribution I use py2exe.

Therefore I use wrapper functions, which will work in either case.

The wrapper could use sys.executable in 'python mode'
and had to call the exe file in 'py2exe mode'

 

[MRAB]

How does a 'pystarter' program know where the file's location is which
is being invoked ?

the file's location would be somewhere in sys.argv
probably in sys.argv[1].
converting it to an abs path would return a directory which the python
file belongs to.

As to the first file line this is completely
unrealistic. What are you going to do, alter the first file line of
every script in a Python distribution and every script in every library
installed in a Python distribution ? Sorry, but a less intrusive
solution is much better and much less of a headache to say the least.

Well I would at least do it for all of my self written scripts.

It could allow a soft transition from 2.6 to 2.7 to 3.0 without having
to upgrade all scripts at the same time.

My intended solution would be a simple program which understands where
each co-existing Python distribution is installed on a system and what
the "name" of that distribution is. Then you tell the program which
Python distribution should be the current one by its "name", the current
one meaning the distribution which you want to be invoked at any given
time. The program then changes the PATH so that any references to the
Python directory and its subdirectories point to the "name" Python
directory tree, and changes the file associations so that the "name"
Python executables handle the Python associations.

This does have the weakness that I can not use more than one Python
distribution while Python is executing scripts. But I can personally
live with that since I have never encountered a situation where I must
use more than one Python distribution at the same time.

I guess it's rather difficult to find a solution which suits all.

The above minor weakness, that you mention would be a killer for me.

Currently I'm looking for solutions, where I can start python scripts
requireing different python versions at the same time.

Currently I'm staring the scripts manually from two different cmd line
windows with a different path name and an explicit python call,

Thus my idea of having a pystarter with a config file
mentioning which directories (tools) should use which python executable

I think that's the wrong way round. A pystarter should ask the _tool_
which version of Python it needs.

 

[Edward Diener]

I doubt many scripts do it. The fact of the matter is: many systems have
multiple Python versions installed in parallel, and it probably will
break somewhere, which will get noticed, and probably fixed.

If a script uses sys.executable instead of "python", there is no
problem, at all.

What a script uses to internally invoke Python I can not control. My
solution seeks to be non-intrusive and lets me run a particular version
of Python, among the co-existing versions installed, at any given time.
I believe that is the best I can do. I neither can control, nor do I
want to control, all of the Python scripts installed on my system, nor
can I worry how they may internally invoke Python. But I do want to be
able to say, at any given time, that when I run Python a particular
version, amidst the co-existing ones on my system, needs to be executed
and therafter all internally executed modules use that version.

Trying to make rules for scripts, such as telling scripts they must use
sys.executable, is pursuing an imaginary solution that can not work
unless one is theoretically willing to manually inspect and change all
Python scripts in some way. To me any intrusive changes to actual
scripts is no solution at all.

 

[Edward Diener]

Am 25.07.2010 21:32, schrieb Thomas Jollans:

It's true that sys.executable is the best way if you have to start a new
Python interpreter. However sys.executable may not be set for NT
services. So there may be a problem after all.

Once you start instrusively changing scripts to find a solution to
multiple versions of Python co-existing in one system, you are heading
down a path of endless problems.

 

[Edward Diener]

How does a 'pystarter' program know where the file's location is which
is being invoked ?

the file's location would be somewhere in sys.argv
probably in sys.argv[1].
converting it to an abs path would return a directory which the python
file belongs to.

As to the first file line this is completely
unrealistic. What are you going to do, alter the first file line of
every script in a Python distribution and every script in every library
installed in a Python distribution ? Sorry, but a less intrusive
solution is much better and much less of a headache to say the least.

Well I would at least do it for all of my self written scripts.

It could allow a soft transition from 2.6 to 2.7 to 3.0 without having
to upgrade all scripts at the same time.

Intrusively changing scripts is a path to Python hell.

I guess it's rather difficult to find a solution which suits all.

The above minor weakness, that you mention would be a killer for me.

Currently I'm looking for solutions, where I can start python scripts
requireing different python versions at the same time.

If you need that, then of course my intended solution would not work.

Currently I'm staring the scripts manually from two different cmd line
windows with a different path name and an explicit python call,

If you start scripts and point to a specific version of Python, this
works in my solution also. But if an internal call to Python exists
thwre is always a problem.

Thus my idea of having a pystarter with a config file
mentioning which directories (tools) should use which python executable

Well, good luck ! I don;t know how this is resolved for you when some
scripts executes 'python xxx yyy' or 'someScript.py yyy'.

 

[Edward Diener]

true


However, why I thought about this is, that
I write sometimes python code, which tries to call other python files.

later on for distribution I use py2exe.

Therefore I use wrapper functions, which will work in either case.

The wrapper could use sys.executable in 'python mode'
and had to call the exe file in 'py2exe mode'

You can control what you do but how are you going to control what any
givemn script does ?

Attempting to intrusively change potentially every script in a
distribution in any way is a path to Python hell IMO.

 

[Edward Diener]

What exactly is it that you're afraid to change?

I do not want to intrusively change any script that has been installed
as part of Python. I shouldn't even have to know about the code in these
scripts other than what good documentation tells me in how to use them.

That's not to say having source is worthless. I am just not going to
change source to get a version of Python to work properly when I have 2
or more versions installed in their own separate folders.

The standard library? There's certainly no need to change that in any way!

So if a standard library module ( or distributed library ) executes a
call internally to 'python xxx yyy' or executes a call internally to
'someScript.py yyy', you're fine with multiple co-existing versions of
Python on your system ?

Because under Windows the first call will look for the python.exe first
found in the PATH while the second call will find the python.exe
associated with the .py extension. And it does not matter in either case
what version of the multiple installed versions of Python which are on
my system is currently executing that script.

And please don't say that there is some sort of guarantee that no
library or installation would invoke Python in such a way as opposed to
the normal 'import AScript.py' method of using functionality in Python
scripts.

 

[Steven D'Aprano]

Whether it is badly written or not in your opinion it is legal and
happens all the time.

Yes, people write poorly written, buggy scripts all the time. Just
because code is legal syntax doesn't mean it does what is intended, or
that what is intended is sensible.

If you have multiple versions of Python installed, and you call "python
somescript.py" without knowing *which* Python will be called, it is
neither sensible nor does it do what you intend. End of story.

This is no different from calling any other application without knowing
what version you will get, then relying on features that are only
available in some versions. It is just buggy code.

Are you going to refuse to use any script, no
matter for what library or for what purpose, that internally invokes
Python either through a 'python' command or through a file with a Python
extension ? And how would you find out if a script did this or not ? Are
going to search every script in every distribution and library to
determine if it does this ? And when you find out a script does this,
what will you do ?

Treat it like any script with a bug: fix the bug, stop using the script,
or determine a work-around that masks the bug. All three are acceptable,
the third being the least acceptable because it just leaves a bug waiting
to bite you again in the future.

Be real. saying you do not like scripts that internally invoke Python
does not solve anything if you have multiple coexisting versions of
Python installed.

No, it solves it completely. Treat it as a bug, and fix it.

If you're not willing to treat it as a bug, then uninstall all but one of
the Python versions, and the problem goes away. You might have a
different problem, namely that some scripts stop working, but now the
solution is obvious and straight-forward: fix the scripts that aren't
working.

Or rename the Python applications, so that scripts can easily call the
right version without getting confused.

Trying to make some brittle, Do-What-I-Mean solution for trying to auto-
magically select between Python versions is pursuing a path of endless
problems. Any solution that doesn't fix the actual problem, namely that
the scripts are buggy, is at best just a work-around and at worst is no
solution at all.

 

[David Robinow]

On 7/25/2010 5:57 PM, Thomas Jollans wrote:
So if a standard library module ( or distributed library ) executes a call
internally to 'python xxx yyy' or executes a call internally to
'someScript.py yyy', you're fine with multiple co-existing versions of
Python on your system ?

Because under Windows the first call will look for the python.exe first
found in the PATH while the second call will find the python.exe associated
with the .py extension. And it does not matter in either case what version
of the multiple installed versions of Python which are on my system is
currently executing that script.

And please don't say that there is some sort of guarantee that no library or
installation would invoke Python in such a way as opposed to the normal
'import AScript.py' method of using functionality in Python scripts.

Edward, I'm having a really hard time understanding your problem.
Could you give an example of some real code that is causing you
difficulty?

 

[Edward Diener]

Yes, people write poorly written, buggy scripts all the time. Just
because code is legal syntax doesn't mean it does what is intended, or
that what is intended is sensible.

If you have multiple versions of Python installed, and you call "python
somescript.py" without knowing *which* Python will be called, it is
neither sensible nor does it do what you intend. End of story.

Somebody is supplying you with a Python script and internally invoking
Python again. But that somebody does not have to be myself.

I am neither buying "End of story" nor that invoking Python internally
is an error. But if you believe it to be then you can root out all such
Python code, or correct it as you like. Even with co-existing versions
of Python installed I have better things to do with my time and
therefore will pursue a solution that will work for me in the face of
such code.

 

[Dennis Lee Bieber]

So if a standard library module ( or distributed library ) executes a
call internally to 'python xxx yyy' or executes a call internally to
'someScript.py yyy', you're fine with multiple co-existing versions of
Python on your system ?

I highly doubt any "standard" library will rely upon the ability to
spawn independent processes that are subject to the vagaries of the
user's system configuration.

Likewise, any 3rd party library that performs process spawning had
better document, right up front (before I even download it) that it
performs such behavior.

To me, a "library", by definition is meant to work within the
process space that invokes it. That process space may not be compatible
with the installed Python -- but that should mean the library itself
will not function/install; not that it will appear to function and then
fail due to some external process not being compatible.

 

[Edward Diener]

Edward, I'm having a really hard time understanding your problem.
Could you give an example of some real code that is causing you
difficulty?

I start a Python script for version X by going to X's root directory and
invoking 'python someScript.py' from the command line. Does that not
sound reasonable ?

In SomeScript.py there is an internal call to 'python someOtherScript.y
someParameters'. But the call invokes another version of Python because
it is that version which is in the PATH. Or in SomeScript.py there is an
internal call to 'someOtherScript.py someParameters'. But the call
invokes another version of Python because the .py extension is
associated with a different version.

My solution is that I will write some code which sets a particular
version of Python as the current version for a particular time, meaning
that version will be in the PATH and associated with Python extensions.
The way I do not have to worry when I externally invoke Python from the
command line that internal calls are going to some other version.

 

[Steven D'Aprano]

On 7/25/2010 10:42 PM, David Robinow wrote: [...]

Edward, I'm having a really hard time understanding your problem. Could
you give an example of some real code that is causing you difficulty?

I start a Python script for version X by going to X's root directory and
invoking 'python someScript.py' from the command line. Does that not
sound reasonable ?

No it doesn't, it's a very unreasonable thing to do.

If you have multiple versions of Python, you should name them
appropriately so you can launch the appropriate version from any
directory:

python26 someScript.py # calls python31 secondScript.py internally
python31 anotherScript.py # calls python25 thirdScript.py internally

etc.

Or give the full path to the executable:

C:\Programs\python26\python.exe someScript.py
# calls C:\Programs\python31\python.exe secondScript.py internally


If a script doesn't do this, then the script should be treated as buggy.

In SomeScript.py there is an internal call to 'python someOtherScript.y
someParameters'.

That's a pretty dodgy thing to do in the first place, unless you can
guarantee that there's only one executable called python. Otherwise, how
do you know which one will be called? You can't easily predict which one
will be called, so don't do it unless you want your scripts to call
arbitrary executables.

But the call invokes another version of Python because
it is that version which is in the PATH. Or in SomeScript.py there is an
internal call to 'someOtherScript.py someParameters'. But the call
invokes another version of Python because the .py extension is
associated with a different version.

Exactly. The root of your problem is that there are multiple executables
called "python" and you don't know which one you will get. So don't do
that.

My solution is that I will write some code which sets a particular
version of Python as the current version for a particular time, meaning
that version will be in the PATH and associated with Python extensions.
/facepalm


The way I do not have to worry when I externally invoke Python from the
command line that internal calls are going to some other version.

Good luck with that.

 

[Gelonida]

Well, good luck ! I don;t know how this is resolved for you when some
scripts executes 'python xxx yyy' or 'someScript.py yyy'.

both could be resolved with a python starter if one wanted.

call the python starter python.exe and put it first in the path.
set the python file associations to the python starter.


By the way:

Something similiar (not identical) has been used for linux hosts in big
companies in order to easily switch between different projects. (which
potentilly had different versions of development tools)

instead of having multiple .cshrc / .bashrc files

the first entry of path has been set to
/_WRAPPER_/bin (or something similiar)
in this directory one found a wrapper script for each tool to be
wrapped. ( one script, many symlinks for each tool)

the wrapper script took a global setup, the project name and a private
setup to finally call the the desired script.

What is missing is to choose the executable depending on the script to
be called.

If one has a wrapper this could be added.

Just a question of style, whether the decision which script to be called
should come from the
- user
- a config file ( No script to be intrusively changed, what Edward seems
to prefer )
- some coding in the script or the tool's directory ( which MRAB seems
to prefer )

 

[Gelonida]

News123 wrote:
I think that's the wrong way round. A pystarter should ask the _tool_
which version of Python it needs.

Hm, it's dfifficult to think about a solution satisfying everyone.

Edward seems to insist on a non intrusive solution, where no script is
changed (thus my idea of a config file)


I personally would probably add information to all of my self written
scripts
and create a config file for all other tools (or add a marker file n the
path of these tools)

 

[Gelonida]

I start a Python script for version X by going to X's root directory and
invoking 'python someScript.py' from the command line. Does that not
sound reasonable ?

Do you have an example of two (not self written) applications requiring
to change the python file association in order to be working?

I never had problems with this, so would be curious about the tools to
avoid.



Apart from that my suggestion for you would be:


Don't write a tool.
Just create one .bat file for each sript to be started.

The bat file should set the python search path.
This should cover 90% of existing python scripts.
(If you want, you could write a tool to create the bat files)


Now the second problem.

if a python script starts another python file without calling python.exe
but via the file associations, then above solution would not be sufficient/
you had to additionally change the file associations,
(which can easily be done from a bat file as well if you insist. The
commands you need are 'assoc' and 'ftype')

But does this really happen to you?



Please note:
Trying to develop a solution, which believes that you will never have
two concurrent applications requiring two differnt python versions
sounds a little dangerous.

 

[Thomas Jollans]

I start a Python script for version X by going to X's root directory and
invoking 'python someScript.py' from the command line. Does that not
sound reasonable ?

yeah, well, sort of. But for a system installation, not really. When
hacking on the interpreter, this makes sense. Otherwise - not so much.

In SomeScript.py there is an internal call to 'python someOtherScript.y
someParameters'.

Is that a fact?

Where on earth did you get this "SomeScript.py"?