Python Gotcha's?

[Bryan]

If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.

Python 3(K) likes to use the same '.py' file extension as its
incompatible predecessors, and in some/many/most *nix implementations,
it likes to install in the same place. Python 3 is an improvement upon
Python 2, but Python went from, "sure... Python just works," to,
"well... that depends... which Python?"

I missed the 1 to 2 transition. I'm not exactly a Python newbie, but
Python 1.5.2 was dead an buried by the time I met the snake^H^H^H^H^H
group of daffy English k-ni-ghits.

We knew that there was no painless path from where we were to where we
saw to be better, and the Python community and our BDFL made many wise
decisions toward the Python 3 transition. This particular matter, we
botched. I am too late in speaking up, so I'm as much blame as anyone.
Something to keep in mind for Python 4.

-Bryan

 

[Steven D'Aprano]

Python 3(K) likes to use the same '.py' file extension as its
incompatible predecessors,

And so it should.

Python 2 and Python 3 are two dialects of the same language, with just a
very few minor changes in syntax (although a fair number of changes in
libraries). With a modicum of care, it is quite possible, even trivially
easy, to write useful code that is compatible with both Python 2 and 3.
Here's an example:

http://pypi.python.org/pypi/pyprimes

and in some/many/most *nix implementations,
it likes to install in the same place.

I won't speak for Unixes, but that is certainly not the case with Linux.
Each Python version gets its own location:

[steve@ando ~]$ for vers in 2.4 2.5 2.6 3.2 ; do which python$vers ; done
/usr/bin/python2.4
/usr/local/bin/python2.5
/usr/local/bin/python2.6
/usr/local/bin/python3.2


The first, 2.4, is the system python; the others were installed by me. I
didn't need to take any special effort to install to versioned locations,
the make script does that by default. The only special effort needed was
to run "make altinstall" instead of "make install" to ensure that
"python" without a version number still points to the system python
rather than the freshly installed version.

Python 3 is an improvement upon
Python 2, but Python went from, "sure... Python just works," to,
"well... that depends... which Python?"

I think that's nonsense. There has never been a time that you didn't have
to think about "which Python", except maybe the very first public
release. Every version has had differences from previous versions -- they
wouldn't be different versions otherwise. Bugs are fixed, but more
importantly new features are added. Libraries are deprecated and then
removed. If you write code using Python 2.4 features, it may not work
with Python 2.3. If you write code using 2.5 features like ternary if
expressions, it certainly will not work in Python 2.4.

The 2to3 transition isn't the only time that Python has made backwards
incompatible changes. I recently had to upgrade some Python code using
string exceptions from Python 2.3 to 2.6. That was tedious but easy.
However getting the tkinter code to upgrade was exciting and difficult.

Another example: the semantics of hex() and oct() have changed, probably
sometime around 2.4 or 2.5. A third: the introduction of nested scopes
back in 2.1. This changed the behaviour of any code using nested
functions, and was the inspiration for the introduction of __future__. A
fourth: turning None into a keyword in version 2.4.

I don't intent to be rude, but anyone who isn't a complete newbie to
programming but is surprised to the point of "gotcha" by version
compatibilities simply hasn't been paying attention.

 

[Bryan]

And so it should.

We disagree. Not surprising in a "gotcha's" thread.

I won't speak for Unixes, but that is certainly not the case with Linux.
Each Python version gets its own location:

Yes, that was just silly of me to write that. All I want is a new
general convention for the most-likely-to-work invocation that won't
break with the change: "#!/usr/bin/env python" for Python 2 versus,
for example, "#!/usr/bin/env python3". Of course that's not an issue
of where python is installed, just a recommended naming convention.

I don't intent to be rude, but anyone who isn't a complete newbie to
programming but is surprised to the point of "gotcha" by version
compatibilities simply hasn't been paying attention.

My perspective is simply different from yours. I'm not the one who
installs python on most of the boxes where I work or play. There's
little consistency, so I love conventions that usually work. I'd like
to advocate for Python 3, but the default install on Windows
commandeers the '.py' extension and breaks stuff that currently works.

Here's a discussion of the issue from late 2008. Amusingly, one of the
arguments for not changing the file extension was that Python 2 would
be gone in a few years.
http://www.velocityreviews.com/forums/t647251-running-python-2-and-python-3-on-the-same-machine.html

-Bryan

 

[Steven D'Aprano]

We disagree. Not surprising in a "gotcha's" thread.

Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file extension
for Python 3 is a good idea, you are keeping it to yourself.

Python and C are different languages. Python 2 and Python 3 are not, they
are the same language with only a few minor dialect differences.

There is a practical argument against separate file extensions: which
extension do you use for code intended to run with both Python 2 and 3?

We didn't need a new file extension for the transition between Python 2.5
(string exceptions are legal) and Python 2.6 (string exceptions cause a
SyntaxError exception). Nor did we need a new file extension for the
transition between Python 2.1 (nested functions behaved one way) and
Python 2.2 (nested functions behaved a different way). We certainly
didn't have a new file extension when the bastion or gopher modules were
removed from the standard library, backwards-incompatible changes if I've
ever seen one.

If there's a good argument in favour of separate file extensions for
Python 2 and Python 3 (one which doesn't also apply to, say, Python 2.5
and 2.6, or 3.1 and 3.2) I'm afraid I don't know it.

 

[Curt]

Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file extension
for Python 3 is a good idea, you are keeping it to yourself.

Didn't you trim his reasons, speaking of trimming (maybe they were
nonsensical, or poor, but it seems they were there before you made them
disappear)?

 

[Steven D'Aprano]

Didn't you trim his reasons, speaking of trimming (maybe they were
nonsensical, or poor, but it seems they were there before you made them
disappear)?

No, you remember wrongly. Bryan states that the use of the same file
extension is a gotcha, but that's it.

Python 3(K) likes to use the same '.py' file extension as
its incompatible predecessors

http://mail.python.org/pipermail/python-list/2012-April/1290909.html


Python 3 uses the same file extension as its incompatible predecessors
for the same reason that Python 2.6 uses the same file extension as its
incompatible predecessors, and 2.5 as as its incompatible predecessors,
and so on all the way back.

"Incompatible" is not a binary state, it is a matter of degree. Python
3.1 is less compatible to Python 2.7 than 2.7 is to 2.6, but the vast
bulk of the language is still identical and code supporting everything
from 2.5 to 3.2 in one code base is possible.

In my experience, it is MUCH easier to write code targeting versions 2.5
through 3.2 than it is to target 2.4 and 2.5 only, on account of how
impoverished 2.4 is compared to 2.5. (I once started a project to
backport useful 2.5 features to 2.4. I gave up because it was just too
painful.)

 

[Bryan]

Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file extension
for Python 3 is a good idea, you are keeping it to yourself.

On Windows the file extension determines what executable opens the
file. Running both Python 2 and Python 3 on Windows is painful where
it doesn't need to be. I'd like to encourage my users to check out
Python 3, but installing it on Windows will take over the '.py'
extension and break stuff that currently works.

Incidentally, I'm not actually advocating for '.py3'. I'm advocating
for '.py4'.

Python and C are different languages. Python 2 and Python 3 are not, they
are the same language with only a few minor dialect differences.

You could think of it as the same file extension in a different
dialect, but really the works/broken distinction is more important
than language/dialect.

There is a practical argument against separate file extensions: which
extension do you use for code intended to run with both Python 2 and 3?

The file extension default should work with the the recommended
transition method, which is not dual-major-version code. Admittedly,
support for such code has improved.

We didn't need a new file extension for the transition between Python 2.5
(string exceptions are legal) and Python 2.6 (string exceptions cause a
SyntaxError exception). Nor did we need a new file extension for the
transition between Python 2.1 (nested functions behaved one way) and
Python 2.2 (nested functions behaved a different way). We certainly
didn't have a new file extension when the bastion or gopher modules were
removed from the standard library, backwards-incompatible changes if I've
ever seen one.

Python's management of backwards compatibility for minor version has
worked pretty well, at least for me. Major version simply do not
attempt backward compatibility.

Your experience seems to be quite different from mine. I don't recall
a minor version upgrade ever giving me significant trouble with my own
code. The issue has been the external libraries upon which I depend,
and they've tended to catch up quickly, unlike what we're seeing with
Python 3.

If there's a good argument in favour of separate file extensions for
Python 2 and Python 3 (one which doesn't also apply to, say, Python 2.5
and 2.6, or 3.1 and 3.2) I'm afraid I don't know it.

Because it would allow Windows user to play with Python 3 alongside
Python 2, while waiting for external libraries to catch up. Of course
they can, as I am, but the gotchas are really annoying. With minor
versions its not a big deal if most users simply wait to do an
upgrade.

-Bryan

 

[Steven D'Aprano]

On Windows the file extension determines what executable opens the file.
Running both Python 2 and Python 3 on Windows is painful where it
doesn't need to be.

And how is that different from any other two versions of Python?

Unless both versions include the same libraries (either standard modules
or third party), and your code is written to use only the lowest common
denominator functionality, you will have problems regardless of which
versions they are.

It is true that simple Python code using no external libraries has a
better chance of working without modification across multiple 2.x
versions than across the 2-3 transition, but that's not an absolute.
Going from 2.5 to 2.6 can cause breakage; on the other hand with care you
can support 2.6-3.2. With some extra effort, you can support 2.5 or even
older too.

(Sufficiently trivial, or careful, code can probably support any version
of Python from 1.4 to 3.2 inclusive. But it wouldn't be pleasant.)

The situation is worse on Windows, as Windows doesn't support hash-bang
syntax. But that is being looked at now:

http://www.python.org/dev/peps/pep-0397/

which should make supporting multiple Python versions much more pleasant
Real Soon Now.

 

[alex23]

The situation is worse on Windows, as Windows doesn't support hash-bang
syntax. But that is being looked at now:

http://www.python.org/dev/peps/pep-0397/

which should make supporting multiple Python versions much more pleasant
Real Soon Now.

I keep hoping it will, but it's still in draft & I haven't seen much
motion lately

 

[Paul Rubin]

And how is that different from any other two versions of Python?

1. The backwards incompatibility between 2 and 3 is much more serious
than between 2.x and 2.(x-1).

2. There is not much reason to run multiple 2.x's on the same system.

Going from 2.5 to 2.6 can cause breakage;

That should be considered a bug in 2.6, but in any case it doesn't come
into play that often. It's nothing like the 3.x changes.

on the other hand with care you can support 2.6-3.2.

The idea of the incompatibilities introduced to 3.x is that people
should USE them, since they supposedly make the language better.
Therefore "with care" goes against their purpose. Also, "with care"
only applies to script you write yourself. If you download a script
from somewhere else that relies on 3.x, you have to modify it to work
under 2.x and vice versa. If you download both kinds of scripts you
have to modify some of them or else run two Pythons.

 

[Bryan]

And how is that different from any other two versions of Python?

Python 3.0, also known as “Python 3000” or “Py3K”, is the first ever
*intentionally backwards incompatible* Python release. --GVR

Unless both versions include the same libraries (either standard modules
or third party), and your code is written to use only the lowest common
denominator functionality, you will have problems regardless of which
versions they are.

*My* code is not what's holding me back. I *like* moving my code to
Python 3. What's holding me back is library availability, and from
various readings on the transition to 3, I gather that's what holding
many, perhaps most, users back.

The need to keep using 2.x should not stop users from installing and
starting to muck with 3.0. That's much more painful than it has to be
because they both use the same file extension.

 

[rusi]

Python 3.0, also known as “Python 3000” or “Py3K”, is the first ever
*intentionally backwards incompatible* Python release. --GVR


*My* code is not what's holding me back. I *like* moving my code to
Python 3. What's holding me back is library availability, and from
various readings on the transition to 3, I gather that's what holding
many, perhaps most, users back.

The need to keep using 2.x should not stop users from installing and
starting to muck with 3.0. That's much more painful than it has to be
because they both use the same file extension.

There are two kinds of languages
1. The supercalifragilisticexpialidocious language that exists
entirely on my own computer (and perhaps a small band of devotees)
(for some value of 'my'). IOW a 'research' language
2. Those that just exist like the sun, moon and taxes (Fortran, Cobol
and C come to mind)
If the ISO-C committee decided that garbage collection should exist in
C, they can legislate it with comparable ease to legislating a change
in the sun's gravitation.

The problem with python is that it sits in between. It is still
alive, growing, learning etc and yet the installed base is so big that
incompatibilities cause real trouble.

Speaking more quantitatively: we can work out the cost/benefit ratio.

Assume that all the changes in py3k are 'good', they are benefits -- b
Costs can be single-cased (s) -- basically those that can be handled
by a 2to3 module
Those that are not so handle-able are the sorry case (multicase m)

And so the cost/benefit ratio is (s + n*m)/b (n is the no of python
users/uses 'out there')
IOW the irony: the success of python (large n) implies the failure of
py3k.

Of course not so ironic if one considers that Fortran, Cobol, C cant
be changed precisely because they are so successful

 

[Paul Rubin]

Costs can be single-cased (s) -- basically those that can be handled
by a 2to3 module

You can't really 2to3 a large python application and expect to then just
start using it without further attention or testing. You may have to do
a fairly complete (i.e. expensive) QA and qualification cycle on the
2to3 output, depending on the application and your environment. It's
easier to just keep running python 2 for older programs even if you're
using python 3 for newer ones.

IOW the irony: the success of python (large n) implies the failure of
py3k... Of course not so ironic if one considers that Fortran, Cobol,
C cant be changed precisely because they are so successful

But those languages do change. We just got a new C11 standard, for
example. And the more radical change to C was of course C++, which
went off in its own direction.

IMHO the main weakness of py3 is that its benefits over py2 are rather
slight. I've been involved in several newly started python projects in
the past few years and they use py2 because what's the point of dealing
with brokenness if you're used to something that works and where the
gain from switching is insignificant? Py3 would enjoy a lot more
success in my opinion, if it broke py2 much more drastically than it
does, but delivered larger gains. I guess maybe that can happen with
py4.

 

[Albert van der Horst]

Except for the most important reason of all: Python's use of alternate
string delimiters is an excellent design, one which Javascript itself
follows.

http://www.javascripter.net/faq/quotesin.htm

I'm not the only one who has had trouble with JSON's poor design choice:

http://stackoverflow.com/a/4612914

For a 21st century programming language or data format to accept only one
type of quotation mark as string delimiter is rather like having a 21st
century automobile with a hand crank to start the engine instead of an
ignition. Even if there's a good reason for it (which I doubt), it's
still surprising.



Ooooh, such complication. I wish my software was that complicated.

The added value includes:

* semantic simplicity -- a string is a string, regardless of which
quotes are used for delimiters;

* reducing the number of escaped quotes needed;

* compatibility with Javascript;

* robustness.

As it stands, JSON fails to live up to the Robustness principle and
Postel's law:

Be liberal in what you accept, and conservative in what you send.


http://en.wikipedia.org/wiki/Robustness_principle


So you say, but it is a fact that people do. And even if they don't hand-
write it, people do *read* it, and allowing both quotation marks aids
readability:

"\"Help me Obiwan,\" she said, \"You're my only hope!\""

Blah. You can cut the number of escapes needed to one:

'"Help me Obiwan," she said, "You\'re my only hope!"'

I still think the doubling convention of Algol68 is superior:
"""Help me Obiwan,"" she said, ""You're my only hope!"""

No special treatment of any other symbol than the quote itself.
A quoting symbol is such a devious syntactic element that I rather
not have two ( " ' ) or even three ( " ' \ )

Groetjes Albert