Thoughts on some stdlib modules

[Terry Hancock]

I'm not talking about things that absolutely have to be in the Python
interpreter core; I'm talking about things that *could* be bundled
with the standard distribution, *without* having to be relicensed,
or be forever maintained by the CPython developers.

(the Linux distributors know how to do this: look for good stuff that's
either actively maintained or simple and solid enough to live for a while,
make sure the licenses are good enough, bundle the latest and greatest
version, ship tested versions at regular intervals, update when necessary,
and pass bugs and patches upstream. why not use the same approach
for Python's standard distribution?)

That sounds like a suggestion that there should be a "greater python"
distribution --- python itself, the standard libraries (now released together
as the "python distribution"), and the most commonly used Python
packages (which presumeably would include the current free PIL, Numeric (or
Numarray more likely), and so on.

One might argue that Linux distributions are already doing this -- Debian
does something like it, for example. But there would be a greater cross-
platform consistency if such a super-distribution existed. I can imagine
it would be particularly valuable to Windows users who can't benefit from
the Linux packaging operations.

I guess the other thing to compare to is something like SciPy, which is
a kind of specialized distribution of Python for scientific applications.

Terry

 

[Robert Kern]

That sounds like a suggestion that there should be a "greater python"
distribution --- python itself, the standard libraries (now released together
as the "python distribution"), and the most commonly used Python
packages (which presumeably would include the current free PIL, Numeric (or
Numarray more likely), and so on.

One might argue that Linux distributions are already doing this -- Debian
does something like it, for example. But there would be a greater cross-
platform consistency if such a super-distribution existed. I can imagine
it would be particularly valuable to Windows users who can't benefit from
the Linux packaging operations.

I guess the other thing to compare to is something like SciPy, which is
a kind of specialized distribution of Python for scientific applications.

No, Scipy is just a (large) package. Enthon, on the other hand, is just
such a distribution.

Windows:
http://www.enthought.com/downloads/downloads.htm
(go to the bottom for the latest and greatest)

Mac:
http://download.enthought.com/MacEnthon/ReadMe.html
http://download.enthought.com/MacEnthon/MacEnthon-0.1.dmg

Linux:
TBA

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Robert Kern]

I'm not talking about things that absolutely have to be in the Python
interpreter core; I'm talking about things that *could* be bundled
with the standard distribution, *without* having to be relicensed,
or be forever maintained by the CPython developers.

(the Linux distributors know how to do this: look for good stuff that's
either actively maintained or simple and solid enough to live for a while,
make sure the licenses are good enough, bundle the latest and greatest
version, ship tested versions at regular intervals, update when necessary,
and pass bugs and patches upstream. why not use the same approach
for Python's standard distribution?)

The reason they don't is because making a distribution like this is a
royal pain in the ass. It makes no sense to make the Python standard
distribution like this. Only insane people do this. I'm one of them.

"Updating when necessary" is problematic and really requires proper
package management. Python is not an OS and cannot control all of the
little factors that make package management feasible for OS
distributions to do.

What's more, once you *do* have proper package management on the
platforms that you care about, it doesn't matter what is or isn't in the
standard distribution. Hopefully, one of the CPyAN dreams will actually
pan out.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Fredrik Lundh]

The reason they don't is because making a distribution like this is a royal pain in the ass. It
makes no sense to make the Python standard distribution like this. Only insane people do this. I'm
one of them.

with enough volunteers, and a distributed development environment, nobody
needs to be insane. if you try to channel everything through a small number of
CPython developers, everyone will loose their sanity, and the users will suffer.

"Updating when necessary" is problematic and really requires proper package management. Python is
not an OS and cannot control all of the little factors that make package management feasible for
OS distributions to do.

"updating when necessary" means "adding new components to the standard library
when there are new stable releases available, and someone has time to run the test
suite and contact the upstream provider if necessary".

What's more, once you *do* have proper package management on the platforms that you care about, it
doesn't matter what is or isn't in the standard distribution.

which means that at the moment, the best way to get a Python module into the
hands of users is to 1) provide windows installers that are compatible with the
"official" distribution, and 2) make it easy for downstream providers to package
your stuff, and leave it to them to take care of the rest.

but if the library in the standard distribution doesn't really matter, it should be
stripped down to minimum, to avoid overloading the CPython maintainers.

</F>

 

[Fredrik Lundh]

Would you like to be?

I can help build an infrastructure that makes it easier to create a more
complete standard distribution, sure. (and since this isn't exactly a new
idea, I'm 100% confident that I don't have to do all the work myself.)

if this is needed is something for the PSF to decide, I suppose.

</F>

 

[Robert Kern]

Robert Kern wrote:




with enough volunteers, and a distributed development environment, nobody
needs to be insane. if you try to channel everything through a small number of
CPython developers, everyone will loose their sanity, and the users will suffer.

Agreed, which is why I think that proper package management and a
CPyAN-type system is the way to go, not trying to stuff everything into
a single distribution.

"updating when necessary" means "adding new components to the standard library
when there are new stable releases available, and someone has time to run the test
suite and contact the upstream provider if necessary".

Yes, that's the "royal pain in the ass" that I was referring to. Sounds
easy, but it soaks up far more time than you (I) possibly thought it
would. I think that pumping out a new release of the whole standard
library every time one package gets updated is rather absurd and
aggravates rather than ameliorates the problem.

Allow me to take an example that I'm sure you are familiar with (and can
correct me if I get things wrong). You were once asked why you didn't
use Numeric arrays as the memory store for PIL image objects. Your
(good) reason was that Numeric was not part of the standard library and
you didn't want to add a dependency on another 3rd party package. Now
let's suppose that Numeric was included in this standard library and PIL
wasn't. If you wanted to use Numeric arrays, you could not simply depend
on the standard library; you would have to depend on specific releases
of the standard library because Numeric is changing what with numarray
and the impending Numeric/numarray unification (actually, we ought to
talk about that[1]). What's more, the version numbers of the standard
library that you would have to depend on would have no particular
correlation with the versioning of the underlying component on which you
are really depending. And then someone is going to want to use a package
that requires stdlib < 2.5.1-1004 *and* a package that requires stdlib >
2.5.1-1010.

CPyAN-type systems will, if they make good on their promises, avoid this
hassle.

[1] As a side note, with the Numeric/numarray unification, the Numeric
people are developing an array interface that will allow seamless and
copyless interfacing between Numeric and other packages that either
produce something array-like (e.g. PIL) or can consume something
array-like (e.g. gui toolkits). With this protocol, PIL can support
Numeric arrays (or anything else array-ish like whatever image classes
in are in a gui toolkit) without needed to depend on the modules themselves.

http://numeric.scipy.org/

Since PIL is one of the compelling reasons for such an interface, we'd
like your input on it. If you have comments or questions, please let us
know on the Numpy mailing list.

http://lists.sourceforge.net/lists/listinfo/numpy-discussion

which means that at the moment, the best way to get a Python module into the
hands of users is to 1) provide windows installers that are compatible with the
"official" distribution, and 2) make it easy for downstream providers to package
your stuff, and leave it to them to take care of the rest.

At the moment, yes, more or less. However, neither the standard Windows
and Mac installers are what I consider to be "proper package management."

but if the library in the standard distribution doesn't really matter, it should be
stripped down to minimum, to avoid overloading the CPython maintainers.

I agree, but the CPyAN systems have to come out of the vapour first.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Fredrik Lundh]

Agreed, which is why I think that proper package management and a CPyAN-type system is the way to
go, not trying to stuff everything into a single distribution.

I think we're looking at this in two different ways: you're looking at it from the "sumo
python distribution" perspective, I'm trying to find a middle ground between a sumo
distribution and today's put-it-in-our-repository-or-else approach.

(I have more comments to your comments, but they have to wait until later)

</F>

 

[Peter Hansen]

I personally find the notion of "path objects" confusing. Is this a
real path name on the local system, or merely a potential one? If
potential, why does it have a .size attribute

(what is the size of a non-existing file)?

Logically (and, I feel, quite intuitively), it's an exception:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "c:\python24\lib\ntpath.py", line 229, in getsize
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: u'testxxx'

-Peter

 

[Skip Montanaro]

Robert> No, Scipy is just a (large) package. Enthon, on the other hand,
Robert> is just such a distribution.

I'm sorry, you lost me there. What's the difference between a large package
and a distribution?

Skip

 

[Robert Kern]

Robert> No, Scipy is just a (large) package. Enthon, on the other hand,
Robert> is just such a distribution.

I'm sorry, you lost me there. What's the difference between a large package
and a distribution?

Scipy is not a "distribution of Python." It's just a single package, a
single library; you still need the interpreter. Enthon for Windows is a
distribution of the Python interpreter and a number of packages.
Currently, MacEnthon is just a collection of packages because I can rely
on the presence of Python 2.3.0 on all OSX 10.3 machines. In the future,
I will be releasing a version for 2.4.x and that will include the
interpreter.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Robert Kern]

Robert Kern wrote:




I think we're looking at this in two different ways: you're looking at it from the "sumo
python distribution" perspective, I'm trying to find a middle ground between a sumo
distribution and today's put-it-in-our-repository-or-else approach.

Okay, then we may be advocating the same thing, and I'm misinterpreting
your comments. Wouldn't be the first time.

(I have more comments to your comments, but they have to wait until later)

I look forward to them.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

I can help build an infrastructure that makes it easier to create a more
complete standard distribution, sure. (and since this isn't exactly a new
idea, I'm 100% confident that I don't have to do all the work myself.)

if this is needed is something for the PSF to decide, I suppose.

I can't get a clear picture what precisely would have to be done, and
how you would imagine to transition from where we are now. However, my
initial reaction would be that the PSF traditionally stays out of any
kind of technical decision on the Python source code and distribution
(*). It is to support the development of Python, not to control it. So
my intuition is that perhaps you could just start. If you need to
arrange something on python.org, you could well do so. If you want to
change the way in which Python is released, you would have to convince
the release manager (primarily) of doing things differently - if your
plan involves changes to the way Python is released.

So in the end, I think this sort of change will require some convincing,
which, in turn, may require to fill paper of what the proposed changes
are and how they could come about.

Regards,
Martin

(*) Atleast formally. Informally, the opinion of the PSF members (not
the formal vote of the directors) clearly does have impact on the future
of Python - as many of the PSF members are also contributors to Python.