Modules for inclusion in standard library?

[Harry George]

How about integrating distutils and PyPI, so that distutils can
automatically download and install packages that are required by the
package it's currently installing? In other words, C-Python-AN.

<mike

I personally don't do auto download/installs, preferring instead to
scan the code before build/install. However, getting a list of
dependencies would be nice. Maybe a setup.py --requires flag.

 

[Rocco Moretti]

I've never heard of Python having such a policy and I don't understand
how such a stupid policy could be considered compatible with a
proclaimed "batteries included" philosophy. Why would Python
advocates want to make Python deliberately uncompetitive with PHP,
Java, and other languages that do include database modules?

Well, since there seems to be an outpouring of disgust at my statement,
and no official confirmation/rejection, it's probably a figment of my
prematurely failing mind.

However, if there was such a policy, it would not be unequivocally
"stupid." First off, there is a bit of flexibility in what is considered
part of the OS. E.g, Linux may properly refer to just the kernel, but
rarely is just the kernel installed. Various utilities and programs
might be considered part of the OS because they are ubiquitously
installed, or are included with the Python distribution itself (as Tk is
with windows Python).

For those programs which aren't ubiquitously installed, or even for ones
that are, but require significant configuration, it is reasonable to
expect that if someone has the ability and goes to the effort of
locating, obtaining, installing, and configuring a third party program,
they can just as easily obtain and install the python module, especially
as it's usually as easy as "python setup.py install".

At any rate, I'm not advocating such a policy, I'm just saying it can
make a bit of sense if you look at it from a certain angle.

 

[Daniel Dittmar]

There are several degrees of std-ness for Python Modules:

Example expat: The sources for this extension module are contained in
the Python tarball, so this module is guaranteed to be part of a Python
distribution

Example zlib: If the include files and libs can be found, this extension
will be built. So it's part of std-Python (the source), but not part of
std-Python (installed). See others in Modules/Setup

Binary distributions may choose additional modules which appear to be
standard. Example: zlib is part of all Python-Installations on Windows
and most Linux-distribution will have readlines as 'standard'.

There seems to be a great reluctance by the Python developers to add
modules of the expat kind, as this means responsibilities for additional
source modules. There's also the problem with incompatible licenses,
integrating a second configure, deciding when to update to the latest
version of the library etc.

Another problem is that there are often several modules for a specific
problem (e.g. several modules interfacing to PostgreSQL), so there's
always a chance to make an anemy for life, no matter which you pick.

And module authors might not be interested as they'd be then bound by
the Python release cycle.

Daniel

 

[Scott David Daniels]

....

There seems to be a great reluctance by the Python developers to add
modules of the expat kind, as this means responsibilities for additional
source modules. There's also the problem with incompatible licenses,
integrating a second configure, deciding when to update to the latest
version of the library etc.

If you haven't noticed, the Python code has a substantial body of unit
tests. Arranging the tests to be easily runnable for all developers
is going to be tough for "third party programs." Making the interfaces
work for differing versions of the 3PPs as the third parties themselves
change their interfaces (see fun with Tcl/Tk versions for example), and
building testbeds to test to all of those differing versions, would
cause a nightmare that would make a knight of Ni scream.

--Scott David Daniels
(e-mail address removed)

 

[Colin J. Williams]

There has been recent discussion about this. Check the python-dev list
archives I think. It's unlikely that all of numeric/numarray could go
into the Python stdlib because there still is disagreement between the
two camps as to the module architecture. However, there does seem to be
some agreement at the level of the basic array object, so it may be
possible that at least the array object itself might join the stdlib in
the not too distant future. I suspect there's more detailed discussion
of this in the numeric/numarray lists.

There was a flurry of exchanges about three months ago but it soon died.

One thing which needs to be sorted out is the silent truncation of
complex values:
http://sourceforge.net/tracker/index.php?func=detail&aid=1216688&group_id=1369&atid=450446

Colin W.

 

[Colin J. Williams]

While that policy does make sense, I think a database program falls
somewhere in between an OS and an everyday third party program. For
web developers, the database might as well be the OS. I use the
database to store everything in my web app. That way I can just worry
about 1 place to access information and not have to fool with files
and other OS issues.

So I humbly suggest the policy should be :

Python will not include interface code for third party programs which
are not part of an operating system or database system.

...

Isn't this where the discussion should start? There should be some
general policy guiding the types of modules which should be in the
standard library. Clearly, size is a factor as the msi version of
Python 2.4 is now 10 MB.

if there were some sort of package manager which took account of
dependencies would that reduce the need to expand the standard library?

Colin W.

 

[Colin J. Williams]

Well, since there seems to be an outpouring of disgust at my statement,
and no official confirmation/rejection, it's probably a figment of my
prematurely failing mind.

However, if there was such a policy, it would not be unequivocally
"stupid." First off, there is a bit of flexibility in what is considered
part of the OS. E.g, Linux may properly refer to just the kernel, but
rarely is just the kernel installed. Various utilities and programs
might be considered part of the OS because they are ubiquitously
installed, or are included with the Python distribution itself (as Tk is
with windows Python).

For those programs which aren't ubiquitously installed, or even for ones
that are, but require significant configuration, it is reasonable to
expect that if someone has the ability and goes to the effort of
locating, obtaining, installing, and configuring a third party program,
they can just as easily obtain and install the python module, especially
as it's usually as easy as "python setup.py install".

At any rate, I'm not advocating such a policy, I'm just saying it can
make a bit of sense if you look at it from a certain angle.

I agree. It makes sense to develop a policy first.

Colin W.

 

[Tom Anderson]

If you haven't noticed, the Python code has a substantial body of unit
tests. Arranging the tests to be easily runnable for all developers
is going to be tough for "third party programs."

The tests for interface modules would have to use mock objects on the back
end. This is pretty standard practice, isn't it?

Making the interfaces work for differing versions of the 3PPs as the
third parties themselves change their interfaces (see fun with Tcl/Tk
versions for example), and building testbeds to test to all of those
differing versions, would cause a nightmare that would make a knight of
Ni scream.

But given that at a number of such modules have in fact been written,
along with tests, why not add them to the standard distribution?

tom

 

[Terry Reedy]

Isn't this where the discussion should start? There should be some
general policy guiding the types of modules which should be in the
standard library.

A couple of times, Guido has given his general policy as generally useful;
best-of-breed, tested and accepted by the community; and backed by a
developer who will adapt it and its doc up to stdlib standards (if
necessary) and commit to maintainence for a few years.

Terry J. Reedy

 

[Colin J. Williams]

A couple of times, Guido has given his general policy as generally useful;
best-of-breed, tested and accepted by the community; and backed by a
developer who will adapt it and its doc up to stdlib standards (if
necessary) and commit to maintainence for a few years.

Terry J. Reedy

This is a good base. Presumably "accepted by the community" means
with some minimum number of ongoing users.

Colin W.

 

[Terry Reedy]

This is a good base. Presumably "accepted by the community" means
with some minimum number of ongoing users.

Yes. Users also indicate practical (versus theoretical) usefullness and
also test beyond what a developer might (especially documentation).

Of course, modules written by the inner core group of active developers or
on their behalf may bypass these criteria, but the context of my answer was
discussion of inclusion of existing modules mostly or all written by other
people.

Part of the maintainence requirement is a willingness to work compatibly
with the Python release cycle. Related to that is something I left out on
the other side: willingness to license the module to the PSF for inclusion.
Some people seem to assume that everyone 'of course' wants their stuff
included and that it is just Guido blocking inclusion, but that just is not
so.

Terry J. Reedy

 

[awbarker]

1. LDAP module should be included in the base distro.
2. DNS library really should be included in the base library, I emailed
Anthony Baxter and he replied, saying it was almost done.
3. Ipython would be nice

 

[=?ISO-8859-1?Q?Gregory_Pi=F1ero?=]

Has anyone recommended ftputil? Either add that to the library or
make the existing ftp module more high level would be my suggestion.

http://www.sschwarzer.net/python/python_software.html

-Greg