python's setuptools (eggs) vs ruby's gems survey/discussion

[Alia Khouri]

Can we open up the discussion here about how to improve setuptools
which has become the de facto standard for distributing / installing
python software. I've been playing around with ruby's gems which seems
to be more more mature and usable.

From my perspective, the relative immaturity of setuptools and its
simultaneous widespread use is a clear python weakness and can make
python less easy to absorb than it should be.

A few questions (please add more) so far are:

(1) Should setuptools be standard?

(2) What bugs you most about the current featureset?

(3) Which features do you need the most (list in order of need)?

(4) Shouldn't we just port gems to python?

(5) What's the best community process to improve setuptools?

(6) What's your ideal conception of the 'standard python package
manager?


To give this discussion some ammunition, I will post the output of the
different '--help' for either tool:

==================================================
SETUPTOOLS
==================================================


C:\TMP>easy_install --help

Global options:
--verbose (-v) run verbosely (default)
--quiet (-q) run quietly (turns verbosity off)
--dry-run (-n) don't actually do anything
--help (-h) show detailed help message

Options for 'easy_install' command:
--prefix installation prefix
--zip-ok (-z) install package as a zipfile
--multi-version (-m) make apps have to require() a version
--upgrade (-U) force upgrade (searches PyPI for
latest
versions)
--install-dir (-d) install package to DIR
--script-dir (-s) install scripts to DIR
--exclude-scripts (-x) Don't install scripts
--always-copy (-a) Copy all needed packages to install
dir
--index-url (-i) base URL of Python Package Index
--find-links (-f) additional URL(s) to search for
packages
--delete-conflicting (-D) no longer needed; don't use this
--ignore-conflicts-at-my-risk no longer needed; don't use this
--build-directory (-b) download/extract/build in DIR; keep
the
results
--optimize (-O) also compile with optimization: -O1
for
"python -O", -O2 for "python -OO",
and -O0 to
disable [default: -O0]
--record filename in which to record list of
installed
files
--always-unzip (-Z) don't install as a zipfile, no matter
what
--site-dirs (-S) list of directories where .pth files
work
--editable (-e) Install specified packages in
editable form
--no-deps (-N) don't install dependencies
--allow-hosts (-H) pattern(s) that hostnames must match
--local-snapshots-ok (-l) allow building eggs from local
checkouts

usage: easy_install-script.py [options] requirement_or_url ...
or: easy_install-script.py --help

==================================================
GEMS
==================================================
C:\TMP>gem --help

RubyGems is a sophisticated package manager for Ruby. This is a
basic help message containing pointers to more information.

Usage:
gem -h/--help
gem -v/--version
gem command [arguments...] [options...]

Examples:
gem install rake
gem list --local
gem build package.gemspec
gem help install

Further help:
gem help commands list all 'gem' commands
gem help examples show some examples of usage
gem help platforms show information about platforms
gem help <COMMAND> show help on COMMAND
(e.g. 'gem help install')
Further information:
http://rubygems.rubyforge.org

C:\TMP>gem help commands
GEM commands are:

build Build a gem from a gemspec
cert Manage RubyGems certificates and signing
settings
check Check installed gems
cleanup Clean up old versions of installed gems in the
local
repository
contents Display the contents of the installed gems
dependency Show the dependencies of an installed gem
environment Display information about the RubyGems
environment
fetch Download a gem and place it in the current
directory
generate_index Generates the index files for a gem server
directory
help Provide help on the 'gem' command
install Install a gem into the local repository
list Display gems whose name starts with STRING
lock Generate a lockdown list of gems
mirror Mirror a gem repository
outdated Display all gems that need updates
pristine Restores installed gems to pristine condition
from files
located in the gem cache
query Query gem information in local or remote
repositories
rdoc Generates RDoc for pre-installed gems
search Display all gems whose name contains STRING
server Documentation and gem repository HTTP server
sources Manage the sources and cache file RubyGems uses
to search
for gems
specification Display gem specification (in yaml)
uninstall Uninstall gems from the local repository
unpack Unpack an installed gem to the current directory
update Update the named gems (or all installed gems) in
the local

repository
which Find the location of a library

For help on a particular command, use 'gem help COMMAND'.

Commands may be abbreviated, so long as they are unambiguous.
e.g. 'gem i rake' is short for 'gem install rake'.

=======================================================

Hope this discussion can be constructive. In any case, I do appreciate
the effort that went into creating setuptools (Thanks Phillip J.
Eby . It's existence is clearly better than otherwise.

Best,

AK

 

[Carl Banks]

Can we open up the discussion here about how to improve setuptools
which has become the de facto standard for distributing / installing
python software. I've been playing around with ruby's gems which seems
to be more more mature and usable.

From my perspective, the relative immaturity of setuptools and its
simultaneous widespread use is a clear python weakness and can make
python less easy to absorb than it should be.

A few questions (please add more) so far are:

Oh boy.

(1) Should setuptools be standard?

Shamelessly speaking as someone who does no wish to enter this
particular niche of Python: I would hope the Python community can do
better.

(2) What bugs you most about the current featureset?

1. setuptools will download and install dependencies on the user's
behalf, without asking, by default. It can be disabled, but I think
it's extremely rude: it should ask by default.

2. One cannot simply import modules from packages that use
entry_points: you have to go through package resources. Very
annoying.

3. Tools such as py2exe don't work with packages that use entry_points
except with some hand tweaks (at least as of the last time I tried).
One might suggest that py2exe should learn how to include setuptools
models, but I don't think people who write tools like py2exe ought to
be burdened with it. py2exe was possible because the import mechanism
was theretofore so straightforward.

FWIW, I've abandoned usage of a package that used entry points because
of this issue.

(3) Which features do you need the most (list in order of need)?

Not that my needs are all important, but:

1. Works.
2. Ability to install to unusual locations (BIG)
3. Packages can be installed separately by hand if the user so desires
(copying to site directory, hand editing if necessary)
4. Ability to easily specify compiler options when rebuilding
extension modules.
5. Handles data files reasonably

(4) Shouldn't we just port gems to python?

Fine with me, some new blood would be useful here.

(5) What's the best community process to improve setuptools?

Have the BDFL declare that it's the official Python package manager.
Of course, I hope if it ever comes to that the BDFL will pronounce
something else to be the official Python package manager, preferrably
something that isn't based on distutils.

Hope this discussion can be constructive. In any case, I do appreciate
the effort that went into creating setuptools (Thanks Phillip J.
Eby . It's existence is clearly better than otherwise.

I don't agree. It's probably better than otherwise for enterprise
applications, which is the environment for which it was designed. For
instance, entry points and dependency bookkeeping are useful in such
environments, since there are policies to ensure consistent and
reasonable usage.

For open source, mom-and-pop operations, and other smaller projects, I
think it adds a whole lot of complexity over what otherwise is a
simple thing.


Carl Banks

 

[Paul Boddie]

Can we open up the discussion here about how to improve setuptools
which has become the de facto standard for distributing / installing
python software. I've been playing around with ruby's gems which seems
to be more more mature and usable.

I'm sure people also regard Perl's CPAN-related tools and
infrastructure to be more mature and usable, but I'd like to widen the
discussion beyond language-specific package and dependency management.

From my perspective, the relative immaturity of setuptools and its
simultaneous widespread use is a clear python weakness and can make
python less easy to absorb than it should be.

A few questions (please add more) so far are:

(1) Should setuptools be standard?

(2) What bugs you most about the current featureset?

(3) Which features do you need the most (list in order of need)?

I'm not really in your target audience for these questions since I
never use setuptools: instead, I use the Debian-based package and
dependency management provided by my system. If any of the system
packages use setuptools, it's to build packages in such a way that
they resemble classic distutils package installations.

However, I do work in environments where I do have to install packages
to non-system locations manually. Even in such situations, the
packages I tend to use employ a plain distutils-based setup script,
and I'm not completely sure that I'd want to use setuptools/
easy_install, since aside from some dependency management (which I
doubt extends to various non-Python libraries) it doesn't provide
compelling advantages over distutils like uninstallation, for example.

(4) Shouldn't we just port gems to python?

(5) What's the best community process to improve setuptools?

(6) What's your ideal conception of the 'standard python package
manager?

Well, I don't deny the utility of a Python package manager given that
it could be useful for people who use systems which don't provide
system package/dependency management (at least in a consistent or
widely-deployed fashion) or who have to work without taking advantage
of system packages (whether it be due to privileges or issues with
decisions taken by the package maintainers), but the most important
thing from my perspective is that it should complement and co-operate
with system packaging activities. Although distutils is often derided
for its architecture and for some odd usability issues, it does take
away a lot of the tedious work required when laying out installations
for subsequent packaging.

Perhaps efforts should be directed towards the distutils type of work,
making it easier to install things like documentation and non-code
resources, potentially even employing a different architecture: once
upon a time, Python libraries were installed using a Makefile-based
scheme, and there are plenty of Makefile-like tools and libraries
written in Python [1]. In addition, there should be efforts which
integrate this packaging with the existing range of package and
dependency managers: it shouldn't be the case that one only considers
a Python-only dependency manager, because that just leads to the usual
reinvention of what has been done before, plus those people packaging
for Debian, Fedora, *BSD and so on won't see any benefit from what has
been done. Indeed, it shouldn't be inconceivable that a Python-only
dependency management solution might be based on existing tools and
infrastructure, rather than trying to figure out issues like
reliability and redundancy all over again.

I note that the overlap between various mature projects of this nature
and the distutils community seems to be minimal. Once again, I suggest
that people take advantage of the expertise and experience built up in
other projects and communities, and not merely those whose products
conveniently resemble a preconceived notion of what such a solution
should be in the Python world. CPAN and friends and all their baggage
are not universally applicable, and any consideration merely of such
solutions will produce a setuptools successor whose relevance is just
as limited as its predecessor.

Paul

[1] http://wiki.python.org/moin/ConfigurationAndBuildTools

 

[rurpy]

Can we open up the discussion here about how to improve setuptools
which has become the de facto standard for distributing / installing
python software. I've been playing around with ruby's gems which seems
to be more more mature and usable.

From my perspective, the relative immaturity of setuptools and its
simultaneous widespread use is a clear python weakness and can make
python less easy to absorb than it should be.

A few questions (please add more) so far are:

(1) Should setuptools be standard?

I hope not. Like many others, I avoid packages
the require installation via setuptools.

My first experience with setuptools (I do not
make much distinction here between setuptools and
packages installed using setuptools) was quite
unpleasant. It was a couple years ago and I
have forgotten a lot of details but it was roughly:
Downloaded a package from the net, did the usual
"python setup.py install" and got a messages that
I needed to install setuptools first. (First time
I'd ever heard of setuptools, not idea what it was
or did.)

Machine not on internet so I looked at readme
for other options for installing. No information
about dependencies, install, easy_install, setuptools
in any of the readme's or other package doc.
Finally found a url buried in one of .py files.
Eventually, from another machine, I got it
downloaded and installed. Ran it, and then
the dependency issues popped up with no internet
connection, recollection is messages were obscure.
Got internet connection and tried again, never
did it tell me what it wanted to download or the
sizes (internet connection was a modem). Somewhere
I got a url for setuptools "help" that pointed
the Peak website. I then had to hunt around for
a long time before finding the setuptools docs.
When I did find them they appeared to be written
for a packaging programmer, not for an someone who
just want to get a package installed so he could
continue with his main task.

The kicker was that after all of this BS, I discovered
that I could install the package simply by copying
it's files to the Python install site-lib directory!
Sheesh!!

I have since used setuptools a few more times and
had problems several times.

Now, my complaint about setuptools is not any of
the specific problems I've had using it -- those
presumably could be (perhaps have been) fixed.
The problem is the disconnect between what I see
as blatantly obvious requirements for any kind of
installer and what setuptools provides. I consider
the need for locally available usage instructions so
obvious as to not needing mentioning. I consider
sensible behavior in the face of no, slow, or bad
internet connections obvious. I consider defaults
(not auto-downloading potentially large files) to
be obvious. That the the setuptools developers do
not share my world view makes me unable to trust
them. Perhaps they will silently change some settings
on my machine that I depend on?

Setuptools' philosophy seems to me to be fundamentally
Microsoftian: sit back, relax, we'll take care of
everything (which requires you to effectively give
your machine to us, but please don't worry about that.)
It is a philosophy I detest, and use free software
to get away from.

(2) What bugs you most about the current featureset?

(3) Which features do you need the most (list in order of need)?

If must *work*, always.

It must have simple, very well written docs, that
are always easily accessible.
Nobody should be expected to become a setuptools
expert in order to install a package, but if they
need to do something out of the ordinary, they
must be able to find out how, quickly and effectively.

It must work reliably and simply in a wide variety
of environments, not just the enviroments the developers
happen to be used to (no, slow, intermittent internet,
unusual install locations, multiple package versions,
multiple python versions, different compilation
environments...)

It's scope should be limited to Python. There are
OS packaging tools for applications (python or
otherwise), and the ongoing problems with them
(c.f. Fedora's rpm/yum) despite a decade of work
make me doubtful a python centric project that
tried to tackle the problem could get it right.

Need to be able to *manage* packages (remove, list,
maintain info about,...) as well as install them.

A common way of packaging / installing documentation.
Most of my Python work in on Windows and whenever I
install a Python package (setuptools or otherwise),
I have to look for a sepreate docs package, and if
not exist, download the source package, find the
docs in it, sometimes build them, and if I can't
to that, try to wget the online docs. Would like
a nerw setuptools to provide the machinery and
conventions to allow packager to easily include
docs in a package and have them installed in a
common, indexed, location on the target machine.
(C.f. Activestate's Perl.)

 

[Sion Arrowsmith]

1. setuptools will download and install dependencies on the user's
behalf, without asking, by default.

It will *attempt* to download etc. etc. on the assumption that you
have convenient, fast network connection. If you don't....

My experience is getting on for a couple of years old now, so things
*may* have improved. But I was trying to install something (Kid, I
think) from behind a coroporate firewall which was very picky about
what outgoing connections were permissible and found trying to
satisfy the dependencies a serious headache. I can't remember why
it wasn't "just" a case of manually installing the dependencies; I
do remember that the documentation was utterly inpenetrable and had
no information on trouble-shooting as an installer.

The egg format may well be OK, but in my experience setuptools is
worse than nothing. A better option to a port of gems would be a
toolset which made eggs usable.

 

[Alia Khouri]

Might as well answer the survey myself:

A few questions (please add more) so far are:

(1) Should setuptools be standard?

Not at this stage. What pythonistas need is a cross-platform package
manager that is included in the stdlib. Setuptools is simply not
mature enough nor pythonic enough to be standard in its current form.

(2) What bugs you most about the current featureset?

- no uninstall so I have to spend time manually removing old egg files/
directories
- pollutes my site-packages directory with lots of sparse egg-info
files
- uglifies my sys.path
- the egg directory structure is unpythonically deep
- confusing documentation
- easy_install is a mouthful (why not 'python -m egg.install')

(3) Which features do you need the most (list in order of need)?

- uninstall
- clean up location of egg files
- better dependency management (doesn't break)
- works with multiple versions of python
- list available packages / versions / dependencies
- local database of packages (why not use sqlite?)

(4) Shouldn't we just port gems to python?

I don't think that's bad idea...

(5) What's the best community process to improve setuptools?

Perhaps we can all contribute some cash/time/effort to improve
setuptools or even rewrite the python package manager of our
collective dreams...

(6) What's your ideal conception of the 'standard python package
manager?

Pure Python. Combines the best features of port/apt-get/gems while
being in the stdlib

AK