Looking for some PyPI query help

[Travis Griggs]

Howdy Python brethren. I'm a Smalltalker, doing a bit of research on packaging ecospaces, in other languages and environments (I just finished examining Debian for example). I found what seems to be the big repository at PyPI. What would be enough for me, is to enumerate all of the packages there, and associate it with the number of prerequisite/dependent/imports it specifies. I don't need it to recurse thru dependents, just the immediate count.

My goal is to be able to histogram the data, and be able to answer questions along the following lines:
1) What's the average/median dependent count in the Python ecospace.
2) For a given threshold (say 90%) when sorted by count, what's the count (e.g. "90% of Python packages depend directly on 5 or less other packages").
3) What's the most number of dependents any package has ever specified?

Thanks for any help or pointers or hints or data you can give me.

(I apologize if this is overtly naive).

 

[Martin v. Loewis]

Am 23.02.2011 07:53, schrieb Travis Griggs:

Howdy Python brethren. I'm a Smalltalker, doing a bit of research on
packaging ecospaces, in other languages and environments (I just
finished examining Debian for example). I found what seems to be the
big repository at PyPI.

Correct. There is a mailing list for it: (e-mail address removed).

What would be enough for me, is to enumerate
all of the packages there, and associate it with the number of
prerequisite/dependent/imports it specifies. I don't need it to
recurse thru dependents, just the immediate count.

Finding all the packages is easy:

http://pypi.python.org/simple/

Finding the respective dependencies is very very difficult.

You will need to download each package. If it is implemented using
setuptools, you can extract dependencies from the setup.py, looking
at setup_requires, install_requires, tests_require, extras_require.
If it is implemented using distutils, you need to read the documentation
to find out what the dependencies are, if any.

For many packages, it should be possible to find the corresponding
Debian or Redhat package, and correlate the dependency lists.

HTH,
Martin