Listing modules from all installed packages

[Julien Phalip]

Hi,

I'm trying to write a function that programmatically obtains and returns the exact location of all first-level modules for all installed packages.

For example, if the packages named 'django' and 'django-debug-toolbar' are installed, I'd like this function to return something like:/Users/my_user/.virtualenvs/my_venv/lib/python2.6/site-packages/django
/Users/my_user/.virtualenvs/my_venv/src/debug_toolbar

That is, this function needs to consider all installed packages, including those that have been installed in "edit" mode (i.e. in the src/ folder). Note also that the main module for the 'django-debug-toolbar' is in fact named 'debug_toolbar'.

So far the closest I've been to retrieving the list of first-level modules is as follows:

import os
import pkg_resources
import setuptools

pkgs = set()

for dist in pkg_resources.working_set:
if os.path.isdir(dist.location):
for pkg in setuptools.find_packages(dist.location):
if '.' not in pkg:
pkgs.add(pkg)

The idea is then to loop through that list of modules, import them and get their exact locations by fetching their __file__ attribute values.

However, this feels very hackish and I don't think it's actually quite correct either. I'm sure there must be a better way. If possible I'd also like to avoid having to use setuptools.

Do you have any tips on how to achieve this?

Many thanks!

Julien

 

[Carlos Nepomuceno]

print '\n'.join([re.findall("from '(.*)'",str(v))[0] for k,v in sys.modules.items() if str(v).find('from')>-1])

 

[Carlos Nepomuceno]

Just realized that you've asked for installed packages. Perhaps the following will do the trick. I don't know why the 'lib-tk' isn't included. Why not?

toplevel_packages = ['%s\\%s'%(ml.path,name)for ml,name,ispkg in pkgutil.iter_modules() if ispkg]
print '\n'.join(toplevel_packages)

 

[cclauss]

Adding : python -c 'help("modules") to the other two suggestions:

#!/usr/bin/env python

import commands, pkgutil, re, sys

print('sys.modules.items()...')
print('\n'.join(sorted([re.findall("from '(.*)'",str(v))[0] for k,v in sys.modules.items() if str(v).find('from')>-1])))

print('\npkgutil.iter_modules()...')
toplevel_packages = ['%s\\%s'%(ml.path,name)for ml,name,ispkg in sorted(pkgutil.iter_modules()) if ispkg]
print '\n'.join(toplevel_packages)

theCommand = "python -c 'help(\"modules\")'"
print('\n{} # this may take a few seconds...'.format(theCommand))
print(commands.getstatusoutput(theCommand)[1]) # help() only works in the python interpreter...

 

[88888 Dihedral]

Carlos Nepomucenoæ–¼ 2013å¹´6月9日星期日UTC+8下åˆ1時23分15秒寫é“：

print '\n'.join([re.findall("from '(.*)'",str(v))[0] for k,v in sys.modules.items() if str(v).find('from')>-1])

Date: Sat, 8 Jun 2013 21:30:48 -0700
Subject: Listing modules from all installed packages
From: (e-mail address removed)
To: (e-mail address removed)

Hi,

I'm trying to write a function that programmatically obtains and returns the exact location of all first-level modules for all installed packages.

For example, if the packages named 'django' and 'django-debug-toolbar' are installed, I'd like this function to return something like:
/Users/my_user/.virtualenvs/my_venv/lib/python2.6/site-packages/django
/Users/my_user/.virtualenvs/my_venv/src/debug_toolbar

That is, this function needs to consider all installed packages, including those that have been installed in "edit" mode (i.e. in the src/ folder).. Note also that the main module for the 'django-debug-toolbar' is in fact named 'debug_toolbar'.

So far the closest I've been to retrieving the list of first-level modules is as follows:

import os
import pkg_resources
import setuptools

pkgs = set()

for dist in pkg_resources.working_set:
if os.path.isdir(dist.location):
for pkg in setuptools.find_packages(dist.location):
if '.' not in pkg:
pkgs.add(pkg)

The idea is then to loop through that list of modules, import them and get their exact locations by fetching their __file__ attribute values.

However, this feels very hackish and I don't think it's actually quite correct either. I'm sure there must be a better way. If possible I'd also like to avoid having to use setuptools.

Do you have any tips on how to achieve this?

Many thanks!

Julien

Please use a dictionary to store a tree first.
Then it is trivial to walk through all nodes of the tree.

 

[Julien Phalip]

Just realized that you've asked for installed packages. Perhaps the following will do the trick. I don't know why the 'lib-tk' isn't included. Why not?

toplevel_packages = ['%s\\%s'%(ml.path,name)for ml,name,ispkg in pkgutil.iter_modules() if ispkg]
print '\n'.join(toplevel_packages)

Thanks a lot Carlos, this gives me exactly what I needed!

Best wishes,

Julien