Enumerating Classes in Modules

[Rob Snyder]

Greetings -

I have a situation where I need to be able to have a Python function that
will take all the modules in a given directory and find all the classes
defined in these modules by name. Ultimately, I would need to loop through
all of these, instantiating each of the classes in turn and calling a
pre-known method of each.

Finding the name of each module is not a problem, but loading the classes
out the module once I know the name is where I'm lost.

The idea is basically to support "plug-ins" for an application I'm building.
I want the end user to be able to sub-class an example plug in class, then
throw the implementation in a given directory, where I'll find it
automatically when I need it.

I *am* still relatively new to Python, so I fully appreciate that I may be
simply going about this all the wrong way. Any guidance or redirection
anyone can provide would be appreciated.

Many Thanks.

Rob Snyder

 

[Jeff Shannon]

Greetings -

I have a situation where I need to be able to have a Python function that
will take all the modules in a given directory and find all the classes
defined in these modules by name. Ultimately, I would need to loop through
all of these, instantiating each of the classes in turn and calling a
pre-known method of each.

Finding the name of each module is not a problem, but loading the classes
out the module once I know the name is where I'm lost.

Hm. Any reason to not simply import the module (using the __import__()
function), going through all names in that module's namespace, and
attempting to call the method on them? (Catching AttributeErrors as
necessary, of course.)

Something like this (untested, unresearched, and totally off the top of
my head):

for modname in nameslist:
plugins[modname] = __import__(modname)
# ....
for item in dir(plugins[modname]):
try:
item.standard_method()
except AttributeError:
pass

If you need something a bit more specific, maybe you could require all
plugin classes to be derived from a common subclass, and before calling
the standard_method() you could check issubclass(item, plugin_base)...

Jeff Shannon
Technician/Programmer
Credit International

 

[Rob Snyder]

Greetings -

I have a situation where I need to be able to have a Python function that
will take all the modules in a given directory and find all the classes
defined in these modules by name. Ultimately, I would need to loop through
all of these, instantiating each of the classes in turn and calling a
pre-known method of each.

Finding the name of each module is not a problem, but loading the classes
out the module once I know the name is where I'm lost.

Hm. Any reason to not simply import the module (using the __import__()
function), going through all names in that module's namespace, and
attempting to call the method on them? (Catching AttributeErrors as
necessary, of course.)

Something like this (untested, unresearched, and totally off the top of
my head):

for modname in nameslist:
plugins[modname] = __import__(modname)
# ....
for item in dir(plugins[modname]):
try:
item.standard_method()
except AttributeError:
pass

If you need something a bit more specific, maybe you could require all
plugin classes to be derived from a common subclass, and before calling
the standard_method() you could check issubclass(item, plugin_base)...

Jeff Shannon
Technician/Programmer
Credit International

Any reason no to? Not other then I couldn't figure it out on my own.

Thanks so much for your help, this will do the trick nicely.

Rob Snyder

 

[Mike Meyer]

Greetings -
The idea is basically to support "plug-ins" for an application I'm building.
I want the end user to be able to sub-class an example plug in class, then
throw the implementation in a given directory, where I'll find it
automatically when I need it.

webcheck <URL: http://www.mired.org/webcheck > supports plugin
modules. You might want to give it a look.

<mike

 

[Rob Snyder]

Greetings -

I have a situation where I need to be able to have a Python function that
will take all the modules in a given directory and find all the classes
defined in these modules by name. Ultimately, I would need to loop
through all of these, instantiating each of the classes in turn and
calling a pre-known method of each.

Finding the name of each module is not a problem, but loading the classes
out the module once I know the name is where I'm lost.

Hm. Any reason to not simply import the module (using the __import__()
function), going through all names in that module's namespace, and
attempting to call the method on them? (Catching AttributeErrors as
necessary, of course.)

Something like this (untested, unresearched, and totally off the top of
my head):

for modname in nameslist:
plugins[modname] = __import__(modname)
# ....
for item in dir(plugins[modname]):
try:
item.standard_method()
except AttributeError:
pass

If you need something a bit more specific, maybe you could require all
plugin classes to be derived from a common subclass, and before calling
the standard_method() you could check issubclass(item, plugin_base)...

Jeff Shannon
Technician/Programmer
Credit International

Any reason no to? Not other then I couldn't figure it out on my own.

Thanks so much for your help, this will do the trick nicely.

Rob Snyder

Just in case anyone "follows in my footsteps" and has the same problem I did
- Jeff's answer is pretty much dead on. The only stumbling block I had was
that the dir() returns a list of strings representating the name of the
attributes, and includes things that are not class definitions (e.g.
'--builtins--'). So two modifications are necessary:

1 - use the getattr() function with each item returned by dir() to actually
load the class object
2 - use type() to determine if it really it really is a classobj

For example:
for modname in nameslist:
    plugins[modname] = __import__(modname)
    # ....
    for item in dir(plugins[modname]):
        try:
a = getattr(plugins[modname],item)
if (type(a).__name__ == 'classobj'):
             item.standard_method()
        except AttributeError:
            pass

(or something similar, you'd probably want to do something with the try...
except).

Again, thanks for the help. I apologize for clogging up people's
newsreaders, I've just found it helpful when poeple post their conculsions
with things, so I wanted to do the same.

Rob

 

[Jeff Shannon]

Just in case anyone "follows in my footsteps" and has the same problem I did
- Jeff's answer is pretty much dead on. The only stumbling block I had was
that the dir() returns a list of strings representating the name of the
attributes, and includes things that are not class definitions (e.g.
'--builtins--'). So two modifications are necessary:

1 - use the getattr() function with each item returned by dir() to actually
load the class object

Right. Forgot about that little detail. Oops.

2 - use type() to determine if it really it really is a classobj

For example:
for modname in nameslist:
plugins[modname] = __import__(modname)
# ....
for item in dir(plugins[modname]):
try:
a = getattr(plugins[modname],item)
if (type(a).__name__ == 'classobj'):
item.standard_method()
except AttributeError:
pass

Be aware that this will fail for new-style classes --
.... pass
....
Unless your required-method name is something that *is* really common
(i.e. likely to be present in __builtins__ or in other modules imported
by the plugin module), you're probably better off not trying to check
the type. Attempting the lookup won't hurt anything, as long as you
catch the AttributeError.
['EINVAL', 'StringIO', '__all__', '__builtins__', '__doc__', '__file__',
'__name__', 'test', 'types'].... obj = getattr(mymodule, item)
.... try:
.... obj.plugin_init()
.... except AttributeError:
.... print "%s has no attribute 'plugin_init'" % item
....
EINVAL has no attribute 'plugin_init'
StringIO has no attribute 'plugin_init'
__all__ has no attribute 'plugin_init'
__builtins__ has no attribute 'plugin_init'
__doc__ has no attribute 'plugin_init'
__file__ has no attribute 'plugin_init'
__name__ has no attribute 'plugin_init'
test has no attribute 'plugin_init'
types has no attribute 'plugin_init'
So as long as you're fairly careful about selecting your method name,
there's no reason to bother with typechecking. And if you *do* need to
be more specific, then you're probably better off defining a specific
plugin_base class and requiring all plugin classes to inherit from it
(enforced with issubclass(), presumably).

Again, thanks for the help. I apologize for clogging up people's
newsreaders, I've just found it helpful when poeple post their conculsions
with things, so I wanted to do the same.

Followups are definitely a good thing.

Jeff Shannon
Technician/Programmer
Credit International

 

[Nick Coghlan]

Just in case anyone "follows in my footsteps" and has the same problem I did
- Jeff's answer is pretty much dead on. The only stumbling block I had was
that the dir() returns a list of strings representating the name of the
attributes, and includes things that are not class definitions (e.g.
'--builtins--'). So two modifications are necessary:

1 - use the getattr() function with each item returned by dir() to actually
load the class object
2 - use type() to determine if it really it really is a classobj

I would suggest "hasattr" with the method you're interested in as a better way
to perform step 2. That should cover both old and new style classes quite happily.

Cheers,
Nick.

 

[Carlos Ribeiro]

Rob Snyder wrote:



I would suggest "hasattr" with the method you're interested in as a better way
to perform step 2. That should cover both old and new style classes quite happily.

Using hasattr() is better than checking the type. But a better, and
more generic solution, is to use the adapt framework. I've written a
short tutorial on adaptation. It's at my blog:

http://pythonnotes.blogspot.com/2004/11/what-is-adaptation.html

In this case, the idea is: get every type you can find, and try to
adapt it to the interface you want. If the adaptation is succesfull,
the result is the object you want.

Also, if you prefer to keep doing things the "old way", a possibility
is to use isclass and issubclass.

issubclass is a builtin. You can just call it, and it will tell you if
an object is a descendant of the given class. But it will raise an
exception if the argument is *not* a class. It belongs to the BAFP -
"better to ask for forgiveness than permission" coding style.

If you prefer to code in the LBYL - "look before you leap" - style,
then you can use isclass, from the inspect module.

from inspect import isclass
if isclass(x) and issubclass(x, myclass):
...


--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Terry Hancock]

I have a situation where I need to be able to have a Python function that
will take all the modules in a given directory and find all the classes
defined in these modules by name. Ultimately, I would need to loop through
all of these, instantiating each of the classes in turn and calling a
pre-known method of each.

Finding the name of each module is not a problem, but loading the classes
out the module once I know the name is where I'm lost.

The idea is basically to support "plug-ins" for an application I'm building.
I want the end user to be able to sub-class an example plug in class, then
throw the implementation in a given directory, where I'll find it
automatically when I need it.

I *am* still relatively new to Python, so I fully appreciate that I may be
simply going about this all the wrong way. Any guidance or redirection
anyone can provide would be appreciated.

You've probably heard all you needed to, but I can show you how I did it:

When I started working on VarImage, I just wanted it to do resizing, but
I quickly realized I wanted to support a wide variety of image filtering
operators, so I devised a plugin system, very similar to what you describe.

The full source can be had at: http://sourceforge.net/projects/narya-project

But the plugin finding code ( Operators/__init__.py ) just does this:

# Find and load all available plugin modules:

operator_path = os.path.abspath(__path__[0])
for module_file in filter(
lambda n: n[-3:]=='.py' and n not in ('__init__.py', 'Operators.py'),
os.listdir(operator_path)):
#print "Loading %s" % module_file
f, e = os.path.splitext(module_file)
__import__(f, globals(), locals(), [])

The magic actually happens in the Operators/Operators.py file where I define
a dictionary called "Ops" that holds all the plugin objects as they are loaded.

Abridgement of part of my "Operator" class:

Ops = {}

class Operator:
"""
Image operator object.
"""
def __init__(self, id, func, ...):
self.id = id
self.op = func
# [... stuff to do with actually using the operator ...]
# Register:
Ops[id] = self

I didn't want the plugin author to have to understand OOP -- just write a
function with the right profile. So I use an instantiation of the class to build
each operator plugin class instance based on the function they provide. All
the plugin author has to do is call this class at the end of their module,
with the appropriate function as argument (example from Operators/pil_crop.py):

The essentials are just:

import Operators

def trim(...):
"Documentation text which my help method reads"
return ...

Operator('trim', trim, 0, 1, (int,))

This says to define an operator with name "trim", calling the
function trim, which can take a minimum of 0 arguments and a
maximum of 1 argument, which must be an int. (not really
type-checking, it just says what it should use to convert
the string it will get before calling this function).

I think the main conceptual difference here is that instead
of trying to use magical class discovery methods, I just let
the plugin author tell me which functions he wants me to make
into operators, and try to make that as simple as possible.

Does this help you at all?

Cheers,
Terry