class / module introspection?

[Brian Munroe]

I'm struggling with an architectural problem and could use some
advice.

I'm writing an application that will gather statuses from disparate
systems. Because new systems show up all the time, I'm trying to
design a plugin architecture that will allow people to contribute new
backends by just dropping a package/module into a specific directory.
The object methods in these backends will conform to a documented API
that the main application will call.

Currently I have something that looks like this:

src/
backends/
system1/
__init__.py
system2/
__init__.py
...

Then from my application (main.py) I can simply call:

from backends import system1

be1 = system1.Backend()
be1.getStatus()

This would work great if I knew about system1 and system2 ahead of
time, but that isn't the case. Having to rewrite main.py every time a
new backend module comes along is obviously a stupid idea too. I've
been thinking I need some kind of introspection, but I've been reading
about it and a whole mess of design pattern stuff, so my head is
swimming and I am totally unsure of what the best approach is.

My guess is that I need to load all the backends at runtime - then
introspect the loaded classes?

Any suggestions would be greatly appreciated.

 

[bruno.desthuilliers]

I'm struggling with an architectural problem and could use some
advice.

I'm writing an application that will gather statuses from disparate
systems. Because new systems show up all the time, I'm trying to
design a plugin architecture that will allow people to contribute new
backends by just dropping a package/module into a specific directory.
The object methods in these backends will conform to a documented API
that the main application will call.

Currently I have something that looks like this:

src/
backends/
system1/
__init__.py
system2/
__init__.py
...

Then from my application (main.py) I can simply call:

from backends import system1

be1 = system1.Backend()
be1.getStatus()

This would work great if I knew about system1 and system2 ahead of
time, but that isn't the case. Having to rewrite main.py every time a
new backend module comes along is obviously a stupid idea too. I've
been thinking I need some kind of introspection, but I've been reading
about it and a whole mess of design pattern stuff, so my head is
swimming and I am totally unsure of what the best approach is.

My guess is that I need to load all the backends at runtime

Anyway, almost everything happens at runtime in Python !-)

More seriously: the answer is in the doc.
http://www.python.org/doc/2.3.5/lib/built-in-funcs.html

read about the __import__ function, experiment in your interactive
python shell, and you should be done in a couple minutes.

 

[Brian Munroe]

More seriously: the answer is in the doc.http://www.python.org/doc/2.3.5/lib/built-in-funcs.html

read about the __import__ function, experiment in your interactive
python shell, and you should be done in a couple minutes.

Well, If I understand the docs correctly, that would work great if
backends/ was a module and not a package? I need to keep backends/
system1/ and backends/system2 as separate directory structures to make
things fairly isolated from each other (for neatness sake)

Currently I'm building the backends/__init__.py __all__ list
dynamically, such as:

backends/__init__.py
--------------------

import os

__all__ = []

for module in os.listdir(__path__[0]):
if not module.startswith("__"):
__all__.append(module)

then from my main application, I can do the following

main.py
-------

import backends

print backends.__all__

This gives me ['system1','system2'] - which I can then use __import__
on.

-- brian

 

[Brian Munroe]

This gives me ['system1','system2'] - which I can then use __import__
on.

Addendum, thanks Bruno!

I also required the helper function (my_import) from the Python docs
you pointed me to, that actually was the key to getting everything
working!

 

[bruno.desthuilliers]

Well, If I understand the docs correctly, that would work great if
backends/ was a module and not a package?

Not necessarily.

I need to keep backends/
system1/ and backends/system2 as separate directory structures to make
things fairly isolated from each other (for neatness sake)

Currently I'm building the backends/__init__.py __all__ list
dynamically, such as:

backends/__init__.py
--------------------

import os

__all__ = []

for module in os.listdir(__path__[0]):
if not module.startswith("__"):
__all__.append(module)

Why not do the import here, so you store a real module instead of a
name ?

ie (not tested):

for module_name in os.listdir(__path__[0]):
if not module_name.startswith("__"):
__all__.append(__import__(module_name, globals(), locals()))

My 2 cents...

 

[Brian Munroe]

Why not do the import here, so you store a real module instead of a
name ?

Right now I'm still in the prototyping phase and haven't really
thought everything through. I needed the names because I am
populating a GUI selection list element. I also assumed that I could
then lazy load the modules I needed...

Thanks for the help though, you've gotten me past my first of many
(I'm sure) hurdles!

-- brian

 

[7stud]

This gives me ['system1','system2'] - which I can then use __import__
on.

Addendum, thanks Bruno!

I also required the helper function (my_import) from the Python docs

You don't need that helper function, which is a little tricky to
follow:

If you have a module name as a string, you can use the builtin
function __import__ to import the module, e.g.

x = __import__("mymod")
x.myfunc()

Note that you have to assign the return value of __import__ to a
variable. Thereafter you use the variable to access the module's
functions.

However, when you use __import__ with a package, it behaves in a
peculiar manner: it returns the top level package object. For
example, if you write:

x = __import__("mypackage.subpack.myfuncs")


then x is a reference to mypackage--not mypackage.subpack.myfuncs. In
order to execute a function in the module myfuncs, you need to write:

x.subpack.myfuncs.f().

Or, if you check the docs, you can make __import__ return the
rightmost module in the module string by writing:

x = __import__("mypackage.subpack.myfuncs", globals(), locals(),
["subpack.myfuncs"])

The last argument is what makes __import__ return a lower level module
in the package hierarchy.


To solve your problem, you can do something like the following:

Directory structure
-------------------

../mypackage
__init__.py

/system1
__init__.py
myfuncs.py

/system2
__init__.py
myfuncs.py


system1/myfuncs.py
------------
def f():
print "system1 here"


system2/myfuncs.py
-----------------
def f():
print "system2 here"




import sys
import os

#Get all the file names contained in mypackage:
fnames = os.listdir("../mypackage")
print fnames

#Separate out the directory names(which are the sub packages):
os.chdir("../mypackage") #so abspath() below can construct full paths
package_dirs = []

for fname in fnames:
abs_path = os.path.abspath(fname) #need full path for isdir()
print abs_path,

if os.path.isdir(abs_path):
print 'dir'
package_dirs.append(fname)
else:
print 'file'

print package_dirs

sys.path.append("../") #directs python to search the specified
#dir for any modules that are imported

#Import the sub packages:
for dir in package_dirs:
modname = "%s.%s.%s" % ("mypackage", dir, "myfuncs")
modname_minus_toplevel_packname = "%s.%s" % (dir, "myfuncs")

mod = __import__(modname, globals(), locals(),
[modname_minus_toplevel_packname])
mod.f()



--output:--
['__init__.py', '__init__.pyc', 'system1', 'system2']
/Users/me/2testing/mypackage/__init__.py file
/Users/me/2testing/mypackage/__init__.pyc file
/Users/me/2testing/mypackage/system1 dir
/Users/me/2testing/mypackage/system2 dir
['system1', 'system2']
system1 here
system2 here

 

[bruno.desthuilliers]

Right now I'm still in the prototyping phase and haven't really
thought everything through. I needed the names because I am
populating a GUI selection list element.

import os
print os.__name__
'os'

I also assumed that I could
then lazy load the modules I needed...

Ok, that's something else. I had (wrongly) assumed you wanted to load
all modules anyway.

Thanks for the help though, you've gotten me past my first of many
(I'm sure) hurdles!

You're welcome.

 

[Brian Munroe]

You don't need that helper function, which is a little tricky to
follow:

Well, I appreciate the code sharing you did, but the helper function
is nice and compact, and I didn't have much trouble following it. I
ended up doing the following in the backends/__init__.py:

import os
availableBackendsList = []

for module in os.listdir(__path__[0]):
if not module.startswith("__") and not module.startswith("."):
availableBackendsList.append("backends." + module)

def subpackage_import(name):
mod = __import__(name)
components = name.split('.')
for comp in components[1:]:
mod = getattr(mod, comp)
return mod

def get_available_backends():
return map(subpackage_import, availableBackendsList)

In then in my application code, it becomes a simple matter of:

import backends

for x in backends.get_available_backends():
print x.PLUGIN_NAME
be = x.Backend()
print be.getStatus()

Basically, I borrowed a page out of Dive into Python[1] and mapped
each module object (system1, system2, ... systemN) to a list.


[1] - Thanks Mark! - http://www.diveintopython.org/functional_programming/all_together.html