Clearing a session and reload() problem (with repro error)

[Rafe]

Hi,

This seems to be an old question, and I've read back a bit, but rather
than assume the answer is "you can't do that", I'd thought I'd post my
version of the question along with a reproducible error to illustrate
my confusion.

My problem is that I'm using Python inside XSI (a 3D graphics
application). If I want to restart Python, I have to restart XSI. This
is no small amount of time wasted.

The solution would be to somehow unload modules and all references and
allow my imports to rebuild the cache. I suppose it would be great to
clear the python session completely, but the 3D application might have
added something to the session at startup (I doubt this though as I
don't see any imported modules outside the norm).

I've tried to use reload with a very simple algorithm. Simply run
through every imported module, ignoring anything that is "None" or on
the C: drive (all of our python is on a network drive so this hack
works for me for now) and reload() it. I've come to realize that this
isn't near intelligent enough to handle sub-packages.

Before I post the repro, my questions are:
1) Does anyone have a work-flow for my situation that doesn't use
Reload, and doesn't involve restarting the app for every edit/fix

2) Can anyone help point me in the right direction to build a
dependable algorithm to cleanly reload all loaded modules? NOTE: I
don't need to maintain anything in memory (i.e. instances, pointers,
etc.) Everything will be initialized again each time. I'm not asking
for code. Just some ideas or pseudo-code would do nicely.

Here is a "simple" repro...

Package Structure:
---------------------------
inheritRepro
__init__.py
baseLib.py
child
__init__.py


__init__.py:
----------------
import sys, os

def reloadModules():
"""
Reload all imported modules that are not on the C: drive.
"""
print "Reloading Python Modules..."

# Iterate over all IMPORTED modules
modules = sys.modules
for modName in modules:
mod = modules[modName]

# Skip None types and other itesm we don't care about
if modName == "__main__" or not hasattr(mod,"__file__"):
continue

# Get the drive and skip anything on the C: drive
drive = os.path.splitdrive(mod.__file__)[0]
if drive != "C:":
reload(mod)
print "Reloaded %s" % mod


baseLib.py:
---------------
class BaseClassA(object):
pass

class BaseClassB(BaseClassA):
def __init__(self):
super(BaseClassB, self).__init__()


child.__init__.py:
------------------------
import inheritRepro.baseLib as baseLib

class ChildClass(baseLib.BaseClassB):
def __init__(self):
super(ChildClass, self).__init__()


RUN:
-------Traceback (most recent call last):
File "<console>", line 0, in <module>
File "\\nas6tb\PROJECTS\tech\users\rafe.sacks\python\inheritRepro
\child\__init__.py", line 5, in __init__
super(ChildClass, self).__init__()
File "\\nas6tb\PROJECTS\tech\users\rafe.sacks\python\inheritRepro
\baseLib.py", line 6, in __init__
super(BaseClassB, self).__init__()
TypeError: super(type, obj): obj must be an instance or subtype of
type


NOTE: this works if I don't use a sub-package for 'child' (child.py
instead). Is it overly simple to assume reloading by file structure
might work?

Right now I'm getting around this by reload()-ing the right package
after running reloadModules() if an error occurs. It's a bit
frustrating that it cost me two days of work before I realized it was
reload() causing this error and not super() or some other unknown-to-
me inheritance/package structure problem. I rebuilt my code module by
module until I noticed, quite by accident, that the thing worked once
and then never again! ....oh well, these are the joys of learning the
hard way.

I know this was a long one. If you made it this far, thanks for
reading,

- Rafe

 

[Gabriel Genellina]

I've tried to use reload with a very simple algorithm. Simply run
through every imported module, ignoring anything that is "None" or on
the C: drive (all of our python is on a network drive so this hack
works for me for now) and reload() it. I've come to realize that this
isn't near intelligent enough to handle sub-packages.

Package Structure:
---------------------------
inheritRepro
__init__.py
baseLib.py
child
__init__.py

To summarize your code: inheritRepro.__init__.py only contains the reload
support; baselib is an independent module; child/__init__.py imports and
uses baseLib.

# Iterate over all IMPORTED modules
modules = sys.modules
for modName in modules:
mod = modules[modName]

(note: sys.modules could change along the iteration; I'd iterate this way
instead)

modules = sys.modules
for modName in modules.keys():
mod = modules.get(modName)

Output:
Reloading Python Modules...
Reloaded <module 'inheritRepro' from
'C:\TEMP\problema_con_reload\inheritRepro\_
_init__.pyc'>
Reloaded <module 'inheritRepro.child' from
'C:\TEMP\problema_con_reload\inheritR
epro\child\__init__.pyc'>
Reloaded <module 'inheritRepro.baseLib' from
'C:\TEMP\problema_con_reload\inheri
tRepro\baseLib.pyc'>

Note that child is reloaded *before* baseLib - when child imports baseLib,
it still finds the old module. One should reload baseLib before reloading
child, in that case it works fine.

This dependency between modules, applied to all modules in your project,
defines a "dependency graph". In some cases, one can define a partial
ordering of its nodes, such that no module depends on any other module
*after* it (it may depend only on modules *before* it). Look for
"topological sort".

Doing that in the generic case is not easy. If you *know* your
dependencies, reload the modules in the right order by hand.

NOTE: this works if I don't use a sub-package for 'child' (child.py
instead). Is it overly simple to assume reloading by file structure
might work?

You're iterating over sys.modules in whatever ordering the keys are in the
dictionary; using other module names may yield the "right" ordering just
by chance.

 

[Rafe]

En Mon, 08 Sep 2008 05:37:24 -0300,Rafe<[email protected]> escribió:
...
This dependency between modules, applied to all modules in your project,
defines a "dependency graph". In some cases, one can define a partial
ordering of its nodes, such that no module depends on any other module
*after* it (it may depend only on modules *before* it). Look for
"topological sort".

Doing that in the generic case is not easy. If you *know* your
dependencies, reload the modules in the right order by hand.
...

Hi Gabriel,

Thank you for clarifying and re-presenting parts of my case. I
appreciate the time.

I was hoping there would be a way to just wipe out the module cache
and let it get rebuilt by executing my code (since I'm not using
reload as part of my program, but rather, to test it in an environment
where I cannot restart the Python session).

I have been keeping a diagram of my module inheritance to make sure it
is as clean as possible, so I could just right a list of reloads as
you suggest. However, one of the sub-packages is designed to allow
users to add more modules. Because these get dynamically imported, a
guess I could add an argument to the reload function to allow a user
to give the 'add-on' module they are working on... so much work just
to get a clean environment...

Separate of my program, I was really hoping to write a generic reload
tool for anyone developing in the same application as I am. I just
don't see a way to trace import dependencies in systems which include
dynamic imports. Any ideas?


Thanks again,

- Rafe

 

[Gabriel Genellina]

I was hoping there would be a way to just wipe out the module cache
and let it get rebuilt by executing my code (since I'm not using
reload as part of my program, but rather, to test it in an environment
where I cannot restart the Python session).

Ok, I think the following sequence *might* work:

- replace the __import__ and reload builtins with a custom callable
object. This way you can hook into any import attempt. The object should
keep a list of already reloaded modules. When a module is imported: if it
is already in the list, just delegate to the original __import__; if it is
not in the list, locate the module in sys.modules and reload it.

- iterate over sys.modules and reload the desired modules, as you did in
your previous attempt.

- restore the original __import__ function.

This way, you effectively transform any import statement into a recursive
reload (for the first time); subsequent imports of the same module behave
as usual. This may work for you, or perhaps not, or it may screw all your
running environment up, or even cause the next global thermonuclear war...
(I hope not!)

Note: some modules don't work well with reload(). A common case: global
mutable values, like a list of objects which starts empty:
my_list = []
To make it more "reload friendly", use this:
try: my_list
except NameError: my_list = []
(this way the list will keep its previous values).

The example below shows how to hook into the import mechanism - it just
prints the module being imported. Implementing the functionality outlined
above is left as an exercise to the reader

py> class ImportHook(object):
.... _orig_import = None
.... #
.... def __call__(self, name, globals={}, locals={}, fromlist=[],
level=-1):
.... if fromlist:
.... print "-> from %s import %s" % (name, ','.join(fromlist))
.... else:
.... print "-> import %s" % name
.... return self._orig_import(name, globals, locals, fromlist,
level)
.... #
.... def hook(self):
.... import __builtin__
.... self._orig_import = __builtin__.__import__
.... __builtin__.__import__ = self
.... #
.... def unhook(self):
.... assert self._orig_import is not None, "unhook called with no
previous hook"
.... import __builtin__
.... __builtin__.__import__ = self._orig_import
.... del self._orig_import
.... #
.... # support the "with" statement
.... def __enter__(self):
.... self.hook()
.... return self
.... #
.... def __exit__(self, type, value, tb):
.... self.unhook()
....
py>
py> ih = ImportHook()
py> ih.hook()
py> import htmllib
-> import htmllib
-> import sgmllib
-> import markupbase
-> import re
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import re
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> from formatter import AS_IS
-> import sys
-> from htmlentitydefs import entitydefs
py> reload(htmllib)
-> import sgmllib
-> from formatter import AS_IS
-> from htmlentitydefs import entitydefs
<module 'htmllib' from 'C:\apps\Python25\lib\htmllib.pyc'>
py> ih.unhook()
-> import __builtin__
py> import htmllib
py>

I have been keeping a diagram of my module inheritance to make sure it
is as clean as possible, so I could just right a list of reloads as
you suggest. However, one of the sub-packages is designed to allow
users to add more modules. Because these get dynamically imported, a
guess I could add an argument to the reload function to allow a user
to give the 'add-on' module they are working on... so much work just
to get a clean environment...

Separate of my program, I was really hoping to write a generic reload
tool for anyone developing in the same application as I am. I just
don't see a way to trace import dependencies in systems which include
dynamic imports. Any ideas?

You may adapt the example above.
Good luck!

 

[Rafe]

En Wed, 10 Sep 2008 00:56:43 -0300,Rafe<[email protected]> escribió:

I was hoping there would be a way to just wipe out the module cache
and let it get rebuilt by executing my code (since I'm not using
reload as part of my program, but rather, to test it in an environment
where I cannot restart the Python session).

Ok, I think the following sequence *might* work:

- replace the __import__ and reload builtins with a custom callable  
object. This way you can hook into any import attempt. The object should  
keep a list of already reloaded modules. When a module is imported: if it  
is already in the list, just delegate to the original __import__; if it is  
not in the list, locate the module in sys.modules and reload it.

- iterate over sys.modules and reload the desired modules, as you did in  
your previous attempt.

- restore the original __import__ function.

This way, you effectively transform any import statement into a recursive  
reload (for the first time); subsequent imports of the same module behave  
as usual. This may work for you, or perhaps not, or it may screw all your  
running environment up, or even cause the next global thermonuclear war....  
(I hope not!)

Note: some modules don't work well with reload(). A common case: global  
mutable values, like a list of objects which starts empty:
        my_list = []
To make it more "reload friendly", use this:
        try: my_list
        except NameError: my_list = []
(this way the list will keep its previous values).

The example below shows how to hook into the import mechanism - it just  
prints the module being imported. Implementing the functionality outlined  
above is left as an exercise to the reader

py> class ImportHook(object):
...     _orig_import = None
... #
...     def __call__(self, name, globals={}, locals={}, fromlist=[],  
level=-1):
...         if fromlist:
...             print "-> from %s import %s" % (name, ','.join(fromlist))
...         else:
...             print "-> import %s" % name
...         return self._orig_import(name, globals, locals, fromlist,  
level)
... #
...     def hook(self):
...         import __builtin__
...         self._orig_import = __builtin__.__import__
...         __builtin__.__import__ = self
... #
...     def unhook(self):
...         assert self._orig_import is not None, "unhook called with no  
previous hook"
...         import __builtin__
...         __builtin__.__import__ = self._orig_import
...         del self._orig_import
... #
...     # support the "with" statement
...     def __enter__(self):
...         self.hook()
...         return self
... #
...     def __exit__(self, type, value, tb):
...         self.unhook()
...
py>
py> ih = ImportHook()
py> ih.hook()
py> import htmllib
-> import htmllib
-> import sgmllib
-> import markupbase
-> import re
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import re
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> import sre_parse
-> from formatter import AS_IS
-> import sys
-> from htmlentitydefs import entitydefs
py> reload(htmllib)
-> import sgmllib
-> from formatter import AS_IS
-> from htmlentitydefs import entitydefs
<module 'htmllib' from 'C:\apps\Python25\lib\htmllib.pyc'>
py> ih.unhook()
-> import __builtin__
py> import htmllib
py>

I have been keeping a diagram of my module inheritance to make sure it
is as clean as possible, so I could just right a list of reloads as
you suggest. However, one of the sub-packages is designed to allow
users to add more modules. Because these get dynamically imported, a
guess I could add an argument to the reload function to allow a user
to give the 'add-on' module they are working on... so much work just
to get a clean environment...

Separate of my program, I was really hoping to write a generic reload
tool for anyone developing in the same application as I am. I just
don't see a way to trace import dependencies in systems which include
dynamic imports. Any ideas?

You may adapt the example above.
Good luck!

Thanks again Gabriel! I'll see if I can take another swing at this
when I catch up to my milestones . I'll post anything useful.

- Rafe