Define type of 'module' object that is imported

[Zachary Pincus]

Hi folks,

I'm sure this has come up before, but the search terms I've been
using are so non-specific that I can't get any traction on Google.

Here's my question:
I have written a subclass of ModuleType that handles lazy loading of
some slow resources. I would like a given module to be created as an
instance of that particular subclass on import, so that if I do:

import foo
type(foo)

I get <type 'myModule'> instead of <type 'module'>.

Is there any way to effect this? Something like __metaclass__ = ...
but at the beginning of a module instead of at the beginning of a class?

Thanks,

Zach Pincus

Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine

 

[robert]

Hi folks,

I'm sure this has come up before, but the search terms I've been using
are so non-specific that I can't get any traction on Google.

Here's my question:
I have written a subclass of ModuleType that handles lazy loading of
some slow resources. I would like a given module to be created as an
instance of that particular subclass on import, so that if I do:

import foo
type(foo)

I get <type 'myModule'> instead of <type 'module'>.

Is there any way to effect this? Something like __metaclass__ = ... but
at the beginning of a module instead of at the beginning of a class?

Would be interesting to know about the motivation. Is foo just your own
module or "any" module.

* you can replace __import__
* you can set sys.modules['foo']=mymod (in sitecustomize.py ? )
* your module can be a class instance as well in newer pythons (2.2+?);
=> you can set sys.modules['foo']=Foo() # Foo having properties ...
* simply import certain expensive modules only ad-hoc
* maybe you don't need it as module at all, but an instance. or you
avoid pre-computing things in global namespace.
....


robert

---

PS:

In the Python standard lib there are some slow importing monsters. The
worst is now urllib2 needing up to a second to import.
Thats because there is a questionable style of importing all kind of
expensive stuff that _might_ be useful in advance as if we had C-style
compiler #include-s. Yet Python allows
best-amongst-most-programming-languages dynamic modularization of code
by local/late imports. Most time you'll see, that the imported modules
are anyway only needed in very few locations. the pychecker helps.

See rejected:
http://sourceforge.net/tracker/index.php?func=detail&aid=1053150&group_id=5470&atid=305470

Meanwhile the cookielib is also amongst those urllib2 inhabitants - and
its the slowest towards my profile runs. I've regular private patches on
my production python installation to get apps (startup) fluid.

Maybe that rejected sf request should be re-opened, and a request put to
optimize the python lib for late dynamic import's - at least in
locations where (regardign a profiler inspection) it pays off well on
low coding costs ?

 

[Zachary Pincus]

I would like a given module to be created as an

Would be interesting to know about the motivation. Is foo just your
own
module or "any" module.

No, just one particular module that needs to do some slow things.
I'd like for a user to just be able to type
"import foo"
and have the foo module's type be some special type. None of the
options below really allow this to happen with one step. It would be
more like "import lazyFoo; import foo".

What I'm doing now is in 'foo.py' performing surgery a la:
sys.modules[__name__] = MyModuleClass(...)
but that's ugly, and can break the reload mechanism, and causes some
other troubles.
I would like to do:
sys.modules[__name__].__class__ = MyModuleClass
but that fails because the module is "not a heap type" or something.

I have also tried:
sys.modules[__name__].__getattribute__ = MethodType(myGetattr,
sys.modules[__name__], ModuleType)
but the custom __getattribute__ never gets called.

Hmm, oh well.

Zach

* you can replace __import__
* you can set sys.modules['foo']=mymod (in sitecustomize.py ? )
* your module can be a class instance as well in newer pythons (2.2
+?);
=> you can set sys.modules['foo']=Foo() # Foo having
properties ...
* simply import certain expensive modules only ad-hoc
* maybe you don't need it as module at all, but an instance. or you
avoid pre-computing things in global namespace.
...

 

[robert]

No, just one particular module that needs to do some slow things.
I'd like for a user to just be able to type
"import foo"
and have the foo module's type be some special type. None of the
options below really allow this to happen with one step. It would be
more like "import lazyFoo; import foo".

What I'm doing now is in 'foo.py' performing surgery a la:
sys.modules[__name__] = MyModuleClass(...)
but that's ugly, and can break the reload mechanism, and causes some
other troubles.

modules must go to sys.modules. reloading would have to be prevented by
simple test. yet if your module is ready no-one would reload.

yet executing the module for reload in that trick namespace will also do
to a certain degree.

you could also try to replace / update the original modules's __dict__

you still didn't mention the original motivation for that
lazy-moduletype replacement.
If its your module anyway, you could (auto-)rework your functions/class
to do delayed initializations at the end of the module.

something like:

for func in <iter-certain-funcs-in-globals()>:
globals()[func_name]=hook_lazy_init(func)


-robert