Automatic import PEP

[Connelly Barnes]

Hi,

I wrote the 'autoimp' module [1], which allows you to import lazy modules:

from autoimp import * (Import lazy wrapper objects around all modules; "lazy
modules" will turn into normal modules when an attribute
is first accessed with getattr()).
from autoimp import A, B (Import specific lazy module wrapper objects).

The main point of autoimp is to make usage of the interactive Python prompt
more productive by including "from autoimp import *" in the PYTHONSTARTUP file.
Thus one can use:

>>> urllib2.urlopen('http://www.google.com').read()
>>> Image.open('test.bmp')
>>> pylab.plot([1,2],[3,4])
>>> scipy.linalg.eig([[1,2],[3,4]])
>>> ...

One can thenceforward use the interactive prompt without using the "import"
statement at all. In practice, I have found that there is little overhead in time
and memory for the initial import of all lazy modules (assuming all of the Python
module paths are on the local disk, and not on a network).

I have also found "autoimp" useful in writing normal Python applications; one can
use "from autoimp import *" at the top of a Python source file and thus have all
Python modules available to that source file, which results in increased
efficiency of writing the program with the downside that one has to think more
carefully about global variables to avoid name conflict bugs.

Now because this feature is useful, I thought of writing a PEP to add the
"lazy import" functionality to the Python interpreter.

Specifically:

lazily import *
lazily import A, B, C

This would involve the introduction of a new keyword, such as "lazily".

The PEP should include extensions to the builtin Python __import__ (__lazilyimport__ ?)
and the imp and zipimport modules so that the new "lazy import" feature can be
fully utilized by "dynamic" Python code.

Closely related to the "autoimp" module is Philip Eby's Importing module [2]. The Importing
module has the ability to lazily import a single module as well as a number of other features,
so Philip may wish to be involved, and features from Importing may be used in the PEP if
desired.

Opinions?

[1]. http://cheeseshop.python.org/pypi/autoimp/1.0.2
[2]. http://peak.telecommunity.com/DevCenter/Importing


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

 

[Michel Claveau]

Hi!

+1

 

[Andrea]

Opinions?

Great

 

[Sybren Stuvel]

Connelly Barnes enlightened us with:

I wrote the 'autoimp' module [1], which allows you to import lazy modules:

The main point of autoimp is to make usage of the interactive Python
prompt more productive by including "from autoimp import *" in the
PYTHONSTARTUP file.

Sounds like a great idea.

I have also found "autoimp" useful in writing normal Python
applications; one can use "from autoimp import *" at the top of a
Python source file and thus have all Python modules available to
that source file

That sounds like a source of new bugs to me. Besides that, you have no
overview at all of which modules are used by the program. It also
prevents automated checking of that using pylint and friends.

Sybren

 

[Lawrence D'Oliveiro]

The main point of autoimp is to make usage of the interactive Python
prompt more productive by including "from autoimp import *" in the
PYTHONSTARTUP file.

The main problem I have with your idea is that any kind of import statement
with wildcards in it automatically starts my bogosity meter flashing
its "RED ALERT" sign and clanging all its alarm bells.

 

[Robert Kern]

The main problem I have with your idea is that any kind of import statement
with wildcards in it automatically starts my bogosity meter flashing
its "RED ALERT" sign and clanging all its alarm bells.

"from ... import *" was *designed* for interactive use, which is exactly what
Connelly is suggesting. It's not a bad thing in that context.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Saizan]

I think this is obviously great in interactive mode and would also be
very good in the early stages of developing if you have several sources
files. A little error prone maybe, and should be avoided in
"production" code I suppose. (I would like to track each name exactly,
on each installation of python that I may move to)
BTW what would the benefit of the form "lazily import A, B"? If you
name the modules why not import them directly? Maybe you are not sure
you would need them, but I don't think that the overhead of importing
them should matter..
Anyhow, is it just for completion maybe?

 

[Robert Kern]

BTW what would the benefit of the form "lazily import A, B"? If you
name the modules why not import them directly? Maybe you are not sure
you would need them, but I don't think that the overhead of importing
them should matter..

It's primarily useful for large packages. scipy used to have a mechanism of
postponed imports because loading all of the extension modules in all of scipy's
subpackages takes a fair bit of time. It especially hurts when you only wanted
one. However, it is quite convenient to simply write "import scipy;
scipy.linalg.eig(matrix)".

Nonetheless, we removed that feature and stopped importing all of the
subpackages because the postponed import mechanism interfered with a number of
other things (namely, getting docstrings through IPython's ? operator and making
py2exe apps that use scipy). Having a third-party library do the same thing
makes interactive use nice without burdening scipy with the problems.

So, "lazily import A" would make sense if A was a large package where all of the
goodies are buried in A.C.D and A.E, A.X, A.Z, etc. You *do* know that you need
A, but you may not know what you need underneath it.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Lawrence D'Oliveiro]

"from ... import *" was *designed* for interactive use, which is exactly
what Connelly is suggesting. It's not a bad thing in that context.

But there is nothing in the language that constrains its use to that
context, is there?

Clang, clang...

 

[Robert Kern]

But there is nothing in the language that constrains its use to that
context, is there?

No. What's your point?

Connelly Barnes states that the main point of autoimp is to make usage of the
interactive prompt better by including "from autoimp import *" into a
PYTHONSTARTUP file. That file is only executed for interactive sessions. He's
not suggesting that people do "from autoimp import *" in their modules.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Dan Bishop]

Hi,

I wrote the 'autoimp' module [1], which allows you to import lazy modules:

from autoimp import * (Import lazy wrapper objects around all modules; "lazy
modules" will turn into normal modules when an attribute
is first accessed with getattr()).
from autoimp import A, B (Import specific lazy module wrapper objects).

The main point of autoimp is to make usage of the interactive Python prompt
more productive by including "from autoimp import *" in the PYTHONSTARTUP file.

And it does. Gets rid of "oops, I forgot to import that module"
moments without cluttering my $PYTHONSTARTUP file with imports. +1.

My only complaint is that it breaks globals().

 

[Georg Brandl]

Hi,

I wrote the 'autoimp' module [1], which allows you to import lazy modules:

from autoimp import * (Import lazy wrapper objects around all modules; "lazy
modules" will turn into normal modules when an attribute
is first accessed with getattr()).
from autoimp import A, B (Import specific lazy module wrapper objects).

The main point of autoimp is to make usage of the interactive Python prompt
more productive by including "from autoimp import *" in the PYTHONSTARTUP file.

And it does. Gets rid of "oops, I forgot to import that module"
moments without cluttering my $PYTHONSTARTUP file with imports. +1.

My only complaint is that it breaks globals().

And startup takes quite long the first time, because a list of all available
modules must be gathered.

To work around that, one can either use a special importing "lib" object,
defined like that:

class _lib:
def __getattr__(self, name):
return __import__(name)
lib = _lib()


or modify the globals() to automatically look up modules on KeyError, like this
(put into PYTHONSTARTUP file):


import sys, code

class LazyImpDict(dict):
def __getitem__(self, name):
try:
return dict.__getitem__(self, name)
except KeyError:
exc = sys.exc_info()
try:
return __import__(name)
except ImportError:
raise exc[0], exc[1], exc[2]

d = LazyImpDict()
code.interact(banner='', local=d)
sys.exit()


Of course, this is not perfect as it may break quite a lot of things,
I haven't tested it thoroughly.

Georg