Import in a separate thread

[cyberco]

I want to import a long list of modules in a separate thread to speed
things up. How can I make the modules imported in that separate thread
accessible outside the method?

===========================
import os

# import rest in a separate thread
def importRest():
import audio
import socket
thread.start_new_thread(importRest,())

# audio.somemethod() would fail here
===========================

 

[Paul Rubin]

I want to import a long list of modules in a separate thread to speed
things up. How can I make the modules imported in that separate thread
accessible outside the method?

It won't speed things up unless the main thread is waiting for user
input during those imports, or something like that. Threads in Python
don't really run in parallel.

# import rest in a separate thread
def importRest():
import audio
import socket
thread.start_new_thread(importRest,())

# audio.somemethod() would fail here

Here's one way:

def importRest():
global audio
import audio as local_audio
audio = local_audio
# etc.

There's surely other ways that are better.

 

[Peter Hansen]

I want to import a long list of modules in a separate thread to speed
things up. How can I make the modules imported in that separate thread
accessible outside the method?

Sounds like premature optimization. Speed "things" up? What things?
How long is it taking now to load the modules you are loading? Even the
wxPython demo takes only a few seconds to load on a decent machine, and
that's loading a *heck* of a lot of stuff. (And it takes the
conventional approach to this issue too, which is to display a splash
screen while things load.)

-Peter

 

[Paul Rubin]

Sounds like premature optimization. Speed "things" up? What things?
How long is it taking now to load the modules you are loading? Even
the wxPython demo takes only a few seconds to load on a decent
machine, and that's loading a *heck* of a lot of stuff.

I know that whenever I start IDLE and it's not already cached, it
takes a significant amount of time, and that doesn't even use
wxPython.

 

[Peter Hansen]

I know that whenever I start IDLE and it's not already cached, it
takes a significant amount of time, and that doesn't even use
wxPython.

I don't dispute that there are in the world programs that take too long
to start by even objective measures. I don't dispute that you, Paul,
might consider IDLE to take too long to start. I don't think I was
responding to you, however, but to the OP, who hasn't described anything
about his situation except that he believes importing modules in a
separate thread might "speed things up". To me, that sounded like the
words of someone who not only might not be aware of the excellent advice
regarding premature optimization, but who might also not suffer from a
significant startup delay but who simply imagined that perhaps using a
"trick" to import things differently might make something run faster.
Understanding why he wants to do this is important to knowing what the
best approach is, whether it's to say "don't do it" or to point to
alternatives (such as a splash screen) which others have found to solve
the problem.

Just because it appears some people prematurely cry "premature
optimization" doesn't mean that it's an invalid thing to ask a poster to
consider.

-Peter

 

[cyberco]

Well, it is for the python implementation for Nokia Series 60 phones,
and loading lots of modules in such constrained environments can
certainly slow things down. The splashscreen idea is what I want to do,
but that requires the loading to continue in a background thread.

 

[Grant Edwards]

Well, it is for the python implementation for Nokia Series 60 phones,
and loading lots of modules in such constrained environments can
certainly slow things down. The splashscreen idea is what I want to do,
but that requires the loading to continue in a background thread.

Ah, so you're _not_ trying to speed things up. You just want
to re-arrange the order in which things happen. While threads
rarely help the former, they do work well for the latter.

 

[Kent Johnson]

I want to import a long list of modules in a separate thread to speed
things up. How can I make the modules imported in that separate thread
accessible outside the method?

===========================
import os

# import rest in a separate thread
def importRest():
import audio
import socket
thread.start_new_thread(importRest,())

# audio.somemethod() would fail here
===========================

Just import the modules again when you need to use them - modules are
cached, only the initial import is expensive.

Of course this won't help if you don't have something else to do (like
wait for the user) while the threaded imports happen.

Another helpful technique is to put imports inside the functions that
need them. If every module imports the modules it needs at global scope,
then when the first module loads it will trigger importing of the whole
program and all needed library modules. If this is expensive, you can
break up the loads by putting key imports into functions instead of at
global scope.

Note to the doubters - I have used these techniques in Jython programs,
where importing java packages can be noticably slow.

Kent