'reload M' doesn't update 'from M inport *'

[Frederic Rentsch]

I develop in an IDLE window.

Module M says 'from service import *'.
Next I correct a mistake in function 'service.f'.
Now 'service.f' works fine.

I do 'reload (service); reload (M)'.
The function 'M.f' still misbehaves.

'print inspect.getsource (service.f)' and
'print inspect.getsource (M.f)' shows the same
corrected code.

'print service.f' and 'print M.f' show different ids.

So I do 'del M; reload (M)'. Nothing changes.

I delete M again and run gc.collect () to really
clean house. I reload M again and still nothing changes.
The id of the reloaded function 'M.f' is still the
same as it was before the purge and so M.f still isn't
fixed.

I know I have more radical options, such as starting
a new IDLE window. That would save me time, but
I'd like to take the opportunity to understand what
is happening. Surely someone out there knows.

Frederic

 

[Aahz]

Module M says 'from service import *'.
Next I correct a mistake in function 'service.f'.
Now 'service.f' works fine.

I do 'reload (service); reload (M)'.
The function 'M.f' still misbehaves.
Absolutely!

'print inspect.getsource (service.f)' and
'print inspect.getsource (M.f)' shows the same
corrected code.

'print service.f' and 'print M.f' show different ids.

So I do 'del M; reload (M)'. Nothing changes.

I delete M again and run gc.collect () to really clean house. I reload
M again and still nothing changes. The id of the reloaded function
'M.f' is still the same as it was before the purge and so M.f still
isn't fixed.

I know I have more radical options, such as starting a new IDLE
window. That would save me time, but I'd like to take the opportunity
to understand what is happening. Surely someone out there knows.

Take a look at sys.modules to get a better idea of what's happening.
(Maybe someone else will have time to write a longer answer.)

But really, relying on reload() is foolish in the general case because
it's nearly impossible to track down every single reference.

 

[Steven D'Aprano]

I develop in an IDLE window.

Module M says 'from service import *'. Next I correct a mistake in
function 'service.f'. Now 'service.f' works fine.

from service import *

should be considered advanced functionality that is discouraged unless
you really know what you are doing, precisely for the problems you are
experiencing. You should try to avoid it.

But putting that aside, if you have done "from service import *" in
module m, where are you getting "service.f" from? The only way that is
possible is if you ALSO say "import service".

I do 'reload (service); reload (M)'.
The function 'M.f' still misbehaves.

'print inspect.getsource (service.f)' and 'print inspect.getsource
(M.f)' shows the same corrected code.

inspect.getsource always looks at the source code on disk, no matter what
the byte code in memory actually says.

'print service.f' and 'print M.f' show different ids.

So I do 'del M; reload (M)'. Nothing changes.

I delete M again and run gc.collect () to really clean house. I reload M
again and still nothing changes. The id of the reloaded function 'M.f'
is still the same as it was before the purge and so M.f still isn't
fixed.

I know I have more radical options, such as starting a new IDLE window.
That would save me time, but I'd like to take the opportunity to
understand what is happening. Surely someone out there knows.

Yes. You have to understand importing. Let's start with the simple:

import m

In *very* simplified pseudo-code, this does:

look for module m in the global cache
if not there, then:
search for m.py
compile it to a Module object
put the Module object in the cache
create a new name "m" in the local namespace
set the name "m" to the Module object in the cache

Now let's compare it to:

from m import f

look for module m in the global cache
if not there, then:
search for m.py
compile it to a Module object
put the Module object in the cache
look for object named "f" in the Module object
create a new name "f" in the local namespace
set the name "f" to cached object

The important thing to notice is the the name "f" is a local variable. It
doesn't, and can't, remember that it comes from module m. Reloading m
can't do anything to f, because the connection is lost.

Now consider that the object "f" that came from m was itself imported
from another module, "service". Reloading service doesn't help, because
m.f doesn't know it came from service. Reloading m doesn't help, because
all that does is run "from service import f" again, and that just fetches
f from the global cache.

The simplest, easiest way of dealing with this is not to have to deal
with it: don't use "from service import f", and ESPECIALLY don't use
"from service import *". Always use fully-qualified importing:

import service
service.f

Now "reload(service)" should do what you expect.

The other way is not to bother with reload. It's not very powerful, only
good for the simplest use in the interactive interpreter. Just exit the
interpreter and restart it.

 

[Frederic Rentsch]

from service import *

should be considered advanced functionality that is discouraged unless
you really know what you are doing, precisely for the problems you are
experiencing. You should try to avoid it.

But putting that aside, if you have done "from service import *" in
module m, where are you getting "service.f" from? The only way that is
possible is if you ALSO say "import service".



inspect.getsource always looks at the source code on disk, no matter what
the byte code in memory actually says.


Yes. You have to understand importing. Let's start with the simple:

import m

In *very* simplified pseudo-code, this does:

look for module m in the global cache
if not there, then:
search for m.py
compile it to a Module object
put the Module object in the cache
create a new name "m" in the local namespace
set the name "m" to the Module object in the cache

Now let's compare it to:

from m import f

look for module m in the global cache
if not there, then:
search for m.py
compile it to a Module object
put the Module object in the cache
look for object named "f" in the Module object
create a new name "f" in the local namespace
set the name "f" to cached object

The important thing to notice is the the name "f" is a local variable. It
doesn't, and can't, remember that it comes from module m. Reloading m
can't do anything to f, because the connection is lost.

Now consider that the object "f" that came from m was itself imported
from another module, "service". Reloading service doesn't help, because
m.f doesn't know it came from service. Reloading m doesn't help, because
all that does is run "from service import f" again, and that just fetches
f from the global cache.

The simplest, easiest way of dealing with this is not to have to deal
with it: don't use "from service import f", and ESPECIALLY don't use
"from service import *". Always use fully-qualified importing:

import service
service.f

Now "reload(service)" should do what you expect.

The other way is not to bother with reload. It's not very powerful, only
good for the simplest use in the interactive interpreter. Just exit the
interpreter and restart it.

Thank you very much for your excellent explanation!
I must say that I haven't been using the "from soandso import ..."
formula at all. I thought it might expose names to collision, and why
should I assume the responsibility if I can avoid the problem altogether
using explicit names. If I used the, shall we say, "direct import" this
time it was in an effort to develop a more extensive program. I thought
if a module grows beyond a size that's comfortable to edit, I could just
move select segments to separate files and replace the vacancy with
"from the_respective_segment_module import *", analogous to "#include"
in C.
The remedy seems to have side-effects that can kill the patient. So
I'll go back to the explicit imports, then. No problem at all.

Thanking you and the other helpers too

Frederic

 

[kedra marbun]

from m import f

look for module m in the global cache
if not there, then:
search for m.py
compile it to a Module object
put the Module object in the cache
look for object named "f" in the Module object agree

create a new name "f" in the local namespace
set the name "f" to cached object

strongly agree

The important thing to notice is the the name "f" is a local variable. It
doesn't, and can't, remember that it comes from module m. Reloading m
can't do anything to f, because the connection is lost.

disagree with 2nd stmt. local 'f' does *remember* (if that's the right
word) where it comes from, because it points to the original obj, as
you said: 'set the name "f" to cached object'

py0.py
======
def f(): ...

from py0 import *
assert f.__module__ == sys.modules['py0'].__name__
assert f.__globals__ is sys.modules['py0'].__dict__

Now consider that the object "f" that came from m was itself imported
from another module, "service". Reloading service doesn't help, because
m.f doesn't know it came from service. Reloading m doesn't help, because
all that does is run "from service import f" again, and that just fetches
f from the global cache.

disagree with 2nd stmt, partially disagree with 3rd stmt

reloading 'service' partially helps, since it updates the mod obj
pointed by 'service' in global cache. it needs to be followed by
reloading m, then we have m.f points to the new obj. the important
part is the order of reloading mods

l.py
====
def f(): ...

m.py
====
from l import *

at this point, the func obj is referenced by 3 distinct variables with
name 'm'(one in each mod)

assert sys.getrefcount(f) == 4
referrers = gc.get_referrers(f)
mod_dicts = [sys.modules[k].__dict__ for k in sys.modules if k == 'l' or k == 'm' or k == __name__]
for d in mod_dicts: .... referrers.remove(d)
assert len(referrers) == 0
imp.reload(sys.modules['l'])

now the original func obj is ref'ed by 2 vars, the new func obj is
ref'ed by 1 var

imp.reload(sys.modules['m'])
f = sys.modules['m'].f

now the original func obj is ready to be recollected, the new func obj
is ref'ed by 3 vars

The simplest, easiest way of dealing with this is not to have to deal
with it: don't use "from service import f", and ESPECIALLY don't use
"from service import *". Always use fully-qualified importing:

import service
service.f

strongly agree

The other way is not to bother with reload. It's not very powerful, only
good for the simplest use in the interactive interpreter. Just exit the
interpreter and restart it.

set, del, reload are useful when it comes to structure manipulation at
runtime, most langs differentiate it, labeling as metaprogramming, py
smashes the diff in an elegant way (oh flattery, i love it)

as their names say, set & del only bind & unbind obj to var, the obj
to bind must be in its bytecode form, in theory one could do it, maybe
thru modules that are categorized as "Python Language Services" in the
lib manual, but it's not practical (e.g. see the last stmt of
types.CodeType.__doc__). this is where 'reload' steps in, and so the
story goes ...