surprising behaviour of global dictionaries

[Michele Simionato]

I have the following module implementing a registry of functions with a decorator:

$ cat x.py
registry = {} # global dictionary

def dec(func):
registry[func.__name__] = func
print registry, id(registry)
return func

if __name__ == '__main__':
import xlib
print registry, id(registry)

The library xlib just defines two dummy functions:

$ cat xlib.py
from x import dec

@dec
def f1():
pass

@dec
def f2():
pass

Then I get the following output:

$ python x.py
{'f1': <function f1 at 0x7f7bce0cd668>} 27920352
{'f1': <function f1 at 0x7f7bce0cd668>, 'f2': <function f2 at 0x7f7bce0cd6e0>} 27920352
{} 27395472

This is surprising since I would expect to have a single global dictionary, not two: how comes the registry inside the ``if __name__ == '__main__'`` block is different from the one seen in the library?

This is python 2.7.3 on Ubuntu.

 

[Peter Otten]

I have the following module implementing a registry of functions with a
decorator:

$ cat x.py
registry = {} # global dictionary

def dec(func):
registry[func.__name__] = func
print registry, id(registry)
return func

if __name__ == '__main__':
import xlib
print registry, id(registry)

The library xlib just defines two dummy functions:

$ cat xlib.py
from x import dec

@dec
def f1():
pass

@dec
def f2():
pass

Then I get the following output:

$ python x.py
{'f1': <function f1 at 0x7f7bce0cd668>} 27920352
{'f1': <function f1 at 0x7f7bce0cd668>, 'f2': <function f2 at
{0x7f7bce0cd6e0>} 27920352 } 27395472

This is surprising since I would expect to have a single global
dictionary, not two: how comes the registry inside the ``if __name__ ==
'__main__'`` block is different from the one seen in the library?

This is python 2.7.3 on Ubuntu.

Welcome to python -- this is a trap every newbie falls into

Seriously, you shouldn't use the main script as a library; it is put into
the sys.modules cache under the "__main__" key. Subsequent imports under its
real name will not find that name in the cache and import another instance
of the module, with puzzling effects, like

$ cat x.py
import x
class A: pass
a = A()
assert isinstance(a, x.A)

$ python x.py
Traceback (most recent call last):
File "x.py", line 4, in <module>
assert isinstance(a, x.A)
AssertionError
$

 

[Michele Simionato]

Seriously, you shouldn't use the main script as a library; it is put into

the sys.modules cache under the "__main__" key. Subsequent imports under its

real name will not find that name in the cache and import another instance

of the module, with puzzling effects

Actually I usually never use the main script as a library, this is why I never experience this puzzling behavior before. But now it is clear, thanks.

 

[Michele Simionato]

Seriously, you shouldn't use the main script as a library; it is put into

the sys.modules cache under the "__main__" key. Subsequent imports under its

real name will not find that name in the cache and import another instance

of the module, with puzzling effects

Actually I usually never use the main script as a library, this is why I never experience this puzzling behavior before. But now it is clear, thanks.

 

[Grant Edwards]

Welcome to python -- this is a trap every newbie falls into

Seriously, you shouldn't use the main script as a library;

There must be something wrong with me. It never even occurred to me
to try to import a file from within that same file. I don't think
I've ever even heard of that before...

 

[Dave Angel]

Actually I usually never use the main script as a library, this is why I never experience this puzzling behavior before. But now it is clear, thanks.

More generally, you should arrange your imports so that there are no
cycles. If module/script "a" imports "b", directly or indirectly, "b"
should not try to import "a." Move the common code to a third place,
and import it from both "a" and from "b".

The other symptoms you can get are more subtle than this one, but just
as surprising.

 

[Peter Otten]

There must be something wrong with me. It never even occurred to me
to try to import a file from within that same file.

It is typically done in two steps:

(1) Write module x, use it in module y.
(2) For convenience add "if __name__ == '__main__'" and import module y.

Hilarity ensues.

I don't think I've ever even heard of that before...

As I was poking fun at Michele who really is an expert and likely knows more
about Python than I do I may have exaggerated a bit

But it does come up, see

"Singleton implementation problems"
http://mail.python.org/pipermail/python-list/2008-July/470770.html

"Dynamically declared shared constant/variable imported twice problem"
http://mail.python.org/pipermail/python-list/2009-May/535619.html

There are more, but I don't know a convenient way to find the posts.
Related problems with reload() or paths into a package occur even more
frequently.