How to debug this import problem?

[Iwan Vosloo]

Hi there,

We have a rather complicated program which does a bit of os.chdir and
sys.path manipulations. In between all of this, it imports the decimal
module several times.

However, it imports a new instance of decimal sometimes. (Which is a
problem, since a decimal.Decimal (imported at point A) now is not the
same as that imported somewhere else.

In trying to figure out what's happening, we've changed the code in
decimal to print out id(sys) when decimal gets imported. This also
gives back different values at different times. My suspicion is that
when importing python tries to check sys.modules - but each time sys is
a different sys already.

Any ideas of how we can gather more data to find out exactly what causes
this? The path manipulations is a suspect, but unfortunately we cannot
remove it. If we can trace the problem from the import end though, we
may be able to isolate the exact one which is the problem.

This code incidentally also runs in a virtualenv environment AND uses
setuptools. None of these complications can be removed...

Regards
- Iwan

 

[Diez B. Roggisch]

Hi there,

We have a rather complicated program which does a bit of os.chdir and
sys.path manipulations. In between all of this, it imports the decimal
module several times.

However, it imports a new instance of decimal sometimes. (Which is a
problem, since a decimal.Decimal (imported at point A) now is not the
same as that imported somewhere else.

In trying to figure out what's happening, we've changed the code in
decimal to print out id(sys) when decimal gets imported. This also
gives back different values at different times. My suspicion is that
when importing python tries to check sys.modules - but each time sys is
a different sys already.

Any ideas of how we can gather more data to find out exactly what causes
this? The path manipulations is a suspect, but unfortunately we cannot
remove it. If we can trace the problem from the import end though, we
may be able to isolate the exact one which is the problem.

This code incidentally also runs in a virtualenv environment AND uses
setuptools. None of these complications can be removed...

Try putting an "import pdb; pdb.set_trace()" on top of the decimal module.
Then you can see whenever it gets imported new, and with "bt" get a
backtrace, and you can inspect (and thus eventually compare) the various
states of sys.path and sys.modules.

Diez

 

[Iwan]

Hi Scott, Diez,

Try putting an "import pdb; pdb.set_trace()" on top of the decimal module.

[snip]
You can also run Python with the "-v" or "-vv" flags to get output
about exactly what files are getting imported from the file
system and when.  "-v" iswhat it got, "-vv" is everywhere it tries.

Thanks, yes we did do the pdb.set_trace(), but everything looked fine
(ie, the paths used to import each module from was the same each time)
in there except for the fact that id(sys) and id(decimal) were
different each time decimal got imported. Python -vv also reveals
that the paths used to import the modules from are always the same
paths and correct.

However, on the stack trace, there is another strangeness:

We have a little script that loads the tests that have to be run.
This loading happens 3 times, and the module is imported 3 times -
each time as a result of loading tests. This happens with the
following line:

module = __import__(moduleName, globals(), locals(), [className])

I was wondering whether this could have something to do with it?

Regards
- Iwan

 

[Iwan]

Mmm, we solved half of the cause of this one.

Test runs are kicked off via setuptools's test command. But this
happens programmatically, and successively in one process. But
setuptools's test command clears all modules imported during a test
run from sys.modules - hence it is intended that modules would be re-
imported.

Why certain of our code managed to still get to the classes contained
in prior imports is still a mystery though.

-Iwan