pickle unable to load collection

[paragk]

Hi,

I am unable to figure out the cause of python pickle unable to find
the collection module.

I am getting

__import__(module)
ImportError: No module named collections

when I try to load a pickled object.

Details:

Python version:

Python 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit
(Intel)] on win32

import sys
import glob
import collections
import pickle

....

SomeListDict = collections.defaultdict(list)

.... # Populate SomeListDict

DictFile = open (options.saveDict, 'w')
pickle.dump(SomeListDict , DictFile, -1)
DictFile.close()

....

# Try loading the dictionary

LoadDictFile = open(options.loadDict, 'rb')
SomeListDict = pickle.load(LoadDictFile)
LoadDictFile.close()

....
 [CODE END]

The pickle.dump(CallGraphDict, DictFile, -1) works fine. The
dictionary is saved.

The interesting thing is, the dump function is able to find the
collections module.
        try:
            __import__(module)
            mod = sys.modules[module]
            klass = getattr(mod, name)

The above try block succeeds.

module
str: collections

mod
module: <module 'collections' from 'C:\Python26\Lib\collections.pyc'>

klass
type: <type 'collections.defaultdict'>


The SomeListDict = pickle.load(LoadDictFile) fails with

    __import__(module)
ImportError: No module named collections


The sequence of calls is:

    def load_global(self):
        module = self.readline()[:-1]
        name = self.readline()[:-1]
        klass = self.find_class(module, name)
        self.append(klass)
    dispatch[GLOBAL] = load_global

Variables:

module
str: collections

name
str: defaultdict


The code where the exception occurs is:

    def find_class(self, module, name):
        # Subclasses may override this
        __import__(module)
        mod = sys.modules[module]
        klass = getattr(mod, name)
        return klass


From the above analysis, clearly the collection module exists, since
the dump works.

What am I missing?

 

[Steven D'Aprano]

Hi,

I am unable to figure out the cause of python pickle unable to find the
collection module.

I am getting

__import__(module)
ImportError: No module named collections

What is the actual value of module? Here's an obvious failure mode:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named collections

module
str: collections

Try printing repr(str) to see what's actually there.


It may also help if you post a copy-and-paste of the complete traceback.

 

[Peter Otten]

Hi,

I am unable to figure out the cause of python pickle unable to find
the collection module.

I am getting

__import__(module)
ImportError: No module named collections

when I try to load a pickled object.

Details:

Python version:

Python 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit
(Intel)] on win32

import sys
import glob
import collections
import pickle

...

SomeListDict = collections.defaultdict(list)

... # Populate SomeListDict

DictFile = open (options.saveDict, 'w')
pickle.dump(SomeListDict , DictFile, -1)
DictFile.close() 
[...]

From the above analysis, clearly the collection module exists, since
the dump works.

What am I missing?[/QUOTE]

You have to open the file in binary mode "wb".
defaultdict(None, {})

Now simulate the effect of writing in text mode and reading in binary mode:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named collections
.... except ImportError as e:
....     e
....
ImportError('No module named collections\r',)

So as Steven suspected there was a whitespace char in the module name: 
pickle.load() was looking for the "collections\r" module.

Peter

 

[paragk]

Hi,

I am unable to figure out the cause of python pickle unable to find
the collection module.

I am getting

    __import__(module)
ImportError: No module named collections

when I try to load a pickled object.

Python version:

Python 2.6 (r26:66721, Oct  2 2008, 11:35:03) [MSC v.1500 32 bit
(Intel)] on win32

[/QUOTE]

import sys
import glob
import collections
import pickle [QUOTE]
...[/QUOTE]

SomeListDict = collections.defaultdict(list)[/QUOTE]
[QUOTE]
... # Populate SomeListDict[/QUOTE]
[QUOTE]
DictFile = open (options.saveDict, 'w')
pickle.dump(SomeListDict , DictFile, -1)
DictFile.close() 
[...]

From the above analysis, clearly the collection module exists, since
the dump works.[/QUOTE]
[QUOTE]
What am I missing?[/QUOTE]

You have to open the file in binary mode "wb".

defaultdict(None, {})

Now simulate the effect of writing in text mode and reading in binary mode:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named collections

... except ImportError as e:
...     e
...
ImportError('No module named collections\r',)

So as Steven suspected there was a whitespace char in the module name:
pickle.load() was looking for the "collections\r" module.

Peter[/QUOTE]

Thank you for the suggestions. Opening the file in 'wb' mode worked.

Thanks,
Parag

 

[Terry Reedy]

Now simulate the effect of writing in text mode and reading in binary mode:

Traceback (most recent call last): ....
ImportError: No module named collections

... except ImportError as e:
... e
...
ImportError('No module named collections\r',)

Nice catch, Peter. I filed a request that the message quote the name it
cannot import, so that the OP would have seen the more informative

ImportError: No module named 'collections\r'

http://bugs.python.org/issue8754

Terry Jan Reedy

 

[Peter Otten]

Nice catch, Peter. I filed a request that the message quote the name it
cannot import, so that the OP would have seen the more informative

ImportError: No module named 'collections\r'

http://bugs.python.org/issue8754

Good idea; thank you for taking the initiative.

Peter