problems with shelve(), collections.defaultdict, self

[7stud]

The following code demonstrates that a collections.defaultdict is
shelve worthy:


import shelve
import collections as c


dd = c.defaultdict(int)
dd["Joe"] = 3
print(dd)

my_shelve = shelve.open('data.shelve')
my_shelve['strike_record'] = dd
my_shelve.close()

my_shelve = shelve.open('data.shelve')
data = my_shelve['strike_record']
my_shelve.close()

dd.clear()
dd.update(data)
print(dd)

--output:--
defaultdict(<class 'int'>, {'Joe': 3})
defaultdict(<class 'int'>, {'Joe': 3})


And the following code demonstrates that a class that inherits from
dict can shelve itself:

import collections as c
import shelve

class Dog(dict):
def __init__(self):
super().__init__(Joe=1)
print('****', self)

def save(self):
my_shelve = shelve.open('data22.shelve')
my_shelve['x'] = self
my_shelve.close()

def load(self):
my_shelve = shelve.open('data22.shelve')
data = my_shelve['x']
my_shelve.close()

print(data)


d = Dog()
d.save()
d.load()

--output:--
**** {'Joe': 1}
{'Joe': 1}


But I cannot get a class that inherits from collections.defaultdict to
shelve itself:


import collections as c
import shelve

class Dog(c.defaultdict):
def __init__(self):
super().__init__(int, Joe=0)
print('****', self)

def save(self):
my_shelve = shelve.open('data22.shelve')
my_shelve['dd'] = self
my_shelve.close()

def load(self):
my_shelve = shelve.open('data22.shelve')
data = my_shelve['dd']
my_shelve.close()

print(data)


d = Dog()
d.save()
d.load()

--output:--

**** defaultdict(<class 'int'>, {'Joe': 30})
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
python3.2/shelve.py", line 111, in __getitem__
value = self.cache[key]
KeyError: 'dd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "3.py", line 95, in <module>
d.load()
File "3.py", line 87, in load
data = my_shelve['dd']
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
python3.2/shelve.py", line 114, in __getitem__
value = Unpickler(f).load()
TypeError: __init__() takes exactly 1 positional argument (2 given)



I deleted all *.shelve.db files between program runs. I can't figure
out what I'm doing wrong.

 

[7stud]

But I cannot get a class that inherits from collections.defaultdict to
shelve itself:

import collections as c
import shelve

class Dog(c.defaultdict):
    def __init__(self):
        super().__init__(int, Joe=0)
        print('****', self)

Whoops. I changed:

super().__init__(int, Joe=0)

to:

super().__init__(int, Joe=30)

hence this output..

 

[7stud]

I don't know if this helps, but I notice when I initially do this:

shelve.open('data22')

the file is saved as 'data22.db'. But on subsequent calls to
shelve.open(), if I use the file name 'data22.db', I get a different
error:

--output:--

**** defaultdict(<class 'int'>, {'Joe': 30})
Traceback (most recent call last):
File "3.py", line 95, in <module>
d.load()
File "3.py", line 86, in load
my_shelve = shelve.open('data22.db')
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
python3.2/shelve.py", line 232, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
python3.2/shelve.py", line 216, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol,
writeback)
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
python3.2/dbm/__init__.py", line 83, in open
raise error[0]("db type could not be determined")
dbm.error: db type could not be determined



The code that produced that error:



import collections as c
import shelve

class Dog(c.defaultdict):
def __init__(self):
super().__init__(int, Joe=30)
print('****', self)

def save(self):
my_shelve = shelve.open('data22')
my_shelve['dd'] = self
my_shelve.close()

def load(self):
my_shelve = shelve.open('data22.db')
data = my_shelve['dd']
my_shelve.close()

print(data)


d = Dog()
d.save()
d.load()


I'm using python 3.2.2.

 

[Ian Kelly]

But I cannot get a class that inherits from collections.defaultdict to
shelve itself:


import collections as c
import shelve

class Dog(c.defaultdict):
   def __init__(self):
       super().__init__(int, Joe=0)
       print('****', self)

   def save(self):
       my_shelve = shelve.open('data22.shelve')
       my_shelve['dd'] = self
       my_shelve.close()

   def load(self):
       my_shelve = shelve.open('data22.shelve')
       data = my_shelve['dd']
       my_shelve.close()

       print(data)


d = Dog()
d.save()
d.load()

--output:--

**** defaultdict(<class 'int'>, {'Joe': 30})
Traceback (most recent call last):
 File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
python3.2/shelve.py", line 111, in __getitem__
   value = self.cache[key]
KeyError: 'dd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "3.py", line 95, in <module>
   d.load()
 File "3.py", line 87, in load
   data = my_shelve['dd']
 File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
python3.2/shelve.py", line 114, in __getitem__
   value = Unpickler(f).load()
TypeError: __init__() takes exactly 1 positional argument (2 given)



I deleted all *.shelve.db files between program runs.  I can't figure
out what I'm doing wrong.

The problem is that defaultdict defines a custom __reduce__ method
which is used by the pickle protocol to determine how the object
should be reconstructed. It uses this to reconstruct the defaultdict
with the same default factory, by calling the type with a single
argument of the default factory. Since your subclass's __init__
method takes no arguments, this results in the error you see.

There are a couple of ways you could fix this. The first would be to
change the signature of the __init__ method to take an optional
argument accepting the default factory instead of hard-coding it, like
this:

def __init__(self, default_factory=int):
super().__init__(default_factory, Joe=0)

The other would be to just fix the __reduce__ method to not pass the
default factory to your initializer, since it is hard-coded. That
would look like this:

def __reduce__(self):
return (type(self), ())

Cheers,
Ian

 

[Ian Kelly]

The problem is that defaultdict defines a custom __reduce__ method
which is used by the pickle protocol to determine how the object
should be reconstructed.  It uses this to reconstruct the defaultdict
with the same default factory, by calling the type with a single
argument of the default factory.  Since your subclass's __init__
method takes no arguments, this results in the error you see.

There are a couple of ways you could fix this.  The first would be to
change the signature of the __init__ method to take an optional
argument accepting the default factory instead of hard-coding it, like
this:

   def __init__(self, default_factory=int):
       super().__init__(default_factory, Joe=0)

The other would be to just fix the __reduce__ method to not pass the
default factory to your initializer, since it is hard-coded.  That
would look like this:

   def __reduce__(self):
       return (type(self), ())

It occurs to me that there's also an option 3: you don't really need a
defaultdict to do what you're trying to do here. You just need a dict
with a custom __missing__ method. That could look something like
this:

class Dog(dict):

def __missing__(self):
return 0

And then you don't have to worry about the weird pickle behavior of
defaultdict at all.

Cheers,
Ian

 

[Ian Kelly]

class Dog(dict):

   def __missing__(self):
       return 0

Sorry, that should have been:

class Dog(dict):

def __missing__(self, key):
return 0

Cheers,
Ian

 

[7stud]

Sorry, that should have been:

class Dog(dict):

    def __missing__(self, key):
        return 0

Cheers,
Ian

Thanks Ian!