why can't I pickle a class containing this dispatch dictionary?

[jkn]

Hi All
I'm clearly not understanding the 'can't pickle instancemethod
objects' error; can someone help me to understand, & maybe suggest a
workaround, (apart from the obvious if ... elif...).

I'm running Python 2.6 on an embedded system.

== testpickle.py ==
import pickle

class Test(object):
def __init__(self):
self.myDict = {
1: self.tag1,
2: self.tag2
}
def dispatch(self, v):
try:
self.myDict[v]()
except KeyError:
print "No corresponding dictionary entry!"
#
def tag1(self):
print "one"
def tag2(self):
print "two"


t = Test()
t.dispatch(1)
t.dispatch(2)
t.dispatch(0)

fd = open("pickle.out", "w")
pickle.dump(t, fd)
fd.close()
# EOF

$ python testpickle.py
one
two
No corresponding dictionary entry!
Traceback (most recent call last):
File "ptest.py", line 29, in <module>
pickle.dump(t, fd)
File "/usr/lib/python2.6/pickle.py", line 1362, in dump
Pickler(file, protocol).dump(obj)
File "/usr/lib/python2.6/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.6/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
save(state)
File "/usr/lib/python2.6/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
save(v)
File "/usr/lib/python2.6/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
save(v)
File "/usr/lib/python2.6/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle instancemethod objects
$


Thanks
J^n

 

[Michael Hrivnak]

Pickle cannot pickle a reference to an instance method. So the
problem is that self.myDict has values which are references to
instance methods.

Without questioning what this is trying to do or why (I assume it's a
proof of concept), here is a way to make it picklable:
http://pastebin.com/1zqE52mD

Michael

 

[Peter Otten]

I'm clearly not understanding the 'can't pickle instancemethod
objects' error; can someone help me to understand,

I think classes implemented in C need some extra work to make them
picklable, and that hasn't been done for instance methods.

& maybe suggest a
workaround, (apart from the obvious if ... elif...).

You can implement pickling yourself:

import copy_reg
import types

def pickle_instancemethod(m):
return unpickle_instancemethod, (m.im_func.__name__, m.im_self,
m.im_class)

def unpickle_instancemethod(name, im_self, im_class):
im_func = getattr(im_class, name)
return im_func.__get__(im_self, im_class)

copy_reg.pickle(types.MethodType, pickle_instancemethod)

I'm running Python 2.6 on an embedded system.

== testpickle.py ==
import pickle

class Test(object):
def __init__(self):
self.myDict = {
1: self.tag1,
2: self.tag2
}
def dispatch(self, v):
try:
self.myDict[v]()
except KeyError:
print "No corresponding dictionary entry!"
#
def tag1(self):
print "one"
def tag2(self):
print "two"


t = Test()
t.dispatch(1)
t.dispatch(2)
t.dispatch(0)

fd = open("pickle.out", "w")
pickle.dump(t, fd)
fd.close()
# EOF

$ python testpickle.py
one
two
No corresponding dictionary entry!

 

[jkn]

Hi Peter

I think classes implemented in C need some extra work to make them
picklable, and that hasn't been done for instance methods.

by 'classes implemented in C', doyou mean new-style classes', or what,
please?

& maybe suggest a
workaround, (apart from the obvious if ... elif...).

You can implement pickling yourself:

[...]

Hmm - interesting, thanks. I'm more trying to understand the issue at
the moment, but it's always nice to learn...

Thanks
J^n

 

[88888 Dihedral]

Peter Ottenæ–¼ 2012å¹´4月3日星期二UTC+8下åˆ3時54分50秒寫é“：

I'm clearly not understanding the 'can't pickle instancemethod
objects' error; can someone help me to understand,

I think classes implemented in C need some extra work to make them
picklable, and that hasn't been done for instance methods.

& maybe suggest a
workaround, (apart from the obvious if ... elif...).

You can implement pickling yourself:

import copy_reg
import types

def pickle_instancemethod(m):
return unpickle_instancemethod, (m.im_func.__name__, m.im_self,
m.im_class)

def unpickle_instancemethod(name, im_self, im_class):
im_func = getattr(im_class, name)
return im_func.__get__(im_self, im_class)

copy_reg.pickle(types.MethodType, pickle_instancemethod)

I'm running Python 2.6 on an embedded system.

== testpickle.py ==
import pickle

class Test(object):
def __init__(self):
self.myDict = {
1: self.tag1,
2: self.tag2
}
def dispatch(self, v):
try:
self.myDict[v]()
except KeyError:
print "No corresponding dictionary entry!"
#
def tag1(self):
print "one"
def tag2(self):
print "two"


t = Test()
t.dispatch(1)
t.dispatch(2)
t.dispatch(0)

fd = open("pickle.out", "w")
pickle.dump(t, fd)
fd.close()
# EOF

$ python testpickle.py
one
two
No corresponding dictionary entry!

TypeError: can't pickle instancemethod objects
$

Save your python files as a package in .pyd or .py and use exec to get what you want. Of course you can use the data compression package to perform
serialization operations, but that will increase start up time in loading your objects.

 

[88888 Dihedral]

Peter Ottenæ–¼ 2012å¹´4月3日星期二UTC+8下åˆ3時54分50秒寫é“：

I'm clearly not understanding the 'can't pickle instancemethod
objects' error; can someone help me to understand,

I think classes implemented in C need some extra work to make them
picklable, and that hasn't been done for instance methods.

& maybe suggest a
workaround, (apart from the obvious if ... elif...).

You can implement pickling yourself:

import copy_reg
import types

def pickle_instancemethod(m):
return unpickle_instancemethod, (m.im_func.__name__, m.im_self,
m.im_class)

def unpickle_instancemethod(name, im_self, im_class):
im_func = getattr(im_class, name)
return im_func.__get__(im_self, im_class)

copy_reg.pickle(types.MethodType, pickle_instancemethod)

I'm running Python 2.6 on an embedded system.

== testpickle.py ==
import pickle

class Test(object):
def __init__(self):
self.myDict = {
1: self.tag1,
2: self.tag2
}
def dispatch(self, v):
try:
self.myDict[v]()
except KeyError:
print "No corresponding dictionary entry!"
#
def tag1(self):
print "one"
def tag2(self):
print "two"


t = Test()
t.dispatch(1)
t.dispatch(2)
t.dispatch(0)

fd = open("pickle.out", "w")
pickle.dump(t, fd)
fd.close()
# EOF

$ python testpickle.py
one
two
No corresponding dictionary entry!

TypeError: can't pickle instancemethod objects
$

Save your python files as a package in .pyd or .py and use exec to get what you want. Of course you can use the data compression package to perform
serialization operations, but that will increase start up time in loading your objects.

 

[Peter Otten]

Hi Peter



by 'classes implemented in C', doyou mean new-style classes', or what,
please?
Given

.... def __init__(self, name):
.... self.name = name
.... def hello(self):
.... print "Hello,", self.name
....Hello, Peter

the object bound to the name 'hello' is an instance of the 'instancemethod'
type:
<type 'instancemethod'>

That type is implemented in C, see

http://hg.python.org/cpython/file/9599f091faa6/Objects/classobject.c

and doesn't support the pickle protocol while a similar class,
functools.partial which is also written in C, see

http://hg.python.org/cpython/file/9599f091faa6/Modules/_functoolsmodule.c

does:
.... print "Hi,", obj.name
....Hi, Peter