Moving class used in pickle

[Jeffrey Barish]

I have a class derived from string that is used in a pickle. In the new
version of my program, I moved the module containing the definition of the
class. Now the unpickle fails because it doesn't find the module. I was
thinking that I could make the unpickle work by putting a copy of the
module in the original location and then redefine the class by sticking a
__setstate__ in the class thusly:

def __setstate__(self, state):
self.__dict__.update(state)
self.__class__ = NewClassName

My plan was to specify the new location of the module in NewClassName.
However, when I do this I get the message "'class' object layout differs
from 'class'". I don't think that they do as the new module is a copy of
the old one. I suspect that I am not allowed to make the class assignment
because my class is derived from string. What is the best way to update
the pickle? The only thought I have is to read all the data with the old
class module, store the data in some nonpickle format, and then, with
another program, read the nonpickle-format file and rewrite the pickle with
the class module in the new location.

 

[Berthold =?utf-8?Q?H=C3=B6llmann?=]

I have a class derived from string that is used in a pickle. In the new
version of my program, I moved the module containing the definition of the
class. Now the unpickle fails because it doesn't find the module. I was
thinking that I could make the unpickle work by putting a copy of the
module in the original location and then redefine the class by sticking a
__setstate__ in the class thusly:

def __setstate__(self, state):
self.__dict__.update(state)
self.__class__ = NewClassName

My plan was to specify the new location of the module in NewClassName.
However, when I do this I get the message "'class' object layout differs
from 'class'". I don't think that they do as the new module is a copy of
the old one. I suspect that I am not allowed to make the class assignment
because my class is derived from string. What is the best way to update
the pickle? The only thought I have is to read all the data with the old
class module, store the data in some nonpickle format, and then, with
another program, read the nonpickle-format file and rewrite the pickle with
the class module in the new location.

You can fiddle with the file class used reading the pickled file. I.e.
the "read()" method could replace each instance of "foo.myclass" by
"greatnewmodule.mynewclass" could bring you back in the game.

Porting some applications of my from 32 to 64 bit i discovered that my
Numeric int arrays really had to be int32. So i opened the (binary)
pickled files and replaced the occurences of "'l'" by "'i'". Later we
replaced Numeric by numpy. I used the editor approach again to replace
"Numeric" string by "numpy" or "numpy.oldnumeric". A collegue of mine
wrote a small reader class implementing the approach working on the
input stream.

Regards
Berthold
--

 

[Peter Otten]

I have a class derived from string that is used in a pickle. In the new
version of my program, I moved the module containing the definition of the
class. Now the unpickle fails because it doesn't find the module. I was
thinking that I could make the unpickle work by putting a copy of the
module in the original location and then redefine the class by sticking a
__setstate__ in the class thusly:

def __setstate__(self, state):
self.__dict__.update(state)
self.__class__ = NewClassName

My plan was to specify the new location of the module in NewClassName.
However, when I do this I get the message "'class' object layout differs
from 'class'". I don't think that they do as the new module is a copy of
the old one. I suspect that I am not allowed to make the class assignment
because my class is derived from string. What is the best way to update
the pickle? The only thought I have is to read all the data with the old
class module, store the data in some nonpickle format, and then, with
another program, read the nonpickle-format file and rewrite the pickle
with the class module in the new location.

You could overwrite Unpickler.find_class():

import pickle
from cStringIO import StringIO

class AliasUnpickler(pickle.Unpickler):
def __init__(self, aliases, *args, **kw):
pickle.Unpickler.__init__(self, *args, **kw)
self.aliases = aliases
def find_class(self, module, name):
module, name = self.aliases.get((module, name), (module, name))
return pickle.Unpickler.find_class(self, module, name)

def loads(aliases, str):
file = StringIO(str)
return AliasUnpickler(aliases, file).load()

if __name__ == "__main__":
import before, after
data = before.A()
print data.__class__, data
dump = pickle.dumps(data)
data = loads({("before", "A"): ("after", "B")}, dump)
print data.__class__, data

In the example the aliases dictionary maps (module, classname) pairs to
(module, classname) pairs. Of course this only works when the class layout
wasn't changed.

Peter

 

[Gabriel Genellina]

En Mon, 21 May 2007 16:24:55 -0300, Berthold Höllmann

You can fiddle with the file class used reading the pickled file. I.e.
the "read()" method could replace each instance of "foo.myclass" by
"greatnewmodule.mynewclass" could bring you back in the game.

There is a hook in the pickle module (load_global or find_global) that you
can override instead, and it's exactly for this usage, see:
http://docs.python.org/lib/pickle-sub.html