pickling instances of metaclass generated classes

[lars van gemerden]

Hello,

Can someone help me with the following:

I am using metaclasses to make classes and these classes to make
instances. Now I want to use multiprocessing, which needs to pickle
these instances.

Pickle cannot find the class definitions of the instances. I am trying
to add a line to the __new__ of the metaclass to add the new class
under the right name in the right module/place, so pickle can find
it.

Is this the right approach? Can anyone explain to me where/how to add
these classes for pickle to find and maybe why?

Thanks in advance,

Lars

 

[Robert Kern]

Hello,

Can someone help me with the following:

I am using metaclasses to make classes and these classes to make
instances. Now I want to use multiprocessing, which needs to pickle
these instances.

Pickle cannot find the class definitions of the instances. I am trying
to add a line to the __new__ of the metaclass to add the new class
under the right name in the right module/place, so pickle can find
it.

Is this the right approach? Can anyone explain to me where/how to add
these classes for pickle to find and maybe why?

Can you post some code (preferably pared down to a minimal example that fails)?
I'm not really clear on what you are doing. I would expect that a class defined
by a class statement would usually work fine unless if the metaclass is doing
something particularly weird to it.

In any case, you can probably just explicitly register a reduction function for
each type using copy_reg.pickle():

http://docs.python.org/library/copy_reg

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Ian Kelly]

Hello,

Can someone help me with the following:

I am using metaclasses to make classes and these classes to make
instances. Now I want to use multiprocessing, which needs to pickle
these instances.

Pickle cannot find the class definitions of the instances. I am trying
to add a line to the __new__ of the metaclass to add the new class
under the right name in the right module/place, so pickle can find
it.

Is this the right approach? Can anyone explain to me where/how to add
these classes for pickle to find and maybe why?

It sounds like you're trying to do something like this?
.... pass
....Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\python27\lib\pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "c:\python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "c:\python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "c:\python27\lib\pickle.py", line 401, in save_reduce
save(args)
File "c:\python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "c:\python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "c:\python27\lib\pickle.py", line 295, in save
self.save_global(obj)
File "c:\python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
it's not found as __main__.<Anonymous>


Yeah, pickle's not going to work with anonymous classes. As you
suggest, you could dynamically add the classes to the module namespace
so that pickle.dumps will find them, but bear in mind that they will
also have to exist when calling pickle.loads, so you will need to be
able to reconstruct the same anonymous classes before unpickling later
on.

Cheers,
Ian

 

[lars van gemerden]

It sounds like you're trying to do something like this?


...     pass


Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\python27\lib\pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "c:\python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "c:\python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "c:\python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "c:\python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "c:\python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "c:\python27\lib\pickle.py", line 295, in save
    self.save_global(obj)
  File "c:\python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
it's not found as __main__.<Anonymous>

Yeah, pickle's not going to work with anonymous classes.  As you
suggest, you could dynamically add the classes to the module namespace
so that pickle.dumps will find them, but bear in mind that they will
also have to exist when calling pickle.loads, so you will need to be
able to reconstruct the same anonymous classes before unpickling later
on.

Cheers,
Ian

Thank you Ian for the minimal example. This is almost the case i was
trying to discribe, however the classes will be named at runtime and
the class definitions will be persisted in a database.

Can you help me with how to add the classes to the correct namespace?
The application spans multiple modules (or compared to the example,
the metaclass definition will be in another module then one where the
class and instance will be generated).

Cheers, Lars

 

[lars van gemerden]

It sounds like you're trying to do something like this?


...     pass


Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\python27\lib\pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "c:\python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "c:\python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "c:\python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "c:\python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "c:\python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "c:\python27\lib\pickle.py", line 295, in save
    self.save_global(obj)
  File "c:\python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
it's not found as __main__.<Anonymous>

Yeah, pickle's not going to work with anonymous classes.  As you
suggest, you could dynamically add the classes to the module namespace
so that pickle.dumps will find them, but bear in mind that they will
also have to exist when calling pickle.loads, so you will need to be
able to reconstruct the same anonymous classes before unpickling later
on.

Cheers,
Ian

Ian also wrote:

'''
Actually, I was wrong, you probably don't need to do that. I suggest
going with Robert Kern's suggestion to either register the class with
the copy_reg module, or (perhaps better since it won't leak
registrations) implement a __reduce__ method on the class. For
example, this seems to work:
.... cls = MetaClass.build_class(*metaclass_args)
.... self = cls.__new__(cls)
.... return self
........ @classmethod
.... def build_class(mcs, arg1, arg2, arg3):
.... # Do something useful with the args...
.... class _AnonymousClass(object):
.... __metaclass__ = mcs
.... def __reduce__(self):
.... return (reconstructor, ('foo', 'bar', 'baz'),
self.__dict__)
.... return _AnonymousClass
....

"c__main__\nreconstructor
\np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI42\nsb."

False

Cheers,
Ian

'''

 

[Peter Otten]

Thank you Ian for the minimal example. This is almost the case i was
trying to discribe, however the classes will be named at runtime and
the class definitions will be persisted in a database.

Can you help me with how to add the classes to the correct namespace?
The application spans multiple modules (or compared to the example,
the metaclass definition will be in another module then one where the
class and instance will be generated).

If the metaclass is global in whatever module you don't need to bother about
that. The problem is that you cannot access the classes (metaclass
instances) under a dotted name. One workaround is a pseudo-module that does
whatever is needed to recreate the class:

import pickle
import sys

class MetaClass(type):
pass

class M(object):
def __init__(self, module):
self.__module = module
def __getattr__(self, name):
print "creating class", name
class_ = MetaClass(name, (), {"__module__": self.__module})
setattr(self, name, class_)
return class_

sys.modules["m"] = M("m")
import m
c = m.x
s = pickle.dumps(c)
print repr(s)
d = pickle.loads(s)

assert c is d

sys.modules["m"] = M("m")
e = pickle.loads(s)

assert c is not e

The official way is probably what Robert mentioned, via the copy_reg module,
but I didn't get it to work.

 

[lars van gemerden]

Ian also wrote:

'''
Actually, I was wrong, you probably don't need to do that.  I suggest
going with Robert Kern's suggestion to either register the class with
the copy_reg module, or (perhaps better since it won't leak
registrations) implement a __reduce__ method on the class.  For
example, this seems to work:


...     cls = MetaClass.build_class(*metaclass_args)
...     self = cls.__new__(cls)
...     return self
...>>> class MetaClass(type):

...     @classmethod
...     def build_class(mcs, arg1, arg2, arg3):
...         # Do something useful with the args...
...         class _AnonymousClass(object):
...             __metaclass__ = mcs
...             def __reduce__(self):
...                 return (reconstructor, ('foo', 'bar','baz'),
self.__dict__)
...         return _AnonymousClass
...>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()

"c__main__\nreconstructor
\np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI 42\nsb.">>> inst2 = pickle.loads(s)

False

Cheers,
Ian

'''

Interesting, though I cannot say I completely understand this solution
(looked up __reduce__, but still). I am trying to adapt this example
to a situation where the metaclass generated classes are named at
runtime (not anonymous), but cannot figure it out.

Cheers, Lars

 

[lars van gemerden]

Interesting, though I cannot say I completely understand this solution
(looked up __reduce__, but still). I am trying to adapt this example
to a situation where the metaclass generated classes are named at
runtime (not anonymous), but cannot figure it out.

Cheers, Lars

Found a way to name the classes:

def reconstructor(*metaclass_args):
cls = MetaClass2.build_class(*metaclass_args)
self = cls.__new__(cls)
return self

class MetaClass(type):
@classmethod
def build_class(mcs, name, arg1, arg2, arg3):
return mcs(name, (object,), {"__reduce__": lambda e:
(reconstructor2, (name, arg1, arg2, arg3), e.__dict__)})

I still wonder whether it might be easier to add the class to the
namespace. Can anyone help me with that?

Regards, Lars

 

[Ian Kelly]

I still wonder whether it might be easier to add the class to the
namespace. Can anyone help me with that?

from mypackage import mymodule

setattr(mymodule, myclass.__name__, myclass)

 

[Peter Otten]

import pickle
import sys

class MetaClass(type):
pass

class M(object):
def __init__(self, module):
self.__module = module
def __getattr__(self, name):
print "creating class", name
class_ = MetaClass(name, (), {"__module__": self.__module})
setattr(self, name, class_)
return class_

sys.modules["m"] = M("m")
import m
c = m.x
s = pickle.dumps(c)
print repr(s)
d = pickle.loads(s)

assert c is d

sys.modules["m"] = M("m")
e = pickle.loads(s)

assert c is not e

The official way is probably what Robert mentioned, via the copy_reg
module, but I didn't get it to work.

I will look further into this. does "sys.modules["m"] = M("m")" create
a new module?

Assigning to sys.modules[modulename] can put arbitrary objects into the
module cache, in this case an M instance. To drive the point home:

import sys
sys.modules["x"] = 42
import x
x 42
sys.modules["x"] = "spam"
import x
x

'spam'

Cheers, Lars

PS: I get an error when posting this to the usenet group

Sorry, that seems to happen when I post via gmane and don't manually clear
the follow-up that my newsreader helpfully (knode) inserts. I've not yet
found a permanent fix, but if that was the problem you should be able to
answer this post.

 

[lars van gemerden]

Hello,

Can someone help me with the following:

I am using metaclasses to make classes and these classes to make
instances. Now I want to use multiprocessing, which needs to pickle
these instances.

Pickle cannot find the class definitions of the instances. I am trying
to add a line to the __new__ of the metaclass to add the new class
under the right name in the right module/place, so pickle can find
it.

Is this the right approach? Can anyone explain to me where/how to add
these classes for pickle to find and maybe why?

Thanks in advance,

Lars

Ok,

After reading all posts (thanks a lot), I am considering to use the
following base metaclass for all metaclasses that must lead to
pickleable instances (not pickleable classes):


import sys

class Meta(type):
def __new__(mcls, name, bases, attrs):
cls = type.__new__(mcls, name, bases, attrs)
setattr(sys.modules[__name__], name, cls)
return cls


if __name__ == '__main__':
instance = Meta("Klass", (str,),{})("apple")
s = pickle.dumps(instance)
delattr(sys.modules[__name__], "Klass")
Meta("Klass", (str,),{})
inst = pickle.loads(s)
print instance
print inst
print type(instance) is type(inst)

Can anyone see any drawbacks to this approach? I've also tested the
case where the Meta metaclass is defined in another module.

Cheers, Lars