Update with pickle

[Nicolas Fleury]

Is it possible to use the pickle module to update an object instead of
creating a new instance?
Thx
Nicolas

 

[Matteo Dell'Amico]

Is it possible to use the pickle module to update an object instead of
creating a new instance?

I don't get what you want to do. Can you explain yourself better? What
do you mean by "updating" an object?

 

[Nicolas Fleury]

I don't get what you want to do. Can you explain yourself better? What
do you mean by "updating" an object?

Use the same reference instead of creating a new instance. Basically
making pickle call __init__ of an existing object. The best solution I
see for now is to copy the __dict__ of the new object to the existing
one. I want to update the content of an object with a dumped object of
the same type so that all reference to existing object are still valid.

Regards,
Nicolas

 

[Matteo Dell'Amico]

Use the same reference instead of creating a new instance. Basically
making pickle call __init__ of an existing object. The best solution I
see for now is to copy the __dict__ of the new object to the existing
one. I want to update the content of an object with a dumped object of
the same type so that all reference to existing object are still valid.

I think it's not possible to do automatically. Beware of copying
__dict__, since starting with python 2.2, it's possible to have data
that doesn't reside in __dict__, for instance when using __slots__.

I guess you have to find a way to encapsulate information, and maybe a
proxy could be OK for you (I can't find the recipe in the Python
Cookbook right now, since it appears to be down). That way, you could
keep all references to the proxy, and change the encapsulated data when
you need it.

 

[Leif K-Brooks]

I want to update the content of an object with a dumped object of the
same type so that all reference to existing object are still valid.

Add interfaces to your object's class for mutability rather than using a
low-level hack. Something like this (untested):


class cls(object):
__slots__ = ('foo')
def __init__(self, foo=None):
self.foo = foo

def set_foo(self, foo):
self.foo = foo


foo = cls(42)
bar = foo
print bar.foo # 42
foo.set_foo(43)
print bar.foo # 43

 

[Nicolas Fleury]

Add interfaces to your object's class for mutability rather than using a
low-level hack. Something like this (untested):

Thx for your answer, but according to what I read about __slots__, this
feature has never been intended to be used that way either, so it's also
a kinda low-level hack, no? It is still an interesting solution.

Regards,
Nicolas

 

[Leif K-Brooks]

Thx for your answer, but according to what I read about __slots__, this
feature has never been intended to be used that way either, so it's also
a kinda low-level hack, no? It is still an interesting solution.

Using __slots__ is just an optimization, it isn't directly related to
the example. This would also work:


class cls:
def __init__(self, foo=None):
self.foo = foo

def set_foo(self, foo):
self.foo = foo


foo = cls(42)
bar = foo
print bar.foo # 42
foo.set_foo(43)
print bar.foo # 43

 

[Stefan Seefeld]

Hi there (,hi Nicolas !),

I'v been looking for something similar recently,
so this thread caught my attention, but...

Using __slots__ is just an optimization, it isn't directly related to
the example. This would also work:


class cls:
def __init__(self, foo=None):
self.foo = foo

def set_foo(self, foo):
self.foo = foo


foo = cls(42)
bar = foo
print bar.foo # 42
foo.set_foo(43)
print bar.foo # 43

....I'm still lost: how is this interface-enriched declaration helping
in the task to retrieve ('internalize') an object's state from a pickle ?

Even if I set the object's state via some 'set' manipulators, I still have
to retrieve the state from the pickle and identify the object it is targetted at.

Looking again into the docs for the pickle protocol, I wonder whether there
isn't any place in the __init__/__reduce__/__setstate__ magic to throw in
some meta-programming (i.e. an intelligent metaclass doing the job), or providing
a clever __new__ operator that doesn't return a new object but an existing one,
etc., etc.

Regards,
Stefan

 

[Matteo Dell'Amico]

I guess you have to find a way to encapsulate information, and maybe a
proxy could be OK for you (I can't find the recipe in the Python
Cookbook right now, since it appears to be down).

Now it works again. Have a look at
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252151 . You
can, without problems, save _obj and then restore it with pickle.
I think this solves your problem.

 

[Nicolas Fleury]

Hi there (,hi Nicolas !),

...I'm still lost: how is this interface-enriched declaration helping
in the task to retrieve ('internalize') an object's state from a pickle ?

Hi Stefan,
You're right, the example shows how to copy and it creates a new
reference, so it doesn't help in the problem (I realized it when going
back home yesterday).

Looking again into the docs for the pickle protocol, I wonder whether there
isn't any place in the __init__/__reduce__/__setstate__ magic to throw in
some meta-programming (i.e. an intelligent metaclass doing the job), or
providing
a clever __new__ operator that doesn't return a new object but an
existing one,
etc., etc.

I just added a feature request on sf to have an updating load function
in pickle. My preferred solution for now might still be the copy of
__dict__ (I could copy slots also), it's simple and not intrusive (using
proxies is, even if less a hack). However, the cleanest solution would
be to implement your solution, but I have no idea how to do it for now.

Regards,
Nicolas

 

[Nicolas Fleury]

Hi Stefan,
You're right, the example shows how to copy and it creates a new
reference, so it doesn't help in the problem (I realized it when going
back home yesterday).

I saying crap; there's no copy in the example, only copy of the
reference itself... and it doesn't help in the problem.

 

[Matteo Dell'Amico]

I just added a feature request on sf to have an updating load function
in pickle. My preferred solution for now might still be the copy of
__dict__ (I could copy slots also), it's simple and not intrusive (using
proxies is, even if less a hack). However, the cleanest solution would
be to implement your solution, but I have no idea how to do it for now.

Leif is essentially proposing a non-transparent proxy, while the
cookbook recipe I pointed out is a transparent proxy (i.e. you are using
a proxy, but you can access it just as if it was the original object).

If I understood correctly, you want something like this:
42

Well, it's not possible: variables are just names attached to objects,
and while you can assign a name to an object, you *can't* change the
object's type, or replace an object with another one.

And I think that copying __dict__ is a hack: it wouldn't work, for
instance, with builtin types, when __slots__ are used, or when you are
using properties.

With a proxy you could do something like this:

and then resume the file with
A transparent proxy is a clean way to do it: through delegation, it
would work with any kind of object, changes in code are localized in the
load/save part, and the rest of the application wouldn't even notice.

 

[Nicolas Fleury]

Well, it's not possible: variables are just names attached to objects,
and while you can assign a name to an object, you *can't* change the
object's type, or replace an object with another one.

I think it is acceptable that this pickle.update function would only
accept reference of same type and raise an exception otherwise. The
point of pickle.update would be to not "replace an object with another
one" but to call __init__ on an existing object.

And I think that copying __dict__ is a hack: it wouldn't work, for
instance, with builtin types, when __slots__ are used, or when you are
using properties.

I agree it's a hack. It think the best solution would be direct support
by pickle module and according to Stefan's post, I'm not the only one
with that need (by the way, not only __dict__ can be copied, all other
necessary stuff also).

A transparent proxy is a clean way to do it: through delegation, it
would work with any kind of object, changes in code are localized in the
load/save part, and the rest of the application wouldn't even notice.

I have experienced problems with proxies. They are never totally
transparent, even if all access to members and methods work. For
example, a call to isinstance with the proxy would fail because it is
not of the same type as the aggregated object. All these glitches can
be fixed, but it makes the solution less transparent.

Another problem is that the solution is intrusive, meaning that all
objects that are dumped must have a corresponding proxy, while the best
solution would be to keep this detail at one place and only in the
load/dump part of the application. So on the design point of vue, I
prefer the hack.

Regards,
Nicolas

 

[Matteo Dell'Amico]

I think it is acceptable that this pickle.update function would only
accept reference of same type and raise an exception otherwise. The
point of pickle.update would be to not "replace an object with another
one" but to call __init__ on an existing object.

I still don't understand how exactly you would want it to work. If you
want to call __init__ on an existing object, why don't you just do it
explicitly? Could you post some equivalent code to what you would want
pickle.update to do?

 

[Nicolas Fleury]

I still don't understand how exactly you would want it to work. If you
want to call __init__ on an existing object, why don't you just do it
explicitly? Could you post some equivalent code to what you would want
pickle.update to do?

Unfortunately I can't, something like what Stefan described would be
necessary to do it. Basically, instead of creating a new instance, the
pickle.update would use an existing one. All the rest after that point
is the same, whatever that means (fill up __dict__ or calling
__setstate__ or whatever). In fact, forget about what I said about
__init__

Regards,
Nicolas

 

[Stefan Seefeld]

I still don't understand how exactly you would want it to work. If you
want to call __init__ on an existing object, why don't you just do it
explicitly? Could you post some equivalent code to what you would want
pickle.update to do?

Think of a 'persistence storage service' that could look like this:

store = Persistence.Store('foobar')

object.store(store) # persists the object's state into the store

....

object.restore(store) # restore the object's state from the store

Of course, here 'object' has to implement some 'Storable' interface,
so it's quite intrusive (and thus not really what Nicolas is looking for).
It's just to illustrate the purpose, anyways.
With python's metaprogramming and introspection capabilities the same
could be achieved without the need for a specific base class. Well,
may be a specific __metaclass__, I don't know.

It really seems to me that pickle already contains the functionality
that would be required (after all pickle *can* deal with object graphs,
so it has to track object ids internally), it just needs to be exposed
to the user. I believe that's precisely what Nicolas' enhancement request
is all about.

Regards,
Stefan

 

[Matteo Dell'Amico]

Think of a 'persistence storage service' that could look like this:

store = Persistence.Store('foobar')

object.store(store) # persists the object's state into the store

...

object.restore(store) # restore the object's state from the store

Of course, here 'object' has to implement some 'Storable' interface,
so it's quite intrusive (and thus not really what Nicolas is looking for).
It's just to illustrate the purpose, anyways.
With python's metaprogramming and introspection capabilities the same
could be achieved without the need for a specific base class. Well,
may be a specific __metaclass__, I don't know.

It really seems to me that pickle already contains the functionality
that would be required (after all pickle *can* deal with object graphs,
so it has to track object ids internally), it just needs to be exposed
to the user. I believe that's precisely what Nicolas' enhancement request
is all about.

Now I understand. If you always use it for restoring a previous state
for the same object, it even looks less evil to me Maybe a metaclass
with some kind transactions and rollback could be interesting...