UnpicklingError: NEWOBJ class argument isn't a type object

[skunkwerk]

Hi,
I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them, based on the code from Shane Hathaway here:
http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-unpicklable-items

It works most of the time, but when I try to unpickle a Django HttpResponse, I get the following error:
UnpicklingError: NEWOBJ class argument isn't a type object

I have no clue what the error actually means. If it pickles okay, why should it not be able to unpickle? Any ideas?

thanks for the help,
imran

Here is my code:

from cPickle import Pickler, Unpickler, UnpicklingError

class FilteredObject:
def __init__(self, about):
self.about = about
def __repr__(self):
return 'FilteredObject(%s)' % repr(self.about)

class MyPickler(object):
def __init__(self, file, protocol=2):
pickler = Pickler(file, protocol)
pickler.persistent_id = self.persistent_id
self.dump = pickler.dump
self.clear_memo = pickler.clear_memo

def persistent_id(self, obj):
if not hasattr(obj, '__getstate__') and not isinstance(obj,
(basestring, bool, int, long, float, complex, tuple, list, set, dict)):
return ["filtered:%s" % str(obj)]
else:
return None

class MyUnpickler(object):
def __init__(self, file):
unpickler = Unpickler(file)
unpickler.persistent_load = self.persistent_load
self.load = unpickler.load
self.noload = unpickler.noload

def persistent_load(self, obj_id):
if obj_id[0].startswith('filtered:'):
return FilteredObject(obj_id[0][9:])
else:
raise UnpicklingError('Invalid persistent id')

###### serialize to file

f = open('test.txt','wb')
p = MyPickler(f)
p.dump(data)
f.close()

###### unserialize from file

f = open('test.txt','rb')
pickled_data = f.read()
f.seek(0)
u = MyUnpickler(f)
data = u.load()

 

[Chris Angelico]

I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them...

If it pickles okay, why should it not be able to unpickle? Any ideas?

Generally, the reason something won't pickle is because it won't be
able to be unpickled. So arbitrarily creating a string might allow the
pickle operation to continue, but might well prevent unpickling still.
I don't know, you'd have to play around with it.

ChrisA

 

[dieter]

Hi,
I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them, based on the code from Shane Hathaway here:
http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-unpicklable-items

It works most of the time, but when I try to unpickle a Django HttpResponse, I get the following error:
UnpicklingError: NEWOBJ class argument isn't a type object

I have no clue what the error actually means.

The pickling protocol uses a form of bytecode which is executed
during the unpickling to reconstruct the python objects based
on their state found in the pickle alongside the bytecode.

"NEWOBJ" is executed in response to such a bytecode operation.
It expects to get a type as a parameter but in your case,
it gets something else.

If it pickles okay, why should it not be able to unpickle? Any ideas?

It is by principle impossible for the pickler to garantee
that an unpickler will later succeed: the pickler does not know
which classes/types are available for the unpickler.

In your special case, the pickler could probably
detect that unpickling will fail - but when
an aim cannot be achieved completely this may provide
motivation to abandon it as a whole - and not put much effort
into a partial achievement.
I have seen many cases where pickling succeeded but
unpickling failed and in principle the pickler could have
already predicted the failure (under the assumption that
the unpickler sees the same classes/types as the pickler).


If it is important for you to get Django HttpResponses
successfully unpickled then you likely need to guide
their pickling process better.

Maybe (as an alternative), you can extract the relevant information
from the "HttpResponse" and pickle that instead of
the response itself?

 

[Peter Otten]

Hi,
I'm using a custom pickler that replaces any un-pickleable objects (such
as sockets or files) with a string representation of them, based on the
code from Shane Hathaway here:
http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with- some-unpicklable-items

It works most of the time, but when I try to unpickle a Django
HttpResponse, I get the following error: UnpicklingError: NEWOBJ class
argument isn't a type object

I have no clue what the error actually means. If it pickles okay, why
should it not be able to unpickle? Any ideas?

A simple way to provoke the error is to rebind the name referring to the
class of the pickled object:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
cPickle.UnpicklingError: NEWOBJ class argument isn't a type object

You may be doing something to that effect.

 

[skunkwerk]

skunkwerk wrote:












A simple way to provoke the error is to rebind the name referring to the

class of the pickled object:




Traceback (most recent call last):

File "<stdin>", line 1, in <module>

cPickle.UnpicklingError: NEWOBJ class argument isn't a type object



You may be doing something to that effect.

Hey Peter,
I tried unpickling even from another file with no other code in it, but came up with the same error - so I don't think it's a rebinding issue.

But I got the error to disappear when I removed the "hasattr(obj, '__getstate__')" from this line of code in the persistent_id function:
if not hasattr(obj, '__getstate__') and isinstance(obj,(basestring, bool, int, long, float, complex, tuple, list, set, dict)):
return ["filtered:%s" % type(obj)]

When I do that, I get a few more FilteredObjects in the result, for things like:
<class 'django.core.handlers.wsgi.WSGIRequest'>
<class 'MySQLdb.connections.Connection'>

I figured these classes must have __getstate__ methods which leads to them being pickled without a persistent_id (it turns out they actually have __repr__ methods).

So these classes get pickled fine, but run into problems when trying to unpickle them. I understand why ImportErrors would happen if the necessary modules haven't been loaded, but this NEWOBJ error is still kind of mystifying.
I guess I just won't pickle any classes for now, if unpickling them is going to be dicey.

thanks for the help guys,
imran

 

[Peter Otten]

skunkwerk wrote:












A simple way to provoke the error is to rebind the name referring to the

class of the pickled object:









Traceback (most recent call last):

File "<stdin>", line 1, in <module>

cPickle.UnpicklingError: NEWOBJ class argument isn't a type object



You may be doing something to that effect.

Hey Peter,
I tried unpickling even from another file with no other code in it, but
came up with the same error - so I don't think it's a rebinding issue.

But I got the error to disappear when I removed the "hasattr(obj,
'__getstate__')" from this line of code in the persistent_id function: if
not hasattr(obj, '__getstate__') and isinstance(obj,(basestring, bool,
int, long, float, complex, tuple, list, set, dict)):
return ["filtered:%s" % type(obj)]

When I do that, I get a few more FilteredObjects in the result, for things
like: <class 'django.core.handlers.wsgi.WSGIRequest'>
<class 'MySQLdb.connections.Connection'>

I figured these classes must have __getstate__ methods which leads to them
being pickled without a persistent_id (it turns out they actually have
__repr__ methods).

So these classes get pickled fine, but run into problems when trying to
unpickle them. I understand why ImportErrors would happen if the
necessary modules haven't been loaded, but this NEWOBJ error is still kind
of mystifying. I guess I just won't pickle any classes for now, if
unpickling them is going to be dicey.

thanks for the help guys,
imran

Maybe you can find the problem by temporarily switching from cPickle to the
pickle module which produces a slightly more helpful traceback:
Traceback (most recent call last):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
obj = cls.__new__(cls, *args)
TypeError: int.__new__(X): X is not a type object (int)