mutiprocessing, manager, list & threading ?

[mika.saari]

Hi,

Testing if I could be able to use multiprocessing BaseManager to manage
list of instance pointers between processes. If my intance inherits
Thread, I get pickling error about _thread.lock.

I found Steven Bethard's recipe for this kind of problem in
http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods,
but I do not to get the idea how to copyreg the _thread.lock.

Is it even possible to save instance pointer to list which is
synchronized between separate processes ?

Thanks a lot,
-Mika


Client side code:

from multiprocessing.managers import BaseManager
from threading import Thread, Lock

class ListManager(BaseManager):
pass

class Testing(Thread):
def __init__(self):
Thread.__init__(self)

def nothing(self):
print("Nothing")

ListManager.register('get_list')
ListManager.register('Testing',Testing)
m = ListManager(address=('', 50000), authkey=b'abc')
m.connect()

list1 = m.get_list()
test = Testing()
print("TEST:",test)
list1.append(test)
print("TEST:",test)


Error:
-----------------------------------------------------------------------
Traceback (most recent call last):
File "mpclit.py", line 23, in <module>
list1.append(test)
File "<string>", line 2, in append
File "/usr/local/lib/python3.1/multiprocessing/managers.py", line 735,
in _callmethod

conn.send((self._id, methodname, args, kwds))
File "/usr/local/lib/python3.1/pickle.py", line 1358, in dumps
Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <built-in method release of
_thread.lock object at 0x1af710f0>: attribute lookup __main__.release
failed


References:

 

[John Nagle]

Hi,

Testing if I could be able to use multiprocessing BaseManager to manage
list of instance pointers between processes. If my intance inherits
Thread, I get pickling error about _thread.lock.

The "multiprocessing" module works by running completely separate
programs which communicate by copying data back and forth through
pipes or sockets. You can't share thread locks. (There's a gimmick
that lets you put an array of fixed type and size in shared memory,
but that's very limited, because Python's locking doesn't really
understand multiprocessing.)

If you need to access a list from several processes, have
one process own the list, and use a Manager from the multiprocessing
module to create a proxy to access the list from the other processes.
Note that this is slower than regular list access. But if you just
need to have a to-do list that feeds multiple processes, that's a way
to do it.

John Nagle