Getting references to objects without incrementing reference counters

[Artur Siekielski]

Hi.
I'm using CPython 2.7 and Linux. In order to make parallel
computations on a large list of objects I want to use multiple
processes (by using multiprocessing module). In the first step I fill
the list with objects and then I fork() my worker processes that do
the job.

This should work optimally in the aspect of memory usage because Linux
implements copy-on-write in forked processes. So I should have only
one physical list of objects (the worker processes don't change the
objects on the list). The problem is that after a short time children
processes are using more and more memory (they don't create new
objects - they only read objects from the list and write computation
result to the database).

After investigation I concluded the source of this must be
incrementing of a reference counter when getting an object from the
list. It changes only one int but OS must copy the whole memory page
to the child process. I reimplemented the function for getting the
element (from the file listobject.c) but omitting the PY_INCREF call
and it solved my problems with increasing memory.

The questions is: are there any better ways to have a real read-only
list (in terms of memory representation of objects)? My solution is of
course not safe. I thought about weakrefs but it seems they cannot be
used here because getting a real reference from a weakref increases a
reference counter. Maybe another option would be to store reference
counters not in objects, but in a separate array to minimize number of
memory pages they occupy...

 

[Jean-Paul Calderone]

Hi.
I'm using CPython 2.7 and Linux. In order to make parallel
computations on a large list of objects I want to use multiple
processes (by using multiprocessing module). In the first step I fill
the list with objects and then I fork() my worker processes that do
the job.

This should work optimally in the aspect of memory usage because Linux
implements copy-on-write in forked processes. So I should have only
one physical list of objects (the worker processes don't change the
objects on the list). The problem is that after a short time children
processes are using more and more memory (they don't create new
objects - they only read objects from the list and write computation
result to the database).

After investigation I concluded the source of this must be
incrementing of a reference counter when getting an object from the
list. It changes only one int but OS must copy the whole memory page
to the child process. I reimplemented the function for getting the
element (from the file listobject.c) but omitting the PY_INCREF call
and it solved my problems with increasing memory.

The questions is: are there any better ways to have a real read-only
list (in terms of memory representation of objects)? My solution is of
course not safe. I thought about weakrefs but it seems they cannot be
used here because getting a real reference from a weakref increases a
reference counter. Maybe another option would be to store reference
counters not in objects, but in a separate array to minimize number of
memory pages they occupy...

It might be interesting to try with Jython or PyPy. Neither of these
Python runtimes uses reference counting at all.

Jean-Paul

 

[Diez B. Roggisch]

Hi.
I'm using CPython 2.7 and Linux. In order to make parallel
computations on a large list of objects I want to use multiple
processes (by using multiprocessing module). In the first step I fill
the list with objects and then I fork() my worker processes that do
the job.

This should work optimally in the aspect of memory usage because Linux
implements copy-on-write in forked processes. So I should have only
one physical list of objects (the worker processes don't change the
objects on the list). The problem is that after a short time children
processes are using more and more memory (they don't create new
objects - they only read objects from the list and write computation
result to the database).

After investigation I concluded the source of this must be
incrementing of a reference counter when getting an object from the
list. It changes only one int but OS must copy the whole memory page
to the child process. I reimplemented the function for getting the
element (from the file listobject.c) but omitting the PY_INCREF call
and it solved my problems with increasing memory.

The questions is: are there any better ways to have a real read-only
list (in terms of memory representation of objects)? My solution is of
course not safe. I thought about weakrefs but it seems they cannot be
used here because getting a real reference from a weakref increases a
reference counter. Maybe another option would be to store reference
counters not in objects, but in a separate array to minimize number of
memory pages they occupy...

You don't say what data you share, and if all of it is needed for each
child. So it's hard to suggest optimizations. And AFAIK there is no
built-in way of doing what you want. It's complex and error-prone.

Maybe mmap + (struct|pickle) help, if what you need can be formulated in a way
that traversing the whole data piecewise by explicitly
marshaling-demarshaling data?

Diez

 

[Artur Siekielski]

It might be interesting to try with Jython or PyPy.  Neither of these
Python runtimes uses reference counting at all.

I have to use CPython because of C extensions I use (and because I use
Python 2.7 features).

 

[Artur Siekielski]

You don't say what data you share, and if all of it is needed for each
child. So it's hard to suggest optimizations.

Here is an example of such a problem I'm dealing with now: I'm
building large index of words in memory, it takes 50% of RAM. After
building it I want to enable querying it using multiple processors
enabling fast handling of multiple clients at once (communication is
done with sockets). So I create the index in the parent process and
than fork children that handle client queries. But after a short time
only traversing the index by children makes them use much memory,
resulting shortly in Out Of Memory Errors.

And AFAIK there is no
built-in way of doing what you want. It's complex and error-prone.

Having read-only objects is complex and error-prone?

Maybe mmap + (struct|pickle) help, if what you need can be formulated in a way
that traversing the whole data piecewise by explicitly
marshaling-demarshaling data?

In case of this word index it is impossible... And probably would be
too slow. Using mmap and mapping the object structure into structs
should probably work...

 

[Diez B. Roggisch]

Here is an example of such a problem I'm dealing with now: I'm
building large index of words in memory, it takes 50% of RAM. After
building it I want to enable querying it using multiple processors
enabling fast handling of multiple clients at once (communication is
done with sockets). So I create the index in the parent process and
than fork children that handle client queries. But after a short time
only traversing the index by children makes them use much memory,
resulting shortly in Out Of Memory Errors.


Having read-only objects is complex and error-prone?

Yes. Python has no built-in semantics for that. You can't declare an
object to be read-only. And if one wanted to implement that, you'd have
to make sure the object consists only of read-only attributes
itself. And circumvene a great deal of the dynamic features in python
(which you don't need for this usecase, but still are there)

Diez

 

[Jean-Paul Calderone]

And circumvene a great deal of the dynamic features in python
(which you don't need for this usecase, but still are there)

Great as the features might be, when you don't need them, it's clearly
a bad thing to have them drag you down. Fortunately the PyPy team is
making great progress in implementing a runtime that transparently
sheds
those dynamic features when running a program that doesn't take
advantage
of them.

Jean-Paul

 

[Artur Siekielski]

Great as the features might be, when you don't need them, it's clearly
a bad thing to have them drag you down.  Fortunately the PyPy team is
making great progress in implementing a runtime that transparently
sheds
those dynamic features when running a program that doesn't take
advantage
of them.

In case of PyPy such design - forks() + relying on copy-on-write - I
think will not work at all, because PyPy GC moves objects so they will
not stay in the same memory pages. Or if they are not modified they
will stay at the same memory address?

CPython's simple runtime (no real GC) makes reasoning about memory-
level structures easy, so I was hoping that copy-on-write will work
here. Anyway for my usecase using unsafe_listget (implemented by me -
doesn't increment reference count) does the job.

 

[John Nagle]

Hi.
I'm using CPython 2.7 and Linux. In order to make parallel
computations on a large list of objects I want to use multiple
processes (by using multiprocessing module). In the first step I fill
the list with objects and then I fork() my worker processes that do
the job.

This should work optimally in the aspect of memory usage because Linux
implements copy-on-write in forked processes.

There used to be a memory leak when using Pickle to talk
to subprocesses. See what I wrote at

"http://groups.google.com/group/comp.lang.python/browse_thread/thread/3f8b999c25af263a""

The Pickle module has an optimization scheme and cache which
requires that both sender and receiver keep a reference to each
object transmitted. Cache clearing wasn't well worked out for
Pickle object reuse, and the receive end had a memory leak.

Did that get fixed?

John Nagle

 

[Artur Siekielski]

    There used to be a memory leak when using Pickle to talk
to subprocesses.  See what I wrote at

It's something different - I'm not using serialization at all - I have
full Python objects "copied" into child processes .