Pointers/References in Python?

[boblatest]

Hello,

I have a long list of memory-heavy objects that I would like to access
in differently sorted order. Let's say I'd like to have lists called
by_date or by_size that I can use to access the objects in the
specified order.

Of course I can just build those lists naively by creating copies of
the original list and then sorting them according to my wishes. But
that would create huge memory overhead. Of course I could use lists of
indices into the "master" list, just as in C I'd create lists or
arrays of pointers into the original data.

Is there a clever Python way to do this, or should I just use lists of
indices?

I know there is a thing called "shallow copy" that has something to do
with not duplicating memory content but I don't understand the
concept. Maybe that's what would help here, too.

Thanks,
robert

 

[bearophileHUGS]

boblatest:

I have a long list of memory-heavy objects that I would like to access
in differently sorted order. Let's say I'd like to have lists called
by_date or by_size that I can use to access the objects in the
specified order.

Just create a new list with a different sorting order, for example
using:

from operator import attrgetter
by_date = sorted(input_data, key=attrgetter("date"))
by_size = sorted(input_data, key=attrgetter("size"))

Python doesn't copy by value by default, so you end having just a
light list of references.

To understand the situation better, there's a FAQ about how Python
associates names to things.

Bye,
bearophile

 

[Maric Michaud]

Le Wednesday 30 July 2008 16:46:37 (e-mail address removed), vous avez écrit :

Hello,

I have a long list of memory-heavy objects that I would like to access
in differently sorted order. Let's say I'd like to have lists called
by_date or by_size that I can use to access the objects in the
specified order.
...
I know there is a thing called "shallow copy" that has something to do
with not duplicating memory content but I don't understand the
concept. Maybe that's what would help here, too.

In fact, all containers in python contains references of objects.

[54]: a = [1]
[55]: b = [a]
[56]: c = list(sorted(b))
[57]: b, c

...[57]: ([[1]], [[1]])

[58]: b.append(3)
[59]: b, c

...[59]: ([[1], 3], [[1]])

[60]: a.append(0)
[61]: b, c

...[61]: ([[1, 0], 3], [[1, 0]])

This is if you want to make a true copy (called deep copy) that you'll have to
do extra steps (using copy module for example).

 

[Gary Herron]

Hello,

I have a long list of memory-heavy objects that I would like to access
in differently sorted order. Let's say I'd like to have lists called
by_date or by_size that I can use to access the objects in the
specified order.

Of course I can just build those lists naively by creating copies of
the original list and then sorting them according to my wishes. But
that would create huge memory overhead. Of course I could use lists of
indices into the "master" list, just as in C I'd create lists or
arrays of pointers into the original data.

Is there a clever Python way to do this, or should I just use lists of
indices?

No need. A Python list contains *references* to objects, not copies of
objects. (The same is true of variables, dictionaries, sets, and so
on...).

For example, in the following code, only one copy of HeavyObject
exists, however, it is referred to many times.

a = HeavyObject()
b = a
A = [a,b]
B = [b,a]
C = set([a,b])
D = {1:a, 2:b}
.... and do on


Implementation wise, a long list consumes about 4 bytes per list element
(that's one address per), plus a tine amount of overhead.


Gary Herron

 

[Terry Reedy]

Hello,

I have a long list of memory-heavy objects that I would like to access
in differently sorted order. Let's say I'd like to have lists called
by_date or by_size that I can use to access the objects in the
specified order.

Of course I can just build those lists naively by creating copies of
the original list and then sorting them according to my wishes. But
that would create huge memory overhead. Of course I could use lists of
indices into the "master" list, just as in C I'd create lists or
arrays of pointers into the original data.

Is there a clever Python way to do this, or should I just use lists of
indices?

I know there is a thing called "shallow copy" that has something to do
with not duplicating memory content but I don't understand the
concept. Maybe that's what would help here, too.

A containers/collections contain/collect references to objects, not the
objects themselves.

lcopy = somelist[:] # or list(somelist)

makes a shallow copy, which only copies the references.

There are applications for lists of indexes, but they are fairly
specialized.

tjr

 

[Robert Latest]

No need. A Python list contains *references* to objects, not copies of
objects. (The same is true of variables, dictionaries, sets, and so
on...).

Good to know. I just wanted to make sure before writing more code which in
the end might not scale well.

Thanks to all for the help!

robert