Please help with MemoryError

[Alf P. Steinbach]

* Antoine Pitrou:

Le Fri, 12 Feb 2010 17:14:57 +0000, Steven D'Aprano a écrit :

Well, I think the Ruby people got it right. Python *does* pass parameters
by reference. After all, we even speak about reference counting,
reference cycles, etc.

So I'm not sure what distinction you're making here.

Steven talks about the standard meaning of "pass by reference".

Essentially that means that you can implement a routine "swap" such that

a = 1; b = 2; swap( a, b )

results in a == 2 and b == 1.

In Pascal that's 'var' parameters, in C there's no such, in C++ it's reference
type parameters using '&' as a type builder, in C# it's 'out' and 'ref'
parameters (I think C# has both the most practical and the most elegant
implementation, you can see at the call site whether it's by reference), so on.

To some degree "pass by reference" is however just a collection of special cases
with some similarity. For example, if one says that the guiding principle & main
criterion is that one should not have to indicate the pass by reference at the
call site, then that would exclude C#. But if one allows some indication at the
call site, then that would perhaps include C (applying the & address op), which
everyone agrees does not have call by reference.

So, to understand the conventional meaning of the term it is perhaps necessary
to have some knowledge of all the special cases, like here C# versus C.


Cheers & hth.,

- Alf

 

[Antoine Pitrou]

Le Fri, 12 Feb 2010 17:10:01 -0500, Steve Holden a écrit :

As has already been pointed out, if Python used call by reference then
the following code would run without raising an AssertionError:

def exchange(a, b):
a, b = b, a

x = 1
y = 2
exchange(x, y)
assert (x == 2 and y == 1)

Since function-local assignment always takes place in the function
call's local namespace

I don't think this is related to a difference in parameter passing style.

It's just that assignment ("=") means a different thing in Python than in
non-object languages (or fake-object languages such as C++ or PHP): it
rebinds instead of mutating in-place. If it mutated, you wouldn't have
the AssertionError.

If you mutate in-place:
.... t = a[:]
.... a[:] = b
.... b[:] = t
....

x, y = [1], [2]
exchange(x, y)
x, y

([2], [1])

Regards

Antoine.

 

[Antoine Pitrou]

Le Fri, 12 Feb 2010 23:12:06 +0100, Alf P. Steinbach a écrit :

Steven talks about the standard meaning of "pass by reference".

See my answer to Steve's message. You can't postulate a "standard
meaning" of "pass by reference" independently of the specificities of
each language. For example a "variable" in Python means something
different than in non-object languages; it is impossible to have an
agnostic definition.

In Python and other true object languages, the operations on references
(especially assignment) don't have the same semantics as in more archaic
languages. This is no reason, IMO, to refuse using the term "pass by
reference". The state of art in computing languages evolves, and it
wouldn't be constructive to remain stuck with definitions from the 1960s.

Regards

Antoine.

 

[Alf P. Steinbach]

* Antoine Pitrou:

Le Fri, 12 Feb 2010 23:12:06 +0100, Alf P. Steinbach a écrit :

See my answer to Steve's message. You can't postulate a "standard
meaning" of "pass by reference" independently of the specificities of
each language.

Agreed. See e.g. my discussion of C versus C# in the article you responded to.

For example a "variable" in Python means something
different than in non-object languages; it is impossible to have an
agnostic definition.

It's possible and relatively speaking easy, but it's bound up with such a tangle
of terminological and religious issues that it's best not discussed here. :-(

But if you find/have a description that helps you understand what's going on,
and that correctly predicts the effect of code, then just Use It.

After all, learning some "simpler" description would at that point just be more
work.

In Python and other true object languages, the operations on references
(especially assignment) don't have the same semantics as in more archaic
languages. This is no reason, IMO, to refuse using the term "pass by
reference". The state of art in computing languages evolves, and it
wouldn't be constructive to remain stuck with definitions from the 1960s.

The main reason for not using that term for Python is that "pass by reference"
has the extremely strong connotation of being able to implement 'swap'.

And since there already is adequate standard terminology, why confuse things.


Cheers & hth.,

- Alf

 

[Gib Bogle]

def swap(a, b):
a, b = b, a

x = 1
y = 2
swap(x, y)
assert (x == 2) and (y==1)

Can't the same point be more simply made with this example:

def setval(a):
a = 12345

x = 1
setval(x)
print x

?

 

[Steve Holden]

Can't the same point be more simply made with this example:

def setval(a):
a = 12345

x = 1
setval(x)
print x

Yes, and it will doubtless be subject to exactly the same bogus refutations.

regards
Steve

 

[Antoine Pitrou]

Le Fri, 12 Feb 2010 23:49:38 +0100, Alf P. Steinbach a écrit :

The main reason for not using that term for Python is that "pass by
reference" has the extremely strong connotation of being able to
implement 'swap'.

But 'swap' is so easy to write as a one-line statement that it's foolish
to want to make a Python function for it...

cheers

Antoine.

 

[Gregory Ewing]

Python's calling convention already has an well-established name,
established over thirty years ago by the distinguished computer scientist
Barbara Liskov, namely call-by-sharing.

And she was mistaken in thinking it needed a new name.

 

[rantingrick]

Steve,
Why do so many of your posts come in as doubles and triples. Is this a
case of "studdering click finger" of some fault of your newsreader?

-- concerned fellow pythonista...

 

[Robert Kern]

Le Fri, 12 Feb 2010 23:49:38 +0100, Alf P. Steinbach a écrit :

But 'swap' is so easy to write as a one-line statement that it's foolish
to want to make a Python function for it...

Yes, but that's besides the point. The exercise at hand is to classify the
semantic behavior of the function call syntax with respect to its arguments.
Being able to implement swap() as a function is a distinguishing feature of the
kind of function call semantics that the computer science community has named
"call by reference."

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Gregory Ewing]

It's just that assignment ("=") means a different thing in Python than in
non-object languages (or fake-object languages such as C++ or PHP): it
rebinds instead of mutating in-place. If it mutated, you wouldn't have
the AssertionError.

It doesn't really have anything to do with assignment semantics.
The essence of pass-by-reference is that the formal parameter
name becomes an alias for the actual parameter in all respects.
Whatever the semantics of assignment, you can come up with
variations of the language that either do or don't support
pass-by-reference in this sense. It's an orthogonal issue.

 

[Steve Holden]

Steve,
Why do so many of your posts come in as doubles and triples. Is this a
case of "studdering click finger" of some fault of your newsreader?

-- concerned fellow pythonista...

I suspect it's because I am being insufficiently disciplined about using
"reply" instead of "reply all". On gmane they only appear once, so I
suspect some other component of the mail/news loop is less assiduous
about de-duping the content.

How do you read the list?

regards
Steve

 

[Steve Howell]

I suspect it's because I am being insufficiently disciplined about using
"reply" instead of "reply all". On gmane they only appear once, so I
suspect some other component of the mail/news loop is less assiduous
about de-duping the content.

How do you read the list?

Hi Steve, I mostly read the list through Google groups, and I have
seen occasional dups from you. Of course, the readers themselves
could be a little smarter about recognizing duplication, but that does
not appear to be the case with Google. On the other hand, I am not
seeing dups from you in a quick skim of this thread, so I wonder if it
just takes Google a little while to weed out the dups.

 

[Steve Howell]

My Python program now consumes over 2 GB of memory and then I get a
MemoryError.  I know I am reading lots of files into memory, but not 2GB
worth.
2.    When do I need
to manually allocate/deallocate memory and when can I trust Python to
take care of it?

You never need to manually allocate memory.

You *may* need to deallocate memory if you make "reference loops", where
one object refers to itself:

l = []  # make an empty list
l.append(l)  # add the list l to itself

Python can break such simple reference loops itself, but for more
complicated ones, you may need to break them yourself:

a = []
b = {2: a}
c = (None, b)
d = [1, 'z', c]
a.append(d)  # a reference loop

Python will deallocate objects when they are no longer in use. They are
always considered in use any time you have them assigned to a name, or in
a list or dict or other structure which is in use.

You can explicitly remove a name with the del command. For example:

x = ['my', 'data']
del x

After deleting the name x, the list object itself is no longer in use
anywhere and Python will deallocate it. But consider:

x = ['my', 'data']
y = x  # y now refers to THE SAME list object
del x

Although you have deleted the name x, the list object is still bound to
the name y, and so Python will *not* deallocate the list.

Likewise:

x = ['my', 'data']
y = [None, 1, x, 'hello world']
del x

Although now the list isn't bound to a name, it is inside another list,
and so Python will not deallocate it.

Another technique that comes up some time is that you have a list of
objects that you are processing:

x = [obj1, obj2, obj3, obj4]

When you are done processing obj1, you want to remove the reference to
it, but you do not necessarily want to change the list itself. You
can break the reference by saying "x[0] = None" when you are done
handling obj1. Of course, if you can avoid creating the list in the
first place, as some people have suggested, then you really get a
savings.

The setting-to-None technique is also occasionally useful with
objects, where you can say foo.bar = None when you are done with "bar"
but not with "foo." Of course, the need to use such a technique often
points out a deeper code smell with Foo itself, but I've seen it come
up.

Steven's examples of a reference loop are deliberately simplified, of
course, but the chain of references in a real world program can get
quite long, and there's often great savings to be reaped if you can
break the "keystone" reference, so to speak. In other words, breaking
just one reference often allows other references to fall down like
dominos.