does python have useless destructors?

[Roy Smith]

1. Objects of classes that declare a __del__ method shall be referred
to as "deterministic" objects. Such objects shall have a reference
count associated with. The reference count shall be incremented
atomically each time an additional reference to the object is created.
The reference count shall be decremented each time a name referring
to the object is deleted explicitly. Except in the situation
described in (2) below, the reference count shall be decremented each
time a name referring to the object becomes inaccessible due to the
set of locals to which the name belongs becoming inaccessible.
[...]
3. When the reference count of a deterministic object reaches zero,
the __del__ method of the object shall be called.

What if you do this...

class Top:
def __init__ (self):
self.bottom = Bottom ()

class Bottom:
def __del__ (self):
do something really important

top = Top ()
del top

The problem here is that while Bottom.__del__ will get called as soon at
top releases it's reference, there's nothing to guarantee that the
reference will disappear until top is garbage collected. You could fix
that by writing Top.__del__, but that assumes Top is a class you have
control of (it might be something out of a library).

Or am I not understanding the situation right?

This particular example you cite is not actually problematic. The
point is that the resources are cleaned up when Top is cleaned up,
which is what we want. How and when Top is cleaned is neither here
nor there.

Well, ok, let's change it a little. Let's do:

# Top is unchanged from earlier example ...
class Top:
def __init__ (self):
self.bottom = Bottom ()

# .... but Bottom is a little different ...
class Bottom:
def __init__ (self):
self.exclusiveAccessResource = getResource ()

def __del__ (self):
del self.exclusiveAccessResource

# ... and the calling sequence changes too
top = Top ()
del top
top = Top ()

Now, I've designed Top to be a sort of bastard cousin (or maybe an evil
twin?) of a singleton. Instead of getting the same one each time, you
get a new one but you're only allowed to create one of them at a time.
The exclusiveAccessResource might be something like a mutex, or it might
be something external like a low-numbered network port. If the "del
Top" doesn't actually free the resource, the second call to Top() will
fail.

 

[Humpty Dumpty]

This is an unrealistic requirement. It implies that all access
to some resource must occur within a single function, which is
too great a limitation for many real applications.

As long as only one ref left on function exit, there is no limitation of
access.

Oliver

 

[Terry Reedy]

Resource Acquisition is Initialization. ,,,
It comes from a common method in C++ of using stack-allocated objects
to own or manage resources, because of the explicit construction and
destruction semantics of C++ objects (and their members) during ....
I'm not sure if RAII is intended to also cover non-stack based
objects, in terms handing ownership of the object reference equating

Thanks. This makes it much clearer what behavior some people miss, and why
the behavior is hard to simulate in CPython (all PyObjects are heap
allocated).

Terry J. Reedy

 

[David Turner]

There's a PEP for something like that in Python:

http://www.python.org/peps/pep-0310.html

The PEP in question describes the "with" statement, which is more or
less analogous to the C# "using" statement.

There is a serious problem with this approach.

The serious problem is that the end-user has to remember to use it.
Yes, it's less typing than try/finally, but it's *still not obvious
when you have to use it*.

The huge advantage that the RAII approach holds in this respect is
that the user of the library just does what comes naturally - for
example, he declares a file object and uses it. He would have done
that anyway. He doesn't need to know whether or not it's a RAII
object that needs a "with" or "using" or "dispose" or "try/finally"
clause.

"With" will save typing, but it won't eliminate the important class of
errors that stem from the user's ignorance of the dispose semantics of
the object he's using. Nor will it eliminate the "careless" errors
that we're all prone to - 30 years' experience notwithstanding.

Well-defined destructor semantics have proven to be a robust,
reliable, and surprisingly general solution to a wide range of
problems. It wouldn't be *that* hard to implement them in Python - so
why not adopt a good idea?

I think that's greatly preferable to dictating that the GC system act
a certain way and free resources as soon as they go out of scope.
That's just not the correct semantics for GC. GC should simply create
the illusion for the application that all objects stay around forever.

Also, I'd like to point out that destructor semantics and GC are not
necessarily related. There's no rule that says the destructor has to
be called at the same time as the memory is freed. In fact, there are
several good reasons for separating the events.

One could conceivably even use a pre-processor to implement Python
deterministic destructor semantics in Python. It's really just a case
of the compiler inserting a number of implicit try/finally blocks, and
reference counting. I can't see any reason why it couldn't be done in
Jython.

Regards
David Turner

 

[David Turner]

Hi

Now, I've designed Top to be a sort of bastard cousin (or maybe an evil
twin?) of a singleton. Instead of getting the same one each time, you
get a new one but you're only allowed to create one of them at a time.
The exclusiveAccessResource might be something like a mutex, or it might
be something external like a low-numbered network port. If the "del
Top" doesn't actually free the resource, the second call to Top() will
fail.

I'm not sure that this is relevant. You know that Top is not a
deterministic object; therefore you know you can't be sure when it (or
anything it owns) should be destroyed. What you *do* know is that it
will take its Bottom along with it when it goes. That's the invariant
we're trying to establish here.

Generally one addresses the problem of the resource-safety of a class
when one is designing it. Top is not resource-safe, and that's a
problem that can't (and shouldn't) be addressed post-hoc.

Regards
David Turner

 

[Duncan Booth]

You have totally confused me now. Ignoring special cases such as
cycles, Python destructs an object when there are no references left
to it, and in CPython that happens at the moment the last reference is
released. So in normal code the objects will go away as names
referencing them go out of scope.

I wouldn't say that cycles are special cases exactly.

C Python destroys an object when no references are left. Jython and
IronPython destroy objects when the underlying garbage collector feels like
it (i.e. when the object is no longer reachable, which even without cycles
is not the same as having no references). In most, if not all of these
cases the underlying garbage collector is generational, so you can't even
be sure that unreachable objects will be destroyed by the next collection,
it may take several collections. Python (as a language) is careful not to
specify a specific implementation (such as reference counting) because that
would prevent efficient implementation on a variety of platforms.

If the Python object is a proxy for another non-Python visible object
(such as a C level filehandle or GUI Window). In those cases
the extension module author has to deal with the situation, either by
adding an extra reference (the C level object is then the remaining
reference) or handling the lifetime of the C level object
independently.

But for this thread, the actual problem is that __del__ methods may
not be called on objects, and if you read the doc implies they almost
never will be called. So we have the counter-intuitive (and IMHO user
hostile) situation where someone marks a class as needing extra code
to run to destroy it, and Python addresses that by saying you will
be lucky if the object will be destroyed (ie by not destroying the
object).

No, Python guarantees that the object will (almost always) be destroyed. It
just doesn't make guarantees about when that will happen. If you want
guarantees about resources being released you have to write the code to do
that yourself (e.g. with try..finally). The same applies for other
languages such as Java, or languages running in Microsoft's .Net framework.

Guaranteeing that all resources will be released at a specific time has
implications for performance, and finalisers are actually pretty rare in
practice, so the usual solution is to compromise.

 

[Isaac To]

David> Also, I'd like to point out that destructor semantics and GC are
David> not necessarily related. There's no rule that says the
David> destructor has to be called at the same time as the memory is
David> freed. In fact, there are several good reasons for separating
David> the events.

David> One could conceivably even use a pre-processor to implement
David> Python deterministic destructor semantics in Python. It's really
David> just a case of the compiler inserting a number of implicit
David> try/finally blocks, and reference counting. I can't see any
David> reason why it couldn't be done in Jython.

Unluckily, currently there is no concept of "local scoped objects" in
Python. What it means is that a variable can be local, but an object
cannot. An object in Python is always heap allocated, so there is no way to
know that whether an object has a reference staying outside the current
scope when the current scope exits---short of dictating how the garbage
collector works (i.e., it must count reference). I think the primary
developer in Python has already completely rejected the very idea, probably
because it is impractical to implement in Jython.

Regards,
Isaac.

 

[Isaac To]

David> I'm not sure if RAII is intended to also cover non-stack based
David> objects, in terms handing ownership of the object reference
David> equating transferring ownership of the resource. Certainly the
David> classic example is the stack based approach.

One can do it in C++ using auto_ptr. The basic use case looks like this:

#include <memory>
#include <iostream>
using namespace std;

struct base_t {
virtual ~base_t() { cout << "Destroying base_t" << endl; }
// something complicated
};

struct derived_t: public base_t {
~derived_t() { cout << "Destroying derived_t" << endl; }
};

base_t *base_factory() {
return new derived_t;
}

void f() {
auto_ptr<base_t> p(base_factory()); // heap allocated, function scoped
}

int main() {
cout << "Call f" << endl;
f();
cout << "Continue main" << endl;
}

Note that in f(), the memory returned by base_factory() is heap allocated,
but function scoped. I.e., upon leaving function f(), either due to
function return or exception being thrown, the memory is deallocated. This
is usually used like above, to achieve polymorphism when otherwise one would
want to use a function scoped object. There is a sense of ownership: when
you do auto-pointer assignment, the original owner lose the ownership, and
the corresponding auto-pointer immediately becomes a NULL pointer. So

void f() {
auto_ptr<base_t> p(base_factory());
auto_ptr<base_t> q = p; // p is now NULL auto-pointer
}

This makes auto-pointer somewhat difficult to use, and unsuitable for
anything that try to do assignment (e.g., one is committing suicide if one
creates a collection of auto-pointers). Next standard (C++0x) probably will
introduce reference counted version, hopefully easier to use.

Regards,
Isaac.

 

[Donn Cave]

Quoth Duncan Booth <[email protected]>:
|
....
|> But for this thread, the actual problem is that __del__ methods may
|> not be called on objects, and if you read the doc implies they almost
|> never will be called. So we have the counter-intuitive (and IMHO user
|> hostile) situation where someone marks a class as needing extra code
|> to run to destroy it, and Python addresses that by saying you will
|> be lucky if the object will be destroyed (ie by not destroying the
|> object).
|
| No, Python guarantees that the object will (almost always) be destroyed. It
| just doesn't make guarantees about when that will happen. If you want
| guarantees about resources being released you have to write the code to do
| that yourself (e.g. with try..finally). The same applies for other
| languages such as Java, or languages running in Microsoft's .Net framework.

Er, Python guarantees that the object will almost always be destroyed?
What the heck does that mean?

| Guaranteeing that all resources will be released at a specific time has
| implications for performance, and finalisers are actually pretty rare in
| practice, so the usual solution is to compromise.

It isn't clear to me what it takes to guarantee immediate finalization,
so if it is to you, that might be worth spelling out a little more.
It doesn't cost anything in the ordinary case, I mean with C Python's
reference count storage model, if you get it, you get it for free.

As for whether finalizers are rare in practice, of course they would be
in languages like Java where they're evidently about useless, and maybe
in Python where they're of limited value and more or less deprecated.
It's kind of a circular argument for leaving it broken, though. (Maybe
broken is the wrong word - finalization really does work. It can take
some extra care to make it work reliably, but that extra attention to
releasing exception tracebacks, avoiding cycles, etc., could be not
altogether a bad thing.)

Donn Cave, (e-mail address removed)

 

[Roger Binns]

I wouldn't say that cycles are special cases exactly.

They are to the point I was trying to make, which is that objects
are freed when there are no references left in the CPython implementation.

The rest of your comment is missing my point. I don't particularly
care about *when* an object is garbage collected, just that *is*
garbage collected at some point.

The whole problem that this thread is about is that Python has this
bizarre scheme that an object will be garbage collected *unless* you
add a __del__ method, at which point the docs imply you will be lucky
for garbage collection to *ever* happen on the object.

No, Python guarantees that the object will (almost always) be destroyed.

The issue is the cases when it doesn't destroy objects. That is the
whole point of this discussion and I refer you to the very first
message:

http://groups.google.com/[email protected]

Guaranteeing that all resources will be released at a specific time has
implications for performance, and finalisers are actually pretty rare in
practice, so the usual solution is to compromise.

It isn't a timing issue. It is a "will it ever happen" issue.

Roger

 

[Slawomir Nowaczyk]

On 12 Jun 2004 03:21:26 -0700
(e-mail address removed) (David Turner) wrote:

#> Well-defined destructor semantics have proven to be a robust,
#> reliable, and surprisingly general solution to a wide range of
#> problems. It wouldn't be *that* hard to implement them in Python -
#> so why not adopt a good idea?

I suppose most people would agree that well-defined destructor
semantic is a good thing to have. And those who praise try/finally
construct are just trying to say that you can use it to do what you
want, even if in a bit more complicated way.

The main problem, I suppose, is that it *would be* pretty hard to
implement RAII in an efficient and portable way.

Not impossible, of course, but hard.

--
Best wishes,
Slawomir Nowaczyk
( (e-mail address removed) )

Only drug dealers and software companies call their customers 'users.'

 

[Humpty Dumpty]

Unluckily, currently there is no concept of "local scoped objects" in
Python. What it means is that a variable can be local, but an object
cannot. An object in Python is always heap allocated, so there is no way to
know that whether an object has a reference staying outside the current
scope when the current scope exits---short of dictating how the garbage
collector works (i.e., it must count reference). I think the primary
developer in Python has already completely rejected the very idea, probably
because it is impractical to implement in Jython.

I have been hearing a lot of reference to Jython. This is yet another
example how coupling languages can stifle their progress: C++ is stifled by
its need for compatilibity with C, now clearly Python is becoming stifled by
a need for compatibility with Jython.
Oliver

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

As soon as I heard about not being able to trust destructors, I was
shocked, because I do rely on guaranteed finalization, and I also
consider it to be very elegant.

"Guarantee" and "able to rely on" are different things, actually.
A guarantee is something that someone gives you, which they might
not do even though they could. For example, you can rely on sunset
to occur before midnight, but nobody might give you a guarantee for
that.

So the Python language specification does not guarantee anything about
invoking __del__. However, you can still rely on C-Python 2.3.4
invoking it eventually. More precisely, C-Python 2.3.4 (and most other
releases of C-Python) will invoke __del__ if the last reference to
an object goes away. A reference goes away if:
- the variable is del'ed, or a different value is assigned, or
- the variable is a local variable, and the function terminates
through a return (if the function terminates through an exception,
a traceback object is constructed which takes over the local
variable).
- the variable is attribute of an object, and the object goes away
- the variable is an implicit variable in the interpreter, and
gets a new value. Some of the implicit variables are:
- the current exception and traceback
- the last exception and traceback
- the sys module
- the codecs module

myfile = open("myfilepath", "w")
myfile.write(reallybigbuffer)
myfile.close()

If the write fails due to a lack of disk space, the exception will
prevent the explicit close() from happening. Now, if myfile goes out of
scope, I would hope that the file object is destroyed and thus the file
is closed, but after reading the python documentation, it seems that
the only way to guarantee closure of the file is using the mentioned
try/finally construct...

C-Python 2.3.4 will close the fil if myfile goes out of scope, unless
there is an exception, in which case myfile is referred to in the
traceback, in which case the file is closed when the traceback object
is released, which happens when the exception handler for the exception
has completed. If the exception was put out through PyErr_Print, the
object stays alive through sys.last_traceback, where it stays until
the next exception occurs.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

I would do the same thing that shutdown does. Run
the destructors, but don't delete the underlying
memory. Set all the fields to None (just as shutdown
points modules at None). Finally in a second round
actually free the memory.

At what point would you do the shutdown, and to what
objects would you apply it?

"the same thing as shutdown" means that you clear out
all modules. If the answer to the first question is
"at the end of the function", then any Python application
will crash as soon as the first function returns, as
then all modules will be useless.

Regards,
Martin

 

[Carl Banks]

Well-defined destructor semantics have proven to be a robust,
reliable, and surprisingly general solution to a wide range of
problems. It wouldn't be *that* hard to implement them in Python - so
why not adopt a good idea?

Wrong. It would be almost impossible in Python.


Python can never rely on someone not taking a reference to your
resource-owning object. Anyone can store an open file object in a
list, for example, and there might even be good reasons to do it.
Thus, you can never guarantee that the garbage collector will pick up
the object when the function exits, even if it were perfect.

OTOH, if you indiscreetly finalize (for example, calling close on a
file object) any resource-owning object as soon as the function that
created it exits, not waiting for garbage collector to pick it up,
then you've pretty much stepped on the toes of someone else
it.


It seems to me that you are failing to taking the possibility that
these thing can happen into account. But, in Python, they can and do.
Finalizing an object as soon as the name binding it goes out of scope
is not always the right thing to do.

The fact is: when you open a file, or acquire a lock, or obtain any
other resource, you, the programmer, have to know whether you want
that resource released when the name bound to the object that owns the
resource goes out of scope or not. The language can only guess; it
happens to guess "yes" in C++ and "no" in Python. When the guess is
wrong, you have to tell the language.

Because of the above problems, "no" is the correct guess in Python.
Forcing the programmer to explicitly release resources (excepting
resources that don't require timely release, like RAM) is a lesser
evil than the two results above.

 

[Marcin 'Qrczak' Kowalczyk]

I have been hearing a lot of reference to Jython. This is yet another
example how coupling languages can stifle their progress: C++ is stifled by
its need for compatilibity with C, now clearly Python is becoming stifled by
a need for compatibility with Jython.

I have a different point of view.

Relying on timely finalization is stifling progress of the Python runtime.
It prevents using better, more efficient GC techniques, which all happen
to not have this property, and it prevents porting Python to other
architectures like JVM and .NET.

It's better to drop compatibility with this artifact of the poor but
conceptually simple GC scheme used in CPython, and declare programs which
rely on that implementation detail as broken.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

As soon as I heard about not being able to trust destructors, I was
shocked, because I do rely on guaranteed finalization, and I also
consider it to be very elegant.

"Guarantee" and "able to rely on" are different things, actually.
A guarantee is something that someone gives you, which they might
not do even though they could. For example, you can rely on sunset
to occur before midnight, but nobody might give you a guarantee for
that.

So the Python language specification does not guarantee anything about
invoking __del__. However, you can still rely on C-Python 2.3.4
invoking it eventually. More precisely, C-Python 2.3.4 (and most other
releases of C-Python) will invoke __del__ if the last reference to
an object goes away. A reference goes away if:
- the variable is del'ed, or a different value is assigned, or
- the variable is a local variable, and the function terminates
through a return (if the function terminates through an exception,
a traceback object is constructed which takes over the local
variable).
- the variable is attribute of an object, and the object goes away
- the variable is an implicit variable in the interpreter, and
gets a new value. Some of the implicit variables are:
- the current exception and traceback
- the last exception and traceback
- the sys module
- the codecs module

myfile = open("myfilepath", "w")
myfile.write(reallybigbuffer)
myfile.close()

If the write fails due to a lack of disk space, the exception will
prevent the explicit close() from happening. Now, if myfile goes out of
scope, I would hope that the file object is destroyed and thus the file
is closed, but after reading the python documentation, it seems that
the only way to guarantee closure of the file is using the mentioned
try/finally construct...

C-Python 2.3.4 will close the fil if myfile goes out of scope, unless
there is an exception, in which case myfile is referred to in the
traceback, in which case the file is closed when the traceback object
is released, which happens when the exception handler for the exception
has completed. If the exception was put out through PyErr_Print, the
object stays alive through sys.last_traceback, where it stays until
the next exception occurs.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

As soon as I heard about not being able to trust destructors, I was
shocked, because I do rely on guaranteed finalization, and I also
consider it to be very elegant.

"Guarantee" and "able to rely on" are different things, actually.
A guarantee is something that someone gives you, which they might
not do even though they could. For example, you can rely on sunset
to occur before midnight, but nobody might give you a guarantee for
that.

So the Python language specification does not guarantee anything about
invoking __del__. However, you can still rely on C-Python 2.3.4
invoking it eventually. More precisely, C-Python 2.3.4 (and most other
releases of C-Python) will invoke __del__ if the last reference to
an object goes away. A reference goes away if:
- the variable is del'ed, or a different value is assigned, or
- the variable is a local variable, and the function terminates
through a return (if the function terminates through an exception,
a traceback object is constructed which takes over the local
variable).
- the variable is attribute of an object, and the object goes away
- the variable is an implicit variable in the interpreter, and
gets a new value. Some of the implicit variables are:
- the current exception and traceback
- the last exception and traceback
- the sys module
- the codecs module

myfile = open("myfilepath", "w")
myfile.write(reallybigbuffer)
myfile.close()

If the write fails due to a lack of disk space, the exception will
prevent the explicit close() from happening. Now, if myfile goes out of
scope, I would hope that the file object is destroyed and thus the file
is closed, but after reading the python documentation, it seems that
the only way to guarantee closure of the file is using the mentioned
try/finally construct...

C-Python 2.3.4 will close the fil if myfile goes out of scope, unless
there is an exception, in which case myfile is referred to in the
traceback, in which case the file is closed when the traceback object
is released, which happens when the exception handler for the exception
has completed. If the exception was put out through PyErr_Print, the
object stays alive through sys.last_traceback, where it stays until
the next exception occurs.

Regards,
Martin

 

[Carl Banks]

C-Python 2.3.4 will close the fil if myfile goes out of scope,

Not so fast, my friend. What if reallybigbuffer is an instance of the
following class, defined in the same function:

class reallybigbufferclass(object):
def __str__(self): __builtins__.myfile = myfile; return "sucker!"

Whoops, looks like there's still a reference hanging around. Python
STILL doesn't guarantee this file will be closed upon exit. Even more
perniciously:

class reallybigbufferclass(object):
def __init__(self): self.self = self
def __str__(self): self.myfile = myfile; return "sucker!"
def __del__(self): pass

I think if reallybigbuffer goes out of scope and no one else
references it, then myfile will never be closed, ever. At least not
until someone gets at it using the gc module.

These are silly examples, of course, but with more intricate stuff (or
with code by other people who are always less disciplined than you)
this can become a real problem. Face it, people, it's ludicrous to
rely on the garbage collector to finalize stuff for us.

 

[David Turner]

David> Also, I'd like to point out that destructor semantics and GC are
David> not necessarily related. There's no rule that says the
David> destructor has to be called at the same time as the memory is
David> freed. In fact, there are several good reasons for separating
David> the events.

Unluckily, currently there is no concept of "local scoped objects" in
Python. What it means is that a variable can be local, but an object
cannot. An object in Python is always heap allocated, so there is no way to
know that whether an object has a reference staying outside the current
scope when the current scope exits---short of dictating how the garbage
collector works (i.e., it must count reference). I think the primary
developer in Python has already completely rejected the very idea, probably
because it is impractical to implement in Jython.

Isaac, I think you've missed the main thrust of my suggestion here.
I'm not talking about "locally scoped objects" such as C++ has. I'm
also not talking about modifying the garbage collector (if any). I'm
talking about a mechanism that is *independent* of the garbage
collector. To briefly resummarize:

Objects with a __del__ method shall be reference counted. When the
reference count reaches zero, the __del__ method shall be called, and
any subobjects that have a __del__ method shall also be unreferenced.

The point at which the memory allocated to the object is freed is
largely irrelevant. The point is that there's a predictable time at
which __del__ is called. This is what enables the RAII idiom.

Now, this could be tricky to implement because we are now separating
the concepts of "destruction" and "finalization". But it's certainly
not impossible, and it would add a powerful new concept to the
language. So I don't think the idea should be rejected out of hand.

Is this clearer?

Regards
David Turner