Cleaning up after failing to contructing objects

[brasse]

Hello!

I have been thinking about how write exception safe constructors in
Python. By exception safe I mean a constructor that does not leak
resources when an exception is raised within it. The following is an
example of one possible way to do it:

class Foo(object):
def __init__(self, name, fail=False):
self.name = name
if not fail:
print '%s.__init__(%s)' % (self.__class__.__name__,
self.name)
else:
print '%s.__init__(%s), FAIL' % (self.__class__.__name__,
self.name)
raise Exception()

def close(self):
print '%s.close(%s)' % (self.__class__.__name__, self.name)

class Bar(object):
def __init__(self):
try:
self.a = Foo('a')
self.b = Foo('b', fail=True)
except:
self.close()


def close(self):
if hasattr(self, 'a'):
self.a.close()
if hasattr(self, 'b'):
self.b.close()

bar = Bar()

As you can see this is less than straight forward. Is there some kind
of best practice that I'm not aware of?

:.:: mattias

 

[Gabriel Genellina]

En Mon, 06 Jul 2009 18:15:44 -0300, Scott David Daniels

Not so tough. Something like this tweaked version of your example:

Another variant: Presumably, there is already a method in Bar responsible
for "normal" cleanup; just make sure it gets called (and write it in a
robust way):

class Foo(object):
def __init__(self, name, fail=False):
self.name = name
if not fail:
print '%s.__init__(%s)' % (type(self).__name__, name)
else:
print '%s.__init__(%s), FAIL' % (type(self).__name__, name)
raise ValueError('Asked to fail: %r' % fail)

def close(self):
print '%s.close(%s)' % (type(self).__name__, self.name)

class Bar(object):
a = None # default values
b = None

def __init__(self):
try:
self.a = Foo('a')
self.b = Foo('b', fail=True)
except Exception, why:
self.cleanup()
raise

def cleanup(self):
if self.a is not None:
self.a.close()
if self.b is not None:
self.b.close()

bar = Bar()

 

[Mattias Brändström]

I have been thinking about how write exception safe constructors in
Python. By exception safe I mean a constructor that does not leak
resources when an exception is raised within it.

...
 > As you can see this is less than straight forward. Is there some kind
 > of best practice that I'm not aware of?

Not so tough.  Something like this tweaked version of your example:

class Foo(object):
     def __init__(self, name, fail=False):
         self.name = name
         if not fail:
             print '%s.__init__(%s)' % (type(self).__name__, name)
         else:
             print '%s.__init__(%s), FAIL' % (type(self).__name__, name)
             raise ValueError('Asked to fail: %r' % fail)

     def close(self):
         print '%s.close(%s)' % (type(self).__name__, self.name)

class Bar(object):
     def __init__(self):
         unwind = []
         try:
             self.a = Foo('a')
             unwind.append(a)
             self.b = Foo('b', fail=True)
             unwind.append(b)
             ...
         except Exception, why:
             while unwind):
                 unwind.pop().close()
             raise

bar = Bar()

OK. That's another way. I have been thinking about this some more and
right now I am experimenting with a decorator that could help me do
these kinds of things. This is what I have right now:

import functools

class Foo(object):
def __init__(self, name, fail=False):
self.name = name
self.closed = False
if not fail:
print '%s.__init__(%s)' % (self.__class__.__name__,
self.name)
else:
print '%s.__init__(%s), FAIL' % (self.__class__.__name__,
self.name)
raise Exception()

def close(self):
print '%s.close(%s)' % (self.__class__.__name__, self.name)

def safe(f):
@functools.wraps(f)
def wrapper(self, *args, **kwargs):
variables = []
def recorder(self, name, value):
if name != '__class__':
variables.append(name)
object.__setattr__(self, name, value)

class Customized(object): pass
setattr(Customized, '__setattr__', recorder)
old_class = self.__class__
self.__class__ = Customized

try:
f(self, *args, **kwargs)
self.__class__ = old_class
except:
# clean up objects created in __init__ (i.e. the call to f
())
for name in reversed(variables):
o = getattr(self, name)
if hasattr(o, 'close'):
o.close()
raise
return wrapper

class Bar(object):
@safe
def __init__(self):
self.a = Foo('a')
self.b = Foo('b')
self.c = Foo('c', fail=True)

bar = Bar()

The safe decorator will record all attributes created on self. If an
exception is raised during the execution of the decorated constructor
it will call close on all the recorded objects. I realize that this
decorator still needs a lot of work before it is usable, but I think
it illustrates the concept. It might even be possible for the
decorator to create a close/cleanup method dynamically.

To have a decorator like this that actually worked would be great
since it would remove the need to write error prone cleanup code. Any
thoughts?

:.:: mattias

 

[Lie Ryan]

Hello!
I have been thinking about how write exception safe constructors in
Python. By exception safe I mean a constructor that does not leak
resources when an exception is raised within it. The following is an
example of one possible way to do it:

First, your automatic cleaning Bar() silently pass an unitialized Bar()
into the calling code. When a code fails, it should fail as loud as
possible. Bar() should raises an Exception and the calling code should
turn into something like:

try:
bar = Bar()
except Exception, e:
print 'bar failed to construct'

And about the cleaning up, how about:

class CleanWrap(object):
def __init__(self, cls, cleanup):
self.cls = cls
self.cleanup = cleanup
def __call__(self, *args, **kargs):
try:
return self.cls(*args, **kargs)
except:
self.cleanup()
raise

class Bar(object):
def __init__(self):
CleanWrappedFoo = CleanWrap(Foo, self.close)
self.a = CleanWrappedFoo('a')
self.b = CleanWrappedFoo('b', fail=True)
def close(self):
if hasattr(self, 'a'):
self.a.close()
if hasattr(self, 'b'):
self.b.close()

try:
bar = Bar()
except Exception, e:
print 'Bar() failed to construct'

==========
My next attempt is pretty neat, using a totally different approach, see
the docstring for details:

class Foo1(object):
def __init__(self, name, fail=False):
self.name = name
cls_name = self.__class__.__name__
if not fail:
print '%s.__init__(%s)' % (cls_name, self.name)
else:
print '%s.__init__(%s), FAIL' % (cls_name, self.name)
raise Exception()

def close(self):
print '%s.close(%s)' % (self.__class__.__name__, self.name)

class Foo2(object):
def __init__(self, name, fail=False):
self.name = name
cls_name = self.__class__.__name__
if not fail:
print '%s.__init__(%s)' % (cls_name, self.name)
else:
print '%s.__init__(%s), FAIL' % (cls_name, self.name)
raise Exception()

def close(self):
print '%s.close(%s)' % (self.__class__.__name__, self.name)


class CleanList(object):
''' Each CleanList() instance is rigged so that if exceptions happen
in the same CleanList() instance's wrapped class, all objects
created from the wrapped classes in the same CleanList()
instance will be cleaned (see "Usage" for much better
explanation).

Usage:
cleaner.cleanall() will clean a, b, and c but not d
exceptions in (F|G|H).__init__ will trigger
cleaner.cleanall()

Can be subclassed if you want to override the conditions
that determines the triggering of cleanups
'''
def wrap(self, cls):
''' Wrapper factory that customizes Wrapper's subclass
'''
class Wrapper(cls):
''' Wraps the class' __init__ with cleanup guard.
Subclasses cls to simulate cls as close as possible
'''

# change __class__ to make printing prettier
# e.g. Foo1.__init__ instead of Wrapper.__init__
# probably should be removed
# I'm not sure of the side effects of changing __class__
__class__ = cls

def __init__(self_in, *args, **kargs):
try:
sup = super(Wrapper, self_in)
ret = sup.__init__(*args, **kargs)
except:
self.cleanall()
raise
else:
self.add_to_list(cls, self_in)
return ret
return Wrapper

def __init__(self):
self.cleaners = {}

def __call__(self, cls, cleanup):
''' wraps the class constructor '''
# cleanup, []:
# cleanup is the function called to clean
# [] is the object list for that `cleanup` function
# may not be the best data structure, but it works...
self.cleaners[cls] = cleanup, []
return self.wrap(cls)

def cleanall(self):
''' clean all objects '''
for cleaner, insts in self.cleaners.values():
for inst in insts:
cleaner(inst)

def add_to_list(self, cls, inst):
''' add objects to the cleanup list '''
self.cleaners[cls][1].append(inst)

class Bar(object):
def __init__(self):
clist = CleanList()
otherclist = CleanList()
CleanFoo1 = clist(Foo1, Foo1.close)
CleanFoo2 = clist(Foo2, Foo2.close)
OtherCleanFoo1 = otherclist(Foo1, Foo1.close)
self.a = CleanFoo1('a')
self.b = CleanFoo2('b')
# self.c should not be close()ed
self.c = OtherCleanFoo1('c')
self.d = CleanFoo1('d', fail=True)
self.e = CleanFoo2('e')

class Car(object):
def __init__(self):
Clean = CleanList()
CleanFoo1 = Clean(Foo1, Foo1.close)
CleanFoo2 = Clean(Foo2, Foo2.close)
self.a = CleanFoo1('a')
self.b = CleanFoo2('b')
self.c = CleanFoo1('c')
self.d = CleanFoo2('d')

try:
bar = Car()
except Exception, e:
print e
print 'Car() failed to construct'
print
try:
bar = Bar()
except Exception, e:
print e
print 'Bar() failed to construct'

 

[Mattias Brändström]

First, your automatic cleaning Bar() silently pass an unitialized Bar()
into the calling code. When a code fails, it should fail as loud as
possible. Bar() should raises an Exception and the calling code should
turn into something like:

try:
    bar = Bar()
except Exception, e:
    print 'bar failed to construct'

I know, I missed that little detail.

And about the cleaning up, how about:

class CleanWrap(object):
    def __init__(self, cls, cleanup):
        self.cls = cls
        self.cleanup = cleanup
    def __call__(self, *args, **kargs):
        try:
            return self.cls(*args, **kargs)
        except:
            self.cleanup()
            raise

class Bar(object):
    def __init__(self):
        CleanWrappedFoo = CleanWrap(Foo, self.close)
        self.a = CleanWrappedFoo('a')
        self.b = CleanWrappedFoo('b', fail=True)
    def close(self):
        if hasattr(self, 'a'):
            self.a.close()
        if hasattr(self, 'b'):
            self.b.close()

I think this example adds about as much overhead as my original
example with the try block. Ideally I would like to be able to write
my classes in the most straight forward manner possible. In my mind
that would be something like a naive constructor and a close/dispose
method:

def __init__(self):
self.a = Foo()
self.b = Bar()

def close(self):
self.a.close()
self.b.close()

See my other post (the one with the decorator named safe) in this
thread for an experiment I thought was promising for a while. However
there are some some use cases it has to cover to be useful:

(1) References to objects owned by someone else
def __init__(self, some_resource):
self.not_my_responsibility = some_resource
self.a = Foo()
# If Foo.__init__ raises an exception self.not_my responsibility
should not be closed.

(2) Local objects
def __init__(self):
x = Bar()
self.a = Foo(x)
# If Foo.__init__ raises an exception x should also be closed.

So far I have not been able to find any good solutions for (at least)
case 1. I'll give it a few more days, but I'm not that hopeful. I
think that I will end up writing my classes very much like Bar in my
original post. If someone has any ideas for how to solve my two use
cases above with my decorator based approach I would really like to
hear them!

:.:: mattias

 

[Shai]

Since nobody else mentioned this...

Python classes have a magic method called __del__ which is usually
called just before an object is garbage-collected. Further, Python
uses reference-counting to tell when an object is no longer
accessible. This means that if your resource classes define __del__
methods, these will be called properly when the object using them is
destroyed, and you don't need to write an explicit close() method for
this.

class Resource(object):
def __init__(self, param):
# acquire resource

def __del__(self):
# release resource

not_my_responsibility = Resource(1)

class Foo(object):
def __init__(self):
self.ref = not_my_responsibility # self.ref.__del__() will not be
called as long as the module exists
local = Resource(2) # local.__del__() will be called as soon as
__init__ is exited
self.held = Resource(3) # self.held.__del__() will be called when
the object dies
z = 1/0 # The exception in the initializer will kill the object,
triggering some Resource.__del__() calls

There are two caveats:

1) __del__ methods prevent instances of your class from being
collected when they are involved in cyclical structures; this means if
your structures start to get complex (sometimes a doubly-linked list
is complex enough), you may find yourself leaking memory.

2) The bit about reference counting, which allows predictable
destruction, holds for CPython, but not for Jython, and IIRC also not
for IronPython (I don't know about PyPy or other implementations). It
is a feature of the reference implementation, not the language
definition.