A "scopeguard" for Python

[Steve Howell]

For C++ Petru Marginean once invented the "scope guard" technique (elaborated on
by Andrei Alexandrescu, they published an article about it in DDJ) where all you
need to do to ensure some desired cleanup at the end of a scope, even when the
scope is exited via an exception, is to declare a ScopeGuard w/desired action.

The C++ ScopeGuard was/is for those situations where you don't have proper
classes with automatic cleanup, which happily is seldom the case in good C++
code, but languages like Java and Python don't support automatic cleanup and so
the use case for something like ScopeGuard is ever present.

For use with a 'with' statement and possibly suitable 'lambda' arguments:

<code>
class Cleanup:
     def __init__( self ):
         self._actions = []

     def call( self, action ):
         assert( is_callable( action ) )
         self._actions.append( action )

     def __enter__( self ):
         return self

     def __exit__( self, x_type, x_value, x_traceback ):
         while( len( self._actions ) != 0 ):
             try:
                 self._actions.pop()()
             except BaseException as x:
                 raise AssertionError( "Cleanup: exception during cleanup" ) from
</code>

I guess the typical usage would be what I used it for, a case where the cleanup
action (namely, changing back to an original directory) apparently didn't fit
the standard library's support for 'with', like

   with Cleanup as at_cleanup:
       # blah blah
       chdir( somewhere )
       at_cleanup.call( lambda: chdir( original_dir ) )
       # blah blah

Another use case might be where one otherwise would get into very deep nesting
of 'with' statements with every nested 'with' at the end, like a degenerate tree
that for all purposes is a list. Then the above, or some variant, can help to
/flatten/ the structure. To get rid of that silly & annoying nesting.

Cheers,

- Alf (just sharing, it's not seriously tested code)

Hi Alf, I think I understand the notion you're going after here. You
have multiple cleanup steps that you want to defer till the end, and
there is some possibility that things will go wrong along the way, but
you want to clean up as much as you can. And, of course, flatter is
better.

Is this sort of what you are striving for?

class Cleanup:
def __init__( self ):
self._actions = []

def call( self, action ):
self._actions.append( action )

def __enter__( self ):
return self

def __exit__( self, x_type, x_value, x_traceback ):
while( len( self._actions ) != 0 ):
try:
self._actions.pop()()
except BaseException as x:
raise AssertionError( "Cleanup: exception during
cleanup" )

def clean_the_floor():
print('clean the floor')

def carouse(num_bottles, accident):
with Cleanup() as at_cleanup:
at_cleanup.call(clean_the_floor)
for i in range(num_bottles):
def take_down(i=i):
print('take one down', i)
at_cleanup.call(take_down)
if i == accident:
raise Exception('oops!')
print ('put bottle on wall', i)

carouse(10, None)
carouse(5, 3)

 

[Jean-Michel Pichavant]

No, I was referring to Jean-Michel, who was not familiar with the
with: statement.


Also talking about Jean-Michel.

I confirm, I am the ignoramus (what a strange word) in this story

JM

 

[Alf P. Steinbach]

* Robert Kern:

* Robert Kern:

On 2010-03-04 12:37 PM, Alf P. Steinbach wrote:
* Robert Kern:
On 2010-03-04 10:56 AM, Alf P. Steinbach wrote:
* Robert Kern:
On 2010-03-03 18:49 PM, Alf P. Steinbach wrote:
[snippety]

If you call the possibly failing operation "A", then that
systematic
approach goes like this: if A fails, then it has cleaned up its own
mess, but if A succeeds, then it's the responsibility of the
calling
code to clean up if the higher level (multiple statements)
operation
that A is embedded in, fails.

And that's what Marginean's original C++ ScopeGuard was designed
for,
and what the corresponding Python Cleanup class is designed for.

And try: finally:, for that matter.

Not to mention "with".

Some other poster made the same error recently in this thread; it
is a
common fallacy in discussions about programming, to assume that since
the same can be expressed using lower level constructs, those are all
that are required.

If adopted as true it ultimately means the removal of all control
structures above the level of "if" and "goto" (except Python doesn't
have "goto").

What I'm trying to explain is that the with: statement has a use even
if Cleanup doesn't. Arguing that Cleanup doesn't improve on try:
finally: does not mean that the with: statement doesn't improve on
try: finally:.

That's a different argument, essentially that you see no advantage for
your current coding patterns.

It's unconnected to the argument I responded to.

The argument that I responded to, that the possibility of expressing
things at the level of try:finally: means that a higher level construct
is superfluous, is still meaningless.

I am attacking your premise that the "with Cleanup():" construct is
higher level than try: finally:. It isn't. It provides the same level
of abstraction as try: finally:.

This is distinct from the accepted uses of the with: statement which
*are* higher level than try: finally: and which do confer practical
benefits over using try: finally: despite being syntactical sugar for
try: finally:.

Both formulations can be correct (and both work perfectly fine
with
the chdir() example being used). Sometimes one is better than the
other, and sometimes not. You can achieve both ways with either
your
Cleanup class or with try: finally:.

I am still of the opinion that Cleanup is not an improvement over
try:
finally: and has the significant ugliness of forcing cleanup code
into
callables. This significantly limits what you can do in your
cleanup
code.

Uhm, not really. As I see it.

Well, not being able to affect the namespace is a significant
limitation. Sometimes you need to delete objects from the
namespace in
order to ensure that their refcounts go to zero and their cleanup
code
gets executed.

Just a nit (I agree that a lambda can't do this, but as to what's
required): assigning None is sufficient for that[1].

Yes, but no callable is going to allow you to assign None to names in
that namespace, either. Not without sys._getframe() hackery, in any
case.

However, note that the current language doesn't guarantee such
cleanup,
at least as far as I know.

So while it's good practice to support it, to do everything to let it
happen, it's presumably bad practice to rely on it happening.


Tracebacks will keep the namespace alive and all objects in it.

Thanks!, I hadn't thought of connecting that to general cleanup
actions.

It limits the use of general "with" in the same way.

Not really.

Sorry, it limits general 'with' in /exactly/ the same way.

It's easy to write context managers that do that [delete objects from
the namespace].

Sorry, no can do, as far as I know; your following example quoted below
is an example of /something else/.

Okay, so what do you mean by 'the use of general "with"'? I'm talking
about writing a context manager or using the @contextmanager decorator
to do some initialization and then later cleaning up that
initialization. That cleaning up may entail deleting an object. You
are correct that the context manager can't affect the namespace of the
with: clause, but that's not the initialization that it would need to
clean up.

Yes, you can write code with a with: statement where you try to clean
up stuff that happened inside of the clause (you did), but that's not
how the with: statement was ever intended to be used nor is it good
practice to do so because of that limitation. Context managers are
designed to initialize specific things, then clean them up. I thought
you were talking about the uses of the with: statement as described in
PEP-343, not every possible misuse of the with: statement.

I'm not the one talking about removing variables or that "it's easy to
write context managers that do that".

You are the one talking about that.

So I have really not much to add.

It seems that you're now agreeing with me that former is not good
practice and that the latter is impossible to do portably, but you now
argue against your earlier stand as if that was something that I had put
forward.

No, I'm still saying that sometimes you do need to remove variables that
you initialized in the initialization section. Writing a purpose-built
context manager which encapsulates the initialization and finalization
allows you to do this easily. Putting the initialization section inside
the with: clause, as you do, and requiring cleanup code to be put into
callables makes this hard.

It's a bit confusing when you argue against your own statements.


Again, this context shifting is bewildering. As you can see, quoted
above, you were talking about a situation where you would have defined a
context manager, presumably because a 'try' would not in your opinion be
simpler for whatever it was that you had in mind. But you are responding
to the code I offered as if it was an alternative to something where you
would find a 'try' to be simplest.

I have consistently put forward that for once-only cases, try: finally:
is a preferable construct to "with Cleanup():". I also put forward that
for repetitive cases, a purpose-built context manager is preferable. I
note that for both try: finally: and a purpose-built context manager,
modifying the namespace is easy to do while it is difficult and less
readable to do with "with Cleanup():". Since you were claiming that the
"generic with:" would have the same problem as "with Cleanup():" I was
trying to explain for why all of the intended use cases of the with:
statement (the purpose-built context managers), there is no problem.
Capisce?

Sorry, as with the places noted above, I can't understand what you're
trying to say here. I don't recommend coding practices where you keep
object references around,

Then how do you clean up locks and transaction objects and similar
things if you don't keep them around during the code suite? Creating an
object, keeping it around while executing a specific chunk of code, and
finalizing it safely is a prime use case for these kinds of constructs.

Where you need a scope, create a scope.

That is, put the logic in a routine.

However, most objects aren't of the sort the requiring to become unreferenced.
Those that require cleanup can be cleaned up and left as zombies, like calling
'close' on a file. And so in the general case what you're discussing, how to get
rid of object references, is a non-issue -- irrelevant.

These are the contortions:

foo = lambda: None
foo.x = create_some_object()
at_cleanup.call( lambda o = foo: delattr( o, "x" ) )

That's code that demonstrates something very different, in response to your
description of doing this.

Since I half suspected that it could be taken out of context I followed that
immediately with

<quote>
2) it is a usage that I wouldn't recommend; instead I recommend adopting
good coding practices where object references aren't kept around.
</quote>

I'm sorry but I don't know how to make that more clear.


Cheers & hth.,

- Alf

 

[Alf P. Steinbach]

* Robert Kern:

I was just trying to interpret what you meant by "Changing 'finally' to
'else' could make it equivalent."

Oh yes, in the article where I gave the example of that, shown above.

Hey, discussing previous discussion is silly.


Cheers,

- ALf

 

[Mike Kent]

No, the try: finally: is not implicit. See the source for
contextlib.GeneratorContextManager. When __exit__() gets an exception from the
with: block, it will push it into the generator using its .throw() method.. This
raises the exception inside the generator at the yield statement.

Wow, I just learned something new. My understanding of context
managers was that the __exit__ method was guaranteed to be executed
regardless of how the context was left. I have often written my own
context manager classes, giving them the __enter__ and __exit__
methods. I had mistakenly assumed that the @contextmanager decorator
turned a generator function into a context manager with the same
behavior as the equivalent context manager class. Now I learn that,
no, in order to have the 'undo' code executed in the presence of an
exception, you must write your own try/finally block in the generator
function.

This raises the question in my mind: What's the use case for using
@contextmanager rather than wrapping your code in a context manager
class that defines __enter__ and __exit__, if you still have to
manager your own try/finally block?

 

[Steve Howell]

Wow, I just learned something new.  My understanding of context
managers was that the __exit__ method was guaranteed to be executed
regardless of how the context was left.  I have often written my own
context manager classes, giving them the __enter__ and __exit__
methods.  I had mistakenly assumed that the @contextmanager decorator
turned a generator function into a context manager with the same
behavior as the equivalent context manager class.  Now I learn that,
no, in order to have the 'undo' code executed in the presence of an
exception, you must write your own try/finally block in the generator
function.

This raises the question in my mind: What's the use case for using
@contextmanager rather than wrapping your code in a context manager
class that defines __enter__ and __exit__, if you still have to
manager your own try/finally block?

Unless I am misunderstanding the question, the use case is that you
still only have to write the context manager once, and you might get
multiple uses out of it where the with-enclosed code blocks work at a
higher level of abstraction.

I actually don't use @contextmanager yet, mainly because I did not
know it existed until recently, but also because I find the __enter__/
__exit__ paradigm straightforward enough to just hand code them that
way.

 

[Steve Howell]

No, the try: finally: is not implicit. See the source for
contextlib.GeneratorContextManager. When __exit__() gets an exception from the
with: block, it will push it into the generator using its .throw() method.. This
raises the exception inside the generator at the yield statement.

See also:

http://docs.python.org/library/contextlib.html

The closing() helper can be used to automatically call thing.close()
even after an exception.

If you do not use the closing() helper and take on the responsibility
yourself of doing try/finally within your generator, I think you still
gain some overall simplicity:

1) You don't need to do try/finally in your with blocks, of course.
2) The generator itself probably reads more straightforwardly then a
hand-coded class with __enter__ and __exit__.

For point #2, I think there are probably different aesthetics.
Generators are more concise, but they are also a bit mind-bending.

 

[Robert Kern]

Wow, I just learned something new. My understanding of context
managers was that the __exit__ method was guaranteed to be executed
regardless of how the context was left.

It is. @contextmanager turns a specially-written generator into a context
manager with an __exit__ that does different things depending on whether or not
and exception was raised. By pushing the exception into the generator, it lets
the author decide what to do. It may catch a subset of exceptions, or no
exceptions, or use a finally:. They all have use cases although finally: is the
usual one.

I have often written my own
context manager classes, giving them the __enter__ and __exit__
methods. I had mistakenly assumed that the @contextmanager decorator
turned a generator function into a context manager with the same
behavior as the equivalent context manager class.

Basically, it does. __exit__() is given the exception information. When you
write such a class, you can decide what to do with the exception. You can
silence it, immediately reraise it, conditionally reraise it, log it and then
reraise it, etc. Pushing the exception into the generator keeps this flexibility
and the equivalence. If it removed that choice, then it would not be equivalent.

Now I learn that,
no, in order to have the 'undo' code executed in the presence of an
exception, you must write your own try/finally block in the generator
function.

This raises the question in my mind: What's the use case for using
@contextmanager rather than wrapping your code in a context manager
class that defines __enter__ and __exit__, if you still have to
manager your own try/finally block?

The @contextmanager generator implementations are often shorter and easier to
read, in my opinion, partly because they use the try: finally: syntax that most
of us are very familiar with. I have to think less when I read it because it
looks so similar to the equivalent code that you would normally write.

The point of context managers isn't to remove the use of try: finally: entirely,
but to implement it once so that it can be reused cleanly. You only have to
write the one try: finally: in the generator and reuse it simply with the with:
statement in many places.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Alf P. Steinbach]

* Mike Kent:

Wow, I just learned something new. My understanding of context
managers was that the __exit__ method was guaranteed to be executed
regardless of how the context was left. I have often written my own
context manager classes, giving them the __enter__ and __exit__
methods. I had mistakenly assumed that the @contextmanager decorator
turned a generator function into a context manager with the same
behavior as the equivalent context manager class. Now I learn that,
no, in order to have the 'undo' code executed in the presence of an
exception, you must write your own try/finally block in the generator
function.

This raises the question in my mind: What's the use case for using
@contextmanager rather than wrapping your code in a context manager
class that defines __enter__ and __exit__, if you still have to
manager your own try/finally block?

Robert Kern and Steve Howell have already given given good answers.

As it happened this was news to me also, because I'm not that well-versed in
Python and it seems contrary to the purpose of providing a simpler way to write
a simple init-cleanup wrapper.

But additionally, if you want that, then you can define it, e.g.


<code>
# Py3

def simplecleanup( generator_func ):
class SimpleCleanup:
def __init__( self, *args, **kwargs ):
self.generator = generator_func( *args, **kwargs )

def __enter__( self ):
self.generator.send( None )
return self

def __exit__( self, x_type, x_obj, x_traceback ):
try:
self.generator.send( x_obj ) # x_obj is None if no exception
except StopIteration:
pass # Expected

return SimpleCleanup


@simplecleanup
def hello_goodbye( name ):
print( "Hello, {}!".format( name ) )
yield
print( "Goodbye {}!".format( name ) )


try:
with hello_goodbye( "Mary" ):
print( "Talk talk talk..." )
raise RuntimeError( "Offense" )
except:
pass
print()


@simplecleanup
def sensitive_hello_goodbye( name ):
print( "Hello, {}!".format( name ) )
x = yield
if x is not None:
print( "Uh oh, {}!".format( x ) )
print( "Good day {}!".format( name ) )
else:
print( "C u, {}!".format( name ) )


try:
with sensitive_hello_goodbye( "Jane" ):
print( "Talk talk talk..." )
raise RuntimeError( "Offense" )
except:
pass
</code>


Cheers,

- Alf

 

[Alf P. Steinbach]

* Steve Howell:

For C++ Petru Marginean once invented the "scope guard" technique (elaborated on
by Andrei Alexandrescu, they published an article about it in DDJ) where all you
need to do to ensure some desired cleanup at the end of a scope, even when the
scope is exited via an exception, is to declare a ScopeGuard w/desired action.

The C++ ScopeGuard was/is for those situations where you don't have proper
classes with automatic cleanup, which happily is seldom the case in good C++
code, but languages like Java and Python don't support automatic cleanup and so
the use case for something like ScopeGuard is ever present.

For use with a 'with' statement and possibly suitable 'lambda' arguments:

<code>
class Cleanup:
def __init__( self ):
self._actions = []

def call( self, action ):
assert( is_callable( action ) )
self._actions.append( action )

def __enter__( self ):
return self

def __exit__( self, x_type, x_value, x_traceback ):
while( len( self._actions ) != 0 ):
try:
self._actions.pop()()
except BaseException as x:
raise AssertionError( "Cleanup: exception during cleanup" ) from
</code>

I guess the typical usage would be what I used it for, a case where the cleanup
action (namely, changing back to an original directory) apparently didn't fit
the standard library's support for 'with', like

with Cleanup as at_cleanup:
# blah blah
chdir( somewhere )
at_cleanup.call( lambda: chdir( original_dir ) )
# blah blah

Another use case might be where one otherwise would get into very deep nesting
of 'with' statements with every nested 'with' at the end, like a degenerate tree
that for all purposes is a list. Then the above, or some variant, can help to
/flatten/ the structure. To get rid of that silly & annoying nesting.

Cheers,

- Alf (just sharing, it's not seriously tested code)

Hi Alf, I think I understand the notion you're going after here. You
have multiple cleanup steps that you want to defer till the end, and
there is some possibility that things will go wrong along the way, but
you want to clean up as much as you can. And, of course, flatter is
better.

Is this sort of what you are striving for?

class Cleanup:
def __init__( self ):
self._actions = []

def call( self, action ):
self._actions.append( action )

def __enter__( self ):
return self

def __exit__( self, x_type, x_value, x_traceback ):
while( len( self._actions ) != 0 ):
try:
self._actions.pop()()
except BaseException as x:
raise AssertionError( "Cleanup: exception during
cleanup" )

def clean_the_floor():
print('clean the floor')

def carouse(num_bottles, accident):
with Cleanup() as at_cleanup:
at_cleanup.call(clean_the_floor)
for i in range(num_bottles):
def take_down(i=i):
print('take one down', i)
at_cleanup.call(take_down)
if i == accident:
raise Exception('oops!')
print ('put bottle on wall', i)

carouse(10, None)
carouse(5, 3)

He he.

I think you meant:

> def carouse(num_bottles, accident):
> with Cleanup() as at_cleanup:
> at_cleanup.call(clean_the_floor)
> for i in range(num_bottles):
> def take_down(i=i):
> print('take one down', i)
> if i == accident:
> raise Exception('oops!')
> print ('put bottle on wall', i)
> at_cleanup.call(take_down)

I'm not sure. It's interesting & fun. But hey, it's Friday evening.

Regarding the "clean the floor", Marginean's original ScopeGuard has a 'dismiss'
method (great for e.g. implementing transactions). There's probably also much
other such functionality that can be added.

The original use case for Cleanup, I just posted this in case people could find
it useful, was a harness for testing that C++ code /fails/ as it should,
<url: http://pastebin.com/NK8yVcyv>, where Cleanup is used at line 479.

Some discussion of that in Usenet message and associated thread
<[email protected]>, "Unit testing of expected failures --
what do you use?" in [comp.lang.c++].

Another similar approach was discussed by Carlo Milanesi in <url:
http://www.drdobbs.com/cpp/205801074>, but he supplied this reference after I'd
done the above. Mainly the difference is that he defines a custom mark-up
language with corresponding source preprocessing, while I use the ordinary C++
preprocessor. There are advantages and disadvantages to both approaches.


Cheers,

- Alf

 

[Alf P. Steinbach]

* Robert Kern:

A custom-written context manager looks nicer and can be more readable.

from contextlib import contextmanager
import os

@contextmanager
def pushd(path):
original_dir = os.getcwd()
os.chdir(path)
try:
yield
finally:
os.chdir(original_dir)


with pushd(somewhere):
...


I don't think a general purpose ScopeGuard context manager has any such
benefits over the try: finally:, though.

class pushd( Cleanup ):
def __init__( self, path ):
original_dir = os.getcwd()
os.chdir( path )
self._actions.append( lambda: os.chdir( original_dir ) )


Disclaimer: haven't tested this, but it just occurred to me that for such small
init/cleanup wrappers the Cleanup class provides a nice alternative to
@contextmanager, as above: fewer lines, and perhaps even more clear code.


Cheers,

- Alf

 

[Gabriel Genellina]

Sorry, as with the places noted above, I can't understand what you're
trying to say here.

Regarding your posts, neither can I. All the time. Sorry, deciphering your
posts would force me to spend much more time than available and I can't
afford that - so I'm blocking your messages from now on.

 

[Alf P. Steinbach]

* Gabriel Genellina:

Regarding your posts, neither can I. All the time. Sorry, deciphering
your posts would force me to spend much more time than available and I
can't afford that - so I'm blocking your messages from now on.

You could just ask if there was anything you didn't understand.

Even with as little you quote readers can see my approach to that problem: asking.

But your response, both the out of context quoting and your comment, seems
solely designed to convey a negative impression instead of addressing any
technical issue.


Cheers,

- Alf

 

[Steve Holden]

* Gabriel Genellina:


You could just ask if there was anything you didn't understand.

Even with as little you quote readers can see my approach to that
problem: asking.

But your response, both the out of context quoting and your comment,
seems solely designed to convey a negative impression instead of
addressing any technical issue.

This isn't an isolated case, Alf. Physician, heal thyself.

regards
Steve

 

[ssteinerX]

This isn't an isolated case, Alf. Physician, heal thyself.

As far as I can tell, this happens on every thread Alf is involved in at one point or another. I'm sure it's "everyone else."

S