with statement and context managers

[Steven D'Aprano]

I'm not greatly experienced with context managers and the with statement, so
I would like to check my logic.

Somebody (doesn't matter who, or where) stated that they frequently use this
idiom:

spam = MyContextManager(*args)
for ham in my_iter:
with spam:
# do stuff


but to me that looks badly wrong. Surely the spam context manager object
will exit after the first iteration, and always raise an exception on the
second? But I don't quite understand context managers enough to be sure.


I've tested it with two examples:

# Simple example using built-in file context manager.
.... with spam:
.... print ham
....
0
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: I/O operation on closed file


# Slightly more complex example.
.... with spam as page:
.... print ham, sum(len(line) for line in page)
....
0 18486
1
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "<stdin>", line 3, in <genexpr>
File "/usr/local/lib/python2.7/socket.py", line 528, in next
line = self.readline()
File "/usr/local/lib/python2.7/socket.py", line 424, in readline
recv = self._sock.recv
AttributeError: 'NoneType' object has no attribute 'recv'




Am I right to expect that the above idiom cannot work? If not, what sort of
context managers do work as shown?

 

[Jack Diederich]

I'm not greatly experienced with context managers and the with statement,so
I would like to check my logic.

Somebody (doesn't matter who, or where) stated that they frequently use this
idiom:

spam = MyContextManager(*args)
for ham in my_iter:
   with spam:
        # do stuff
[snip]
# Simple example using built-in file context manager.
...     with spam:
...             print ham
...
0
Traceback (most recent call last):
 File "<stdin>", line 2, in <module>
ValueError: I/O operation on closed file

file_context = lambda: open('aaa')
for i in range(3):
with file_context():
print "hello"

... but if the context is short it is clearer and time saving to _not_
alias it. If the context is sufficiently complicated then it is worth
it to make the complex code into a first class context manager -
contextlib.contextmanager makes this very easy and extremely readable.

-Jack

 

[Nobody]

I'm not greatly experienced with context managers and the with statement, so
I would like to check my logic.

Somebody (doesn't matter who, or where) stated that they frequently use this
idiom:

spam = MyContextManager(*args)
for ham in my_iter:
with spam:
# do stuff


but to me that looks badly wrong. Surely the spam context manager object
will exit after the first iteration, and always raise an exception on the
second? But I don't quite understand context managers enough to be sure.

It depends upon the implementation of MyContextManager. If it's
implemented using the contextlib.contextmanager decorator, then you're
correct: you can only use it once. OTOH, if you implement your own class
with __enter__ and __exit__ methods, you can use the same context manager
object multiple times.

 

[Thomas Rachel]

Am 03.08.2011 04:15 schrieb Steven D'Aprano:

I'm not greatly experienced with context managers and the with
statement, so I would like to check my logic.

Somebody (doesn't matter who, or where) stated that they frequently
use this idiom:

spam = MyContextManager(*args)
for ham in my_iter:
with spam:
# do stuff


but to me that looks badly wrong. Surely the spam context manager
object will exit after the first iteration, and always raise an
exception on the second? But I don't quite understand context
managers enough to be sure.

Depends on the implementation. As already stated, a
contextlib.contextmanager will only run once. But it is easy to turn it
into a persistent one - which internally initializes as often as needed.

class PersistentCM(object):
def __init__(self, target):
self.target = target
self.current = []
def __enter__(self):
cm = self.target()
cm.__enter__()
self.current.append(cm)
def __exit__(self, *e):
return self.current.pop(-1).__exit__(*e)

(untested)

This would allow for a CM to be used multiple times. It internally
generates a new one on every __enter__() call and disposes of it on
every __exit__(). As __enter__() and __exit__() are supposed to be used
symmetrical, it should(!) work this way.

Many CMs don't allow your shown use, though, partly due to obvious reasons:

* The ...ing named ones do the action they are named after in the
__exit__() part and do nothing in the __enter__() part. (E.g. closing.)

* The ...ed named ones do the named action in __enter__() and undo it in
__exit__(). They may or may not work multiple times.

For example, take threading.Lock et al., which you can have locked
several times.

... with spam:
... print ham
...
0
Traceback (most recent call last):
File "<stdin>", line 2, in<module>
ValueError: I/O operation on closed file

To be honest, this one I don't understand as well. __exit__() closes the
file spam, but even before "print ham" can be executed for a second
time, there is a check if the file is still open. What does __enter__()
do here? Just check if it is closed, or is it more? Nevertheless, it
makes sense.

# Slightly more complex example.

... with spam as page:
... print ham, sum(len(line) for line in page)
...
0 18486
1
Traceback (most recent call last):
File "<stdin>", line 3, in<module>
File "<stdin>", line 3, in<genexpr>
File "/usr/local/lib/python2.7/socket.py", line 528, in next
line = self.readline()
File "/usr/local/lib/python2.7/socket.py", line 424, in readline
recv = self._sock.recv
AttributeError: 'NoneType' object has no attribute 'recv'

Here the above said applies: after closing happens in __exit__() and
_sock gets set to None, the object is not usable any longer.

Am I right to expect that the above idiom cannot work? If not, what sort of
context managers do work as shown?

As said, take threading.Lock as an example: they can be acquire()d and
release()d as often as you want, and these actions happen in __enter__()
and __exit__(), respectively.


HTH,

Thomas

 

[Steven D'Aprano]

Am 03.08.2011 04:15 schrieb Steven D'Aprano: [...]

but to me that looks badly wrong. Surely the spam context manager
object will exit after the first iteration, and always raise an
exception on the second? But I don't quite understand context
managers enough to be sure.

Depends on the implementation. As already stated, a
contextlib.contextmanager will only run once. But it is easy to turn it
into a persistent one - which internally initializes as often as needed.

Thanks, that's exactly the sort of information I was after.

Thank you to everyone who answered.