Scope of variable inside list comprehensions?

[Roy Smith]

Consider the following django snippet. Song(id) raises DoesNotExist if theid is unknown.

try:
songs = [Song(id) for id in song_ids]
except Song.DoesNotExist:
print "unknown song id (%d)" % id

Is id guaranteed to be in scope in the print statement? I found one thread(http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html)which says yes, but hints that it might not always be in the future. Now that we're in the future, is that still true? And for Python 3 also?

The current docs, http://docs.python.org/tutorial/datastructures.html#list-comprehensions, are mute on this point.

 

[Jussi Piitulainen]

Consider the following django snippet. Song(id) raises DoesNotExist
if the id is unknown.

try:
songs = [Song(id) for id in song_ids]
except Song.DoesNotExist:
print "unknown song id (%d)" % id

Is id guaranteed to be in scope in the print statement? I found one
thread
(http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html)
which says yes, but hints that it might not always be in the future.
Now that we're in the future, is that still true? And for Python 3
also?

Another id is in scope in this Python3 example (3.1.2, Ubuntu):
.... songs = [1/0 for id in [1]]
.... except Exception:
.... print('Caught', id)
....
Caught <built-in function id>

 

[Peter Otten]

Consider the following django snippet. Song(id) raises DoesNotExist if
the id is unknown.

try:
songs = [Song(id) for id in song_ids]
except Song.DoesNotExist:
print "unknown song id (%d)" % id

Is id guaranteed to be in scope in the print statement? I found one
thread
(http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html)
which says yes, but hints that it might not always be in the future. Now
that we're in the future, is that still true? And for Python 3 also?

The current docs,
http://docs.python.org/tutorial/datastructures.html#list-comprehensions,
are mute on this point.

If you are using a generator expression id will already be out of scope in
Python 2. In Python 3 list comprehensions have been changed to work the same
way:

$ cat song.py
class DoesNotExist(Exception):
pass

class Song:
def __init__(self, id):
if id == 2:
raise DoesNotExist

ids = [1, 2]
try:
songs = [Song(i) for i in ids]
except DoesNotExist as e:
print("song #%d does not exist" % i)
$ python song.py
song #2 does not exist
$ python3 song.py
Traceback (most recent call last):
File "song.py", line 11, in <module>
songs = [Song(i) for i in ids]
File "song.py", line 11, in <listcomp>
songs = [Song(i) for i in ids]
File "song.py", line 7, in __init__
raise DoesNotExist
__main__.DoesNotExist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "song.py", line 13, in <module>
print("song #%d does not exist" % i)
NameError: name 'i' is not defined
$


$ cat song_gen.py
class DoesNotExist(Exception):
pass

class Song:
def __init__(self, id):
if id == 2:
raise DoesNotExist

ids = [1, 2]
try:
songs = list(Song(i) for i in ids)
except DoesNotExist as e:
print("song #%d does not exist" % i)
$ python song_gen.py
Traceback (most recent call last):
File "song_gen.py", line 13, in <module>
print("song #%d does not exist" % i)
NameError: name 'i' is not defined

So you'd rather store the id in the exception.

 

[Jean-Michel Pichavant]

Consider the following django snippet. Song(id) raises DoesNotExist if the id is unknown.

try:
songs = [Song(id) for id in song_ids]
except Song.DoesNotExist:
print "unknown song id (%d)" % id

Is id guaranteed to be in scope in the print statement? I found one thread (http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html) which says yes, but hints that it might not always be in the future. Now that we're in the future, is that still true? And for Python 3 also?

The current docs, http://docs.python.org/tutorial/datastructures.html#list-comprehensions, are mute on this point.

For python 2, id will always be defined *as you meant it*. But you're
doing something wrong : overiding the 'id' builtin function.
With python 3 you will probably print the 'id' builtin function
representation, which is correct but not want you want to achieve.

The proper way to propagate information with exceptions is using the
exception itself:

try:
songs = [Song(_id) for _id in song_ids]
except Song.DoesNotExist, exc:
print exc

class DoesNotExist(Exception):
def __init__(self, songId):
self.songId = songId
def __str__(self):
return "Unkown Song Id %s" % self.songId

class Song:
def __init__(self, songId):
if whatever:
raise DoesNotExist(songId)
self.id=songId

JM

 

[Roy Smith]

Hmmm, the use of id was just a simplification for the sake of posting. The real code is a bit more complicated and used a different variable name, but that's a good point.

As far as storing the value in the exception, unfortunately, DoesNotExist is not my exception; it comes from deep within django. I'm just passing it along.

 

[Roy Smith]

Hmmm, the use of id was just a simplification for the sake of posting. The real code is a bit more complicated and used a different variable name, but that's a good point.

As far as storing the value in the exception, unfortunately, DoesNotExist is not my exception; it comes from deep within django. I'm just passing it along.

 

[Terry Reedy]

Hmmm, the use of id was just a simplification for the sake of
posting. The real code is a bit more complicated and used a
different variable name, but that's a good point.

As far as storing the value in the exception, unfortunately,
DoesNotExist is not my exception; it comes from deep within django.
I'm just passing it along.

It is hard to sensibly answer a question when the questioner
*significantly* changes the problem in the process of 'simplifying' it.
Both of those are significant changes ;-)

Changing a name to a built-in name is a complexification, not a
simplification, because it introduces new issues that were not in the
original.

Changing the exception from one you do not control to one you apparently
do also changes the appropriate answer. If you do not control the
exception and you want guaranteed access to the loop variable with
Python 3 (and the upgrade of django to work with Python 3 is more or
less done), then use an explicit loop. If you want the loop to continue
after an error, instead of stopping, you can put the try/except within
the loop, instead of without. This possibility is one advantage of using
an explicit loop.

songs = []
for song_id in song_ids:
try:
songs.append(Song(song_id))
except django.error:
print("unknown song id {}".format(song_id))

 

[Steven D'Aprano]

The proper way to propagate information with exceptions is using the
exception itself:

try:
songs = [Song(_id) for _id in song_ids]
except Song.DoesNotExist, exc:
print exc

I'm not entirely sure that this is the proper way to propagate the
exception. I see far to many people catching exceptions to print them, or
worse, to print a generic, useless message like "an error occurred".

The problem here is that having caught the exception, songs now does not
exist, and will surely cause another, unexpected, exception in a moment
or two when the code attempts to use it. Since the error is (apparently)
unrecoverable, the right way as far as I can see is:

songs = [Song(_id) for _id in song_ids]

allowing any exception to be fatal and the traceback to be printed as
normal.

If the error is recoverable, you will likely need to do more to recover
from it than merely print the exception and continue.

 

[Roy Smith]

Well, in my defense, I did ask a pretty narrow question, "Is id guaranteed to be in scope in the print statement?". While I will admit that not knowing whether I could alter the exception, or whether id masked a builtin or not does complexify answering some questions, those are questions I didn't ask

 

[Roy Smith]

Well, in my defense, I did ask a pretty narrow question, "Is id guaranteed to be in scope in the print statement?". While I will admit that not knowing whether I could alter the exception, or whether id masked a builtin or not does complexify answering some questions, those are questions I didn't ask

 

[Roy Smith]

Sigh. I attempted to reduce this to a minimal example to focus the discussion on the question of list comprehension variable scope. Instead I seem to have gotten people off on other tangents. I suppose I should post more of the real code...

song_ids = request.POST.getlist('song_id')
try:
songs = [Song.get(int(id)) for id in song_ids]
except Song.DoesNotExist:
return HttpResponseBadRequest("unknown song id (%d)" % id)

I may be in the minority here, but it doesn't bother me much that my use of'id' shadows a built-in. Especially in small scopes like this, I use whatever variable names make the the code easiest to read and don't worry aboutshadowing builtins. I don't have an exhaustive list of builtins in my head, so even if I worried about common ones like file or id, I'm sure I'd miss some others. So I don't sweat it.

If shadowing builtins was really evil, they'd be reserved keywords and thenyou wouldn't be able to do it.

 

[Terry Reedy]

Well, in my defense, I did ask a pretty narrow question, "Is id
guaranteed to be in scope in the print statement?".

Yes for 2.x, guaranteed no for 3.x.

If you had simply asked "Is the loop variable of a list comprehension
guaranteed to be in scope after the list comprehension?", without a
distracting example, that is the answer you would have received.

I intend(ed) to inform, not attack, hence no 'defense' needed.

While I will
admit that not knowing whether I could alter the exception, or
whether id masked a builtin or not does complexify answering some
questions, those are questions I didn't ask

Except that it bears on the question you did ask because it means that
the code will run in 3.x but with different results, whereas a random
name will fail in 3.x with a NameError.

 

[Chris Angelico]

I may be in the minority here, but it doesn't bother me much that my use of 'id' shadows a built-in.  Especially in small scopes like this, I use whatever variable names make the the code easiest to read and don't worry about shadowing builtins.  I don't have an exhaustive list of builtins in my head, so even if I worried about common ones like file or id, I'm sure I'd miss some others.  So I don't sweat it.

If shadowing builtins was really evil, they'd be reserved keywords and then you wouldn't be able to do it.

Agreed. The name 'id' is one that's shadowed benignly in a lot of
code. In the context of the original question, it's not an issue -
there'll be no confusion.

Python's scoping rules are "do what the programmer probably wants".
This works most of the time, but occasionally you get edge cases where
things are a bit weird, and C-style explicit scoping (with infinitely
nested block scope) begins to look better. But for probably 99% of
situations, Python's system "just works".

If you need more flexibility in the exception you throw, it may be
worth putting together a filter:

def HttpSongId(id):
try:
return Song.get(int(id))
except Song.DoesNotExist:
return HttpResponseBadRequest("unknown song id (%d)" % id)

song_ids = request.POST.getlist('song_id')
songs = [HttpSongId(id) for id in song_ids]

This encapsulates things in a somewhat weird way, but if there's any
other work to be done at the same time, you could probably come up
with a better name for the exception filter function.

ChrisA

 

[Rainer Grimm]

Hello,

try:
songs = [Song(id) for id in song_ids]
except Song.DoesNotExist:
print "unknown song id (%d)" % id

that's is a bad programming style. So it will be forbidden with python 3. The reason is that list comprehension is a construct from the functional world. It's only syntactic sugar for the functions map and filter. So functions have to be pure functions. To say it in other words, they have to be side-effect free. But the python construct from above pollutes the namespace with name id.

Greetings from Rottenburg,
Rainer

 

[88888 Dihedral]

Hello,

try:
songs = [Song(id) for id in song_ids]
except Song.DoesNotExist:
print "unknown song id (%d)" % id

that's is a bad programming style. So it will be forbidden with python 3.The reason is that list comprehension is a construct from the functional world. It's only syntactic sugar for the functions map and filter. So functions have to be pure functions. To say it in other words, they have to be side-effect free. But the python construct from above pollutes the namespace with name id.

Greetings from Rottenburg,
Rainer

The list might have to grow in a careless way that might lead to a crash
in the for inside a list that can't be trapped for errors directly.

 

[Jean-Michel Pichavant]

The proper way to propagate information with exceptions is using the
exception itself:

try:
songs = [Song(_id) for _id in song_ids]
except Song.DoesNotExist, exc:
print exc

I'm not entirely sure that this is the proper way to propagate the
exception. I see far to many people catching exceptions to print them, or
worse, to print a generic, useless message like "an error occurred".

[snip]

You misread me, I was referering to passing *information* with exception
(in other words, use the exception attributes). In the example I gave,
the exception has the songId value responsible for raising the error.
I totaly second your opinion on how poor the above handler is (hmm not
sure about this grammar construct, it sounds like a Yoda sentence).

JM