How to modify local variables from internal functions?

[kj]

I like Python a lot, and in fact I'm doing most of my scripting in
Python these days, but one thing that I absolutely *****DETEST*****
about Python is that it does allow an internal function to modify
variables in the enclosing local scope. This willful hobbling of
internal functions seems to me so perverse and unnecessary that it
delayed my adoption of Python by about a decade. Just thinking
about it brings me to the brink of blowing a gasket... I must go
for a walk...



OK, I'm better now.

Anyway, I recently wanted to write a internal helper function that
updates an internal list and returns True if, after this update,
the list is empty, and once more I bumped against this hated
"feature". What I wanted to write, if Python did what I wanted it
to, was this:

def spam():
jobs = None
def check_finished():
jobs = look_for_more_jobs()
return not jobs

if check_finished():
return

process1(jobs)

if check_finished():
return

process2(jobs)

if check_finished():
return

process3(jobs)

In application in question, the availability of jobs can change
significantly over the course of the function's execution (jobs
can expire before they are fully processed, and new ones can arise),
hence the update-and-check prior to the calls to process1, process2,
process3.

But, of course, the above does not work in Python, because the jobs
variable local to spam does not get updated by check_finished.
Grrrr!

I ended up implementing check_finished as this stupid-looking
monstrosity:

def spam():
jobs = []
def check_finished(jobs):
while jobs:
jobs.pop()
jobs.extend(look_for_more_jobs())
return not jobs

if check_finished(jobs):
return

# etc.

Is there some other trick to modify local variables from within
internal functions?

TIA!

kynn

 

[Chris Rebert]

I like Python a lot, and in fact I'm doing most of my scripting in
Python these days, but one thing that I absolutely *****DETEST*****
about Python is that it does allow an internal function to modify
variables in the enclosing local scope.  This willful hobbling of
internal functions seems to me so perverse and unnecessary that it
delayed my adoption of Python by about a decade.  Just thinking
about it brings me to the brink of blowing a gasket...  I must go
for a walk...

Anyway, I recently wanted to write a internal helper function that
updates an internal list and returns True if, after this update,
the list is empty, and once more I bumped against this hated
"feature".  What I wanted to write, if Python did what I wanted it
to, was this:

def spam():
   jobs = None
   def check_finished():
      jobs = look_for_more_jobs()
      return not jobs

   if check_finished():
       return

   process1(jobs)

   if check_finished():
       return

   process2(jobs)

   if check_finished():
       return

   process3(jobs)

Is there some other trick to modify local variables from within
internal functions?

The `nonlocal` statement?:
http://www.python.org/dev/peps/pep-3104/

Cheers,
Chris

 

[MRAB]

I like Python a lot, and in fact I'm doing most of my scripting in
Python these days, but one thing that I absolutely *****DETEST*****
about Python is that it does allow an internal function to modify
variables in the enclosing local scope. This willful hobbling of
internal functions seems to me so perverse and unnecessary that it
delayed my adoption of Python by about a decade. Just thinking
about it brings me to the brink of blowing a gasket... I must go
for a walk...



OK, I'm better now.

Anyway, I recently wanted to write a internal helper function that
updates an internal list and returns True if, after this update,
the list is empty, and once more I bumped against this hated
"feature". What I wanted to write, if Python did what I wanted it
to, was this:

def spam():
jobs = None
def check_finished():
jobs = look_for_more_jobs()
return not jobs

if check_finished():
return

process1(jobs)

if check_finished():
return

process2(jobs)

if check_finished():
return

process3(jobs)

In application in question, the availability of jobs can change
significantly over the course of the function's execution (jobs
can expire before they are fully processed, and new ones can arise),
hence the update-and-check prior to the calls to process1, process2,
process3.

But, of course, the above does not work in Python, because the jobs
variable local to spam does not get updated by check_finished.
Grrrr!

So you didn't try this?

def spam():
jobs = []

def check_finished():
jobs[:] = look_for_more_jobs()
return not jobs

if check_finished():
return

process1(jobs)

if check_finished():
return

process2(jobs)

if check_finished():
return

process3(jobs)


Actually, I'd refactor and reduce it to this:

def spam():
for proc in [process1, process2, process3]:
jobs = look_for_more_jobs()
if not jobs:
return
proc(jobs)

 

[Steven D'Aprano]

I like Python a lot, and in fact I'm doing most of my scripting in
Python these days, but one thing that I absolutely *****DETEST*****
about Python is that it does allow an internal function to modify
variables in the enclosing local scope.

You can't rebind names in the enclosing scope (not without the nonlocal
keyword, which I believe is introduced in Python 3.0):

.... x = 1
.... def inner():
.... x = 2
.... inner()
.... print x
....1


However, naturally you can call methods on objects bound in the outer
scope. It would be rather inconvenient if you couldn't access them from
inner functions, but that's how Python was prior to the introduction of
nested scopes in 2.1.


Given this code:

def f():
L = [1, 2, 3]
def inner():
return L.index(1)
return inner()

f()


This is what you get in Python 1.5:

Traceback (innermost last):
File "<stdin>", line 1, in ?
File "<stdin>", line 5, in f
File "<stdin>", line 4, in inner
NameError: L


while in current versions of Python, you get 0.

Of course, once you allow inner functions to call methods on objects in
the outer scope, you allow methods which mutate the object:
.... L = []
.... def inner():
.... L.append(None)
.... inner()
.... print L
....[None]

What other behaviour would you expect?

This willful hobbling of
internal functions seems to me so perverse and unnecessary that it
delayed my adoption of Python by about a decade. Just thinking about it
brings me to the brink of blowing a gasket... I must go for a walk...

I think you don't understand the object and assignment model of Python if
you think this is "hobbling of internal functions".

Anyway, I recently wanted to write a internal helper function that
updates an internal list and returns True if, after this update, the
list is empty, and once more I bumped against this hated "feature".
What I wanted to write, if Python did what I wanted it to, was this: [...]
But, of course, the above does not work in Python, because the jobs
variable local to spam does not get updated by check_finished. Grrrr!

Okay, now I'm completely confused. You "DETEST" (shouting was yours) that
Python objects can be modified by inner functions, and yet here you are
writing a function which relies on that -- and gets it wrong, because
you're trying to rebind a name rather than modify an object.

Others have suggested the nonlocal keyword in 3.x. Here's another
solution:


def spam():
jobs = look_for_more_jobs()
if not jobs:
return
process1(jobs)
jobs = look_for_more_jobs()
if not jobs:
return
process2(jobs)
jobs = look_for_more_jobs()
if not jobs:
return
process3(jobs)



which immediately suggests refactoring:


def spam():
for process in (process1, process2, process3):
jobs = look_for_more_jobs()
if not jobs:
return
process(jobs)

 

[Steven D'Aprano]

I like Python a lot, and in fact I'm doing most of my scripting in
Python these days, but one thing that I absolutely *****DETEST*****
about Python is that it does allow an internal function to modify
variables in the enclosing local scope.

You can't rebind names in the enclosing scope (not without the nonlocal
keyword, which I believe is introduced in Python 3.0):

.... x = 1
.... def inner():
.... x = 2
.... inner()
.... print x
....1


However, naturally you can call methods on objects bound in the outer
scope. It would be rather inconvenient if you couldn't access them from
inner functions, but that's how Python was prior to the introduction of
nested scopes in 2.1.


Given this code:

def f():
L = [1, 2, 3]
def inner():
return L.index(1)
return inner()

f()


This is what you get in Python 1.5:

Traceback (innermost last):
File "<stdin>", line 1, in ?
File "<stdin>", line 5, in f
File "<stdin>", line 4, in inner
NameError: L


while in current versions of Python, you get 0.

Of course, once you allow inner functions to call methods on objects in
the outer scope, you allow methods which mutate the object:
.... L = []
.... def inner():
.... L.append(None)
.... inner()
.... print L
....[None]

What other behaviour would you expect?

This willful hobbling of
internal functions seems to me so perverse and unnecessary that it
delayed my adoption of Python by about a decade. Just thinking about it
brings me to the brink of blowing a gasket... I must go for a walk...

I think you don't understand the object and assignment model of Python if
you think this is "hobbling of internal functions".

Anyway, I recently wanted to write a internal helper function that
updates an internal list and returns True if, after this update, the
list is empty, and once more I bumped against this hated "feature".
What I wanted to write, if Python did what I wanted it to, was this: [...]
But, of course, the above does not work in Python, because the jobs
variable local to spam does not get updated by check_finished. Grrrr!

Okay, now I'm completely confused. You "DETEST" (shouting was yours) that
Python objects can be modified by inner functions, and yet here you are
writing a function which relies on that -- and gets it wrong, because
you're trying to rebind a name rather than modify an object.

Others have suggested the nonlocal keyword in 3.x. Here's another
solution:


def spam():
jobs = look_for_more_jobs()
if not jobs:
return
process1(jobs)
jobs = look_for_more_jobs()
if not jobs:
return
process2(jobs)
jobs = look_for_more_jobs()
if not jobs:
return
process3(jobs)



which immediately suggests refactoring:


def spam():
for process in (process1, process2, process3):
jobs = look_for_more_jobs()
if not jobs:
return
process(jobs)