closure = decorator?

[Tim]

I've read a couple of articles about this, but still not sure.
When someone talks about a closure in another language (I'm learning Lua on the side), is that the same concept as a decorator in Python?

It sure looks like it.
thanks,
--Tim

 

[Chris Angelico]

I've read a couple of articles about this, but still not sure.
When someone talks about a closure in another language (I'm learning Lua on the side), is that the same concept as a decorator in Python?

No, they're quite different. A decorator (let's look at function
decorators; classes can have them too, and they work pretty much the
same way) is syntactic sugar for this:

def func(args):
blah blah blah
func = decorator(func)

You can do all sorts of things with that. Even stupid things:
def foo():
pass

True

Using print as a decorator does work, though hardly usefully

A closure, on the other hand, is a function that has some extra context:

def outer(x):
x += 1
def inner():
return x
return inner

The function inner() "knows" its context. When it's called, it'll use
the same value for x that would have been used in the outer function,
even though it's a separate function:
6

The terminology is that inner() "closes over" x, if I have that
correct (I've not been all that big in functional programming and
lambda calculus). Someone will correct me if I'm not.

It's very common for a decorator to use closures, but the two are
completely different. They're no more connected than, say, for loops
and lists. They just happen to work well together.

Closures in other languages will, as far as I know, be the same thing
as closures in Python. (And their presence and functionality in
JavaScript leaves me wondering why on earth the 'this' reference can't
be considered "closed over" in the same way. But that's hardly the
worst of the language's warts.)

ChrisA

 

[Jussi Piitulainen]

I've read a couple of articles about this, but still not sure.

When someone talks about a closure in another language (I'm learning
Lua on the side), is that the same concept as a decorator in Python?

It sure looks like it.

I don't see how. Wikipedia's opening paragraph on "closure" seems good
to me - closures are a way to implement lexical scoping when functions
that have free variables are passed as arguments and returned as
values:

<http://en.wikipedia.org/wiki/Closure_(computer_science)>
# In programming languages, a closure (also lexical closure or
# function closure) is a function or reference to a function together
# with a referencing environment—a table storing a reference to each
# of the non-local variables (also called free variables or upvalues)
# of that function.[1] A closure—unlike a plain function
# pointer—allows a function to access those non-local variables even
# when invoked outside its immediate lexical scope.

There's an example in Python on that page.

 

[Jussi Piitulainen]

def outer(x):
x += 1
def inner():
return x
return inner ....
The terminology is that inner() "closes over" x, if I have that
correct (I've not been all that big in functional programming and
lambda calculus). Someone will correct me if I'm not.

I don't actually know for sure what the most correct terminology is,
but I like to think that a closure is a procedure (function) closed in
the environment (namespace) where it was created, so even when it is
called from another environment, it uses the one where it was born.

But that may be private to me.

 

[Piet van Oostrum]

I don't actually know for sure what the most correct terminology is,
but I like to think that a closure is a procedure (function) closed in
the environment (namespace) where it was created, so even when it is
called from another environment, it uses the one where it was born.

I usually say that a closure is a package, containing a function with
some additional data it needs. The data usually is in the form of name
bindings.

On the other hand, an object (instance) is a package containg data, with
one of more functions that work on this data.

So an object is more or less the dual of a closure, and in many cases
they can be used for the same purpose. In most programming languages the
difference is that closures can be called directly, whereas an object
needs to be used with a method call to do the same (this makes Java so
ugly in this area). In Python, however, you can define the __call__
method and with this they become almost identical in behaviour.

 

[Roy Smith]

I usually say that a closure is a package, containing a function with
some additional data it needs. The data usually is in the form of name
bindings.

That's pretty close to the way I think about it. The way it was
originally described to me is, "A closure is a function bundled up with
it's arguments".

To make a real-life analogy, let's say you're modeling a parking garage.
I want to be able to walk up to the attendant and say, "Please bring my
car around front at 5 O'Clock. It's that one" (pointing to the slightly
dented green Ford in spot 37). So, you've got a class:

class DeliveryRequest:
def __init__(self, spot, time):
self.spot = spot
self.time = time

Now, over the course of the day, the garage attendants shuffle cars
around to make room and retrieve vehicles that packed in the back.
Comes 5 O'Clock, what vehicle do you want the attendant to deliver to
the front? The one that was in spot 37 at the time you made the
request, or the one that's in spot 37 at 5 O'Clock?

Unless you want somebody else's car (perhaps you'd like something better
than a slightly dented Ford), you want the attendant to capture the
current state of spot 37 and remember that until 5 O'Clock when it's
time to go get the car, no matter where it happens to be right now.

That's a closure.

 

[Jussi Piitulainen]

That's pretty close to the way I think about it. The way it was
originally described to me is, "A closure is a function bundled up
with it's arguments".

Really? It should be more like "a function bundled up with some other
function's arguments" and even more like "a function bundled up with
bindings for its free variables".

And the data that makes a function a closure is bindings always, by
definition, not just usually.

To make a real-life analogy, let's say you're modeling a parking
garage. I want to be able to walk up to the attendant and say,
"Please bring my car around front at 5 O'Clock. It's that one"
(pointing to the slightly dented green Ford in spot 37). So, you've
got a class:

class DeliveryRequest:
def __init__(self, spot, time):
self.spot = spot
self.time = time

Now, over the course of the day, the garage attendants shuffle cars
around to make room and retrieve vehicles that packed in the back.
Comes 5 O'Clock, what vehicle do you want the attendant to deliver
to the front? The one that was in spot 37 at the time you made the
request, or the one that's in spot 37 at 5 O'Clock?

Unless you want somebody else's car (perhaps you'd like something
better than a slightly dented Ford), you want the attendant to
capture the current state of spot 37 and remember that until 5
O'Clock when it's time to go get the car, no matter where it happens
to be right now.

That's a closure.

I fail to see a closure here. I see a class. I see an implied object
that could as well be dict(spot=37, time=5). Other entities (garage
and attendants) are not made sufficiently explicit.

There's another, more widely used word that means object. There's
another, more widely used word that means state. (I'm referring to the
words "object" and "state".) I see no need to use "closure" to mean
object or state, especially when the term has a more specific
established meaning.

 

[Steven D'Aprano]

Really? It should be more like "a function bundled up with some other
function's arguments" and even more like "a function bundled up with
bindings for its free variables".

Closures have nothing to do with *arguments*. A better definition of a
closure is that it is a function together with a snapshot of the
environment it was called from.

def func(arg):
y = arg + 1
def inner():
return y + 1000
return inner

f = func(1)

At this point, f is a closure. It needs to know the value of y (not the
argument to func) in order to work, and the implementation is to store
that information inside f.func_closure (or f.__closure__ in Python 3).
The part of the calling environment which is saved is y:

py> f.func_closure[0].cell_contents
2

And the data that makes a function a closure is bindings always, by
definition, not just usually.

Its not just *any* bindings though, it is specifically bindings to
variables in the environment from which it was called.


[...]

I fail to see a closure here. I see a class. I see an implied object
that could as well be dict(spot=37, time=5). Other entities (garage and
attendants) are not made sufficiently explicit.

In general, anything you can do with a closure, you can do with an object
explicitly recording whatever state you want. A closure is just one
implementation of "callable object with state that can be set when you
create it". The closure f defined above could instead be written as:

class Func:
def __init__(self, arg):
self.y = arg + 1
def __call__(self):
return self.y + 1000

f = Func(1)


Which is better? If you want to expose the value of y to the outside
world to modify, the class solution is better. If you don't, the closure
is better. Closures tend to be more compact, and I suspect more
efficient, but there's nothing you can do with one you can't do with the
other.

 

[Jussi Piitulainen]

Closures have nothing to do with *arguments*. A better definition of
a closure is that it is a function together with a snapshot of the
environment it was called from.

Well, first, I was only trying to see something good in Piet's and
Roy's formulations.

Second, it's precisely not (a snapshot of) the environment where the
function is *called* from, it's (a snapshot of) the environment where
the function was *created* in. This is the whole *point*.

Third, to be even more pedantic, in the context where I think closures
originally appeared as an innovation, all local variables are bound by
a lambda. There the (non-global) free variables of a function *are*
arguments of *another* function. I can expand on this if you like, but
it will be in terms of another language, and not terribly relevant to
this discussion anyway.

def func(arg):
y = arg + 1
def inner():
return y + 1000
return inner

f = func(1)

At this point, f is a closure. It needs to know the value of y (not
the argument to func) in order to work, and the implementation is to
store that information inside f.func_closure (or f.__closure__ in
Python 3). The part of the calling environment which is saved is y:

py> f.func_closure[0].cell_contents
2

Whether there is a y in the *calling* environment or not is
*irrelevant*.

1002


Its not just *any* bindings though, it is specifically bindings to
variables in the environment from which it was called.

In the environment where it was created.

[...]

I fail to see a closure here. I see a class. I see an implied
object that could as well be dict(spot=37, time=5). Other entities
(garage and attendants) are not made sufficiently explicit.

In general, anything you can do with a closure, you can do with an
object explicitly recording whatever state you want. A closure is
just one implementation of "callable object with state that can be
set when you create it". The closure f defined above could instead
be written as:

class Func:
def __init__(self, arg):
self.y = arg + 1
def __call__(self):
return self.y + 1000

f = Func(1)

Which is better? If you want to expose the value of y to the outside
world to modify, the class solution is better. If you don't, the
closure is better. Closures tend to be more compact, and I suspect
more efficient, but there's nothing you can do with one you can't do
with the other.

Sure.

 

[Franck Ditter]

Closures have nothing to do with *arguments*. A better definition of a
closure is that it is a function together with a snapshot of the
environment it was called from.

def func(arg):
y = arg + 1
def inner():
return y + 1000
return inner

f = func(1)

Maybe a better example of closure would be (just for the nonlocal) :

def fib() :
(a,b) = (0,1)
def producer() :
nonlocal a,b # Python 3
old = a
(a,b) = (b,a+b)
return old
return producer

f = fib()
[f() for i in range(10)]

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

At this point, f is a closure. It needs to know the value of y (not the
argument to func) in order to work, and the implementation is to store
that information inside f.func_closure (or f.__closure__ in Python 3).
The part of the calling environment which is saved is y

Shouldn't it be the (a,b) pair here ? But :

f.__closure__[0].cell_contents # access to what ?

55

Shouldn't cell_contents keep the current (a,b) pair, a part of the snapshot of
the creation environment (private variables of the closure) ?
Instead it seems to returns only a (which is the next production)...

franck

 

[Steven D'Aprano]

Closures have nothing to do with *arguments*. A better definition of a
closure is that it is a function together with a snapshot of the
environment it was called from.

[...]
Second, it's precisely not (a snapshot of) the environment where the
function is *called* from, it's (a snapshot of) the environment where
the function was *created* in. This is the whole *point*.

Ah yes, of course you are right. I actually knew that, it was a slip of
the brain that I wrote it wrong :-(

Thanks for the correction.

 

[Terry Reedy]

Closures have nothing to do with *arguments*. A better definition of a
closure is that it is a function together with a snapshot of the
environment it was called from.

[...]
Second, it's precisely not (a snapshot of) the environment where the
function is *called* from, it's (a snapshot of) the environment where
the function was *created* in. This is the whole *point*.

Ah yes, of course you are right. I actually knew that, it was a slip of
the brain that I wrote it wrong :-(

Thanks for the correction.

The closure is also not a 'snapshot' but a reference to (or preservation
of) (relevant parts of) the environment. A snapshot of the environment
at the time of definition would have been much easier to implement.

x = 1
def outer():
y = 1
def inner():
return x + y
y = 2
return inner
x = 2
print(outer()())
# 4

In a sense, all user functions are closures in that they have, and have
always had, a reference to their definition module environment -- the
readonly .__globals__ attribute (probably .func_globals in 2.x).

This lexical, as opposed to dynamic scoping, becomes noticable when one
import a function from another module, as for testing. Because
..__globals__ is read-only, one must monkey-patch the module of
definition to change the function's global (modular) environment, as
when replacing an object it uses with a mock. I ran into this when
testing Idle methods that use a tk message box to display a message and
wait for input (sometimes text, always a mouse click) from a human user.

What is relatively new (and tricky) is capturing local names of
surrounding functions while maintaining both late binding and
independent writability for each closure. This last means that the
following works:

def account():
balance = 0
def trans(amt):
nonlocal balance
balance += amt
return balance
return trans

xmasfund = account()
pettycash = account()
print(xmasfund(100))
# 100
print(pettycash(50))
# 50
print(xmasfund(-100))
# 0
print(pettycash(-25))
# 25

Closures and decorators are *really* two different subjects.

 

[Terry Reedy]

Maybe a better example of closure would be (just for the nonlocal) :

def fib() :
(a,b) = (0,1)

a,b = 0,1 is the same thing.

a and b are separate local names and are in no sense a 'pair'.

def producer() :
nonlocal a,b # Python 3
old = a
(a,b) = (b,a+b)
return old
return producer

f = fib()
[f() for i in range(10)]

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

At this point, f is a closure. It needs to know the value of y (not the
argument to func) in order to work, and the implementation is to store
that information inside f.func_closure (or f.__closure__ in Python 3).
The part of the calling environment which is saved is y

Shouldn't it be the (a,b) pair here ? But :

f.__closure__[0].cell_contents # access to what ?

55

Shouldn't cell_contents keep the current (a,b) pair, a part of the snapshot of
the creation environment (private variables of the closure) ?
Instead it seems to returns only a (which is the next production)...

Look as f.__closure__[1] (.cell_contents) for b.

 

[Peter Cacioppi]

I've read a couple of articles about this, but still not sure.

When someone talks about a closure in another language (I'm learning Lua on the side), is that the same concept as a decorator in Python?



It sure looks like it.

thanks,

--Tim

In the proper lambda calculus, you don't have side effects. So Terry's balance example is helpful for Python, but perhaps it might be better think of it as a Pythonic extension to the lambda calculus closure. In other words, Python's closure handles cases that don't present themselves in the lambda calculus.

When you are taught about closures in a purely formal setting the example will not include a side effect nor any need for a statement like "nonlocal balance". The closed variable (here it is balance) simply remains part of the scope of the inner function and can referenced appropriately.

This might distract the original questioner, I only mention that "closure" probably means different things to different people.

 

[Tim]

Thanks everyone for discussing. I see that closures are completely different from decorators and now I'm not really sure how I got them confused in the first place.

I will have to stare at the replies for a while before I can say I really 'get' closures. thanks,
--Tim