Need help with Python scoping rules

[Carl Banks]

You can probably work around this by replacing the staticmethod
decorator with an equivalent function call:.

class Demo9(object):
    def fact(n):
        if n < 2:
            return 1
        else:
            return n * Demo.fact(n - 1)

    _classvar = fact(5)
    fact = staticmethod(fact)

print Demo9._classvar
xx = Demo9()
print xx.fact(6)
print Demo9.fact(8)

This won't work normally. It only worked for you because you made a
typo.


Carl Banks

 

[Steven D'Aprano]

In <[email protected]> "Diez B. Roggisch"


This looks to me like a major wart, on two counts.

First, one of the goals of OO is encapsulation, not only at the level of
instances, but also at the level of classes. Your comment suggests that
Python does not fully support class-level encapsulation.

I don't even know what that is supposed to mean. If anything, your
problem could be because Python has TOO MUCH "class-level encapsulation"
compared to what you are expecting: functions inside a class don't see
the class attributes you expect them too.

Actually, I think your problem is that you are interpreting what you're
seeing in terms of C++ or Java, and assuming that whatever they do is the
One True Definition of OO. That's hardly valid -- if any language
deserves the label of the One True OO Design, it might be Smalltalk, on
the basis that it was the first OO language. But even that is silly --
just because something was the first, that doesn't make it the best
example of something.

Second, my example shows that Python puts some peculiar restrictions on
recursion.

Incorrect. Recursive calls are just like any other function call in
Python: when you call *any* function from the body of a function, it must
be in scope at runtime.

def fact(n):
print g() # g must be in scope when fact is called
if n < 2: return 1
return n*fact(n-1) # fact must be in scope when fact is called


Python doesn't treat recursive functions specially in any way. It is
*classes*, not functions, that are special. This is explained in the docs:

http://docs.python.org/reference/executionmodel.html

It is also discussed in the PEP introducing nested scopes to Python:

http://www.python.org/dev/peps/pep-0227/

It's even eluded to in the tutorial:

http://docs.python.org/tutorial/classes.html

Recursion! One of the central concepts in the theory of
functions! This is shown most clearly by the following elaboration of
my original example:

class Demo(object):
def fact_rec(n):
if n < 2:
return 1
else:
return n * fact_rec(n - 1)

Why are you defining a method without a self parameter?

In any case, the problem has nothing to do with recursion:

.... def f():
.... return g()
.... def g():
.... return "gee"
.... x = f()
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in Broken
File "<stdin>", line 3, in f
NameError: global name 'g' is not defined

def fact_iter(n):
ret = 1
for i in range(1, n + 1):
ret *= i
return ret

classvar1 = fact_iter(5)
classvar2 = fact_rec(5)

This code won't compile as shown,

That's nonsense. It compiles, and then it suffers a runtime error when
you *execute* it. (NameError occurs at runtime, not at compile time.)
You can prove this for yourself by putting that above class definition
inside a string, s:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "", line 1, in <module>
File "", line 15, in Demo
File "", line 6, in fact_rec
NameError: global name 'fact_rec' is not defined


If you expect us to take your criticisms seriously, at least get your
basic facts right.


[...]

Is there any good reason (from the point of view of Python's overall
design) for not fixing this?

It doesn't need to be "fixed" because it's not broken. The question is,
should it be *changed* to match C++ programmers' expectations?

 

[kj]

Yeah, it's a little surprising that you can't access class scope from
a function, but that has nothing to do with encapsulation.

It does: it thwarts encapsulation. The helper function in my
example is one that clearly rightfully belongs nowhere else than
the class itself, i.e. encapsulated within the class. It is only
this silly prohibition against recursive functions in a class
statement that forces one to put it outside the class statement.

kynn

 

[Steven D'Aprano]

In <[email protected]> "Martin P. Hellwig"


Python itself: it already offers a limited form of class encapsulation
(e.g. class variables).

An int variable is an int. A string variable is a string. A bool variable
is a bool. A class variable is a class.

Perhaps you mean a class attribute?

It would be nice if it went all the way and
gave classes their own bona fide scope.

Classes have their own bona fide scope. It just doesn't work the way you
expect it to.

(But I hasten to add: I *still*
don't understand the Python scope model, and not for lack of trying.
I've only succeeded in finding fragments of this model explained here
and there, like pottery shards: a bit lost in a tutorial, or some
comment in a newsgroup thread, etc. Maybe the full, authoritative
documentation of Python's scope model got lost somewhere, and will be
found by archaeologists in the 25th century...)

It's all in the Fine Manual:

http://docs.python.org/reference/executionmodel.html

...except, apparently, when that code is a recursive function, in which
case one's out of luck.

Incorrect. As I showed in my previous post, your problem has nothing to
do with recursion.

If recursion is so evil, and Python so intent in saving us from shooting
ourselves in the foot, why does it allow recursion at all?

Recursion isn't evil, and Python isn't intent on preventing foot-shooting.

 

[kj]

Why are you defining a method without a self parameter?

Because, as I've explained elsewhere, it is not a method: it's a
"helper" function, meant to be called only once, within the class
statement itself.

Well, this is not strictly true, because the function is recursive,
so it will also call itself a few times. But still, all these
calls happen (loosely speaking) "within the class statement".

In fact the only reason to use a function for such initialization
work is when one needs recursion; otherwise there's no point in
defining a function that will only be invoked exactly once.

kynn

 

[Nigel Rantor]

Needless to say, I'm pretty beat by this point. Any help would be
appreciated.

Thanks,

Based on your statement above, and the fact that multiple people have
now explained *exactly* why your attempt at recursion hasn't worked, it
might be a good idea to step back, accept the advice and walk away
instead of trying to convince people that the language forbids recursion
and doesn't provide decent OO ecapsulation.

Otherwise I'd wager you'll soon be appearing in multiple kill-files.

n

 

[Steven D'Aprano]

It would be trivial to define a keyword (e.g. this, or if you prefer,
__this__), valid only within a class statement, and that the interpreter
would recognize as "the current class", even before this class is full
defined.

Python doesn't treat the addition of new keywords, and the corresponding
breakage of code which used that word as a normal name, as "trivial" --
the Python dev team takes their responsibilities to not casually break
people's code seriously. That pretty much rules out "this", although it
would allow "__this__".

However, what's your use-case?

class Demo(object):
def f(n):
return n+1
x = f(10)


is a poorly written class, because the user will expect to do this:

instance = Demo()
assert instance.c == 11 # suceeds
instance.f(10) == 11 # fails


Since the function f doesn't refer to a Demo instance, or the Demo class,
why do you put it inside the Demo class? It doesn't belong there, it
belongs in the global (module) scope. Move it out, and your problem goes
away.

The only use-case I can think of for __this__ is the following:

class Example(object):
@staticmethod
def rec(n):
if n < 2: return "x"
return "x" + __this__.rec(n/2)

Example.rec(15)


but again, why include rec() in the class if it doesn't actually have
anything to do with the class or the instance? Or why make it a
staticmethod? Just make it a class method:

class Example(object):
@classmethod
def rec(cls, n):
if n < 2: return "x"
return "x" + cls.rec(n/2)

 

[Carl Banks]

It does: it thwarts encapsulation.  The helper function in my
example is one that clearly rightfully belongs nowhere else than
the class itself, i.e. encapsulated within the class. It is only
this silly prohibition against recursive functions in a class
statement that forces one to put it outside the class statement.

Oh well, I guess that sucks for you.


Carl Banks

 

[Steven D'Aprano]

In <1bf83a7e-f9eb-46ff-84fe-cf42d9608e71@j21g2000yqe.googlegroups.com>


It does: it thwarts encapsulation. The helper function in my example is
one that clearly rightfully belongs nowhere else than the class itself,
i.e. encapsulated within the class.

There's nothing "clearly" about it.

This helper function doesn't reference the class, or any instance of the
class. Why should it be encapsulated in the class? It's not *part* of the
class, it shouldn't be *inside* the class.

Look at it this way: classes are made from building blocks. Just because
a building block ends up in a class, doesn't mean the function that makes
the building block has to be inside the class too.

It's an accident of Python's execution model that a single function call
*sometimes* works as you expect inside the class statement:

class Demo:
def f():
return 2
def g():
return f()+1
x = f() # works
y = g() # fails


As you can see, the problem is not because of recursion, but because the
class scope is not inserted into the function scope. As I said earlier,
your problem isn't too little class encapsulation, but too much: the
class scope doesn't leak into the function scope.

Python could, of course, behave the way you want, but it would lead to
some odd interactions:

class Weird(object):
x = 1
def __init__(self):
self.y = 2
def test(self):
print self.x # refers to attribute x with value 1
print x # refers to attribute x with value 1
print self.y # refers to attribute y with value 2
print y # refers to global y

In existing Python, both x and y will refer to globals, because it's
considered more important for all attribute access to be consistently
explicit than to insert the class scope into the function scope. This
isn't a design flaw, or a bug, it's a feature. Yes, it makes it hard for
you to solve your problem the way you want to solve it, but it doesn't
stop you from solving your problem. The module is encapsulation enough.

 

[Ulrich Eckhardt]

class Demo(object):
def fact_iter(n):
ret = 1
for i in range(1, n + 1):
ret *= i
return ret

def fact_rec(n):
if n < 2:
return 1
else:
return n * fact_rec(n - 1)

classvar1 = fact_iter(5)
classvar2 = fact_rec(5)


In the initialization of classvar1, fact_iter is invoked without
any problem even though the class is not yet created: no need to
qualify it with the name of the class. This is completely inconsistent
with the requirement that fact_rec be so qualified when invoked
within fact_rec.

Let me take a shot at explaining this, maybe it helps that I'm mostly a C++
guy....


Firstly, if Python sees a name, it looks up that name first in the local
scope and then in the global scope. This is very simple (especially
compared to C++ when adding ADL there...), but it is something most people
can live with. So, even inside a class function (static or member), you
have to qualify a function call like 'ClassName.function_name', because
neither is 'function_name' a local nor is it a global.

Secondly, and that is due to the very dynamic nature of Python's types, the
class doesn't exist until its definition is finished. Therefore, any
attempt (directly or indirectly) to access a class member will usually fail
before that time. The exception is that inside the class definition you can
access the members directly, because they are in the same scope (access to
locals). Note that the scope already seems to exist but that it is not yet
available under any name! Looking at your example, you can not
write 'classvar1 = Demo.fact_iter(42)', because the lookup of 'Demo' will
fail.

Now, what actually causes problems is that 'fact_rec' is not universally
accessible until class 'Demo' is fully defined, but you need this in order
to fully define it. Here comes a drawback of the dynamic nature of Python,
that the declaration phase (compiling) is not separate from the execution.

I fully agree that this case is rather vexing to my (and obviously your)
biased brain. I wouldn't call this a bug before fully understanding why
e.g. you can not access a class before its definition is finished. I think
someone mentioned one or two PEPs, which are the design documents that
explain Python including the rationale, I'd use that as a starting point.


Suggestion: someone mentioned that you should make the function a normal
function. You can mark it as private using an underscore as prefix. If you
want to make it foolproof, you could even delete ("del fact_rec") the
function after use. Other alternatives would be to put it into a separate
baseclass, use a local function as implementation or to add the class
attributes after defining the class. Neither of these provide the 'perfect'
encapsulation of things in a class that you might be used to from e.g.
Java, but they have proven to be very useful nonetheless.


Uli

 

[Terry Reedy]

Classes are objects. In particular, they are (by default) instances of
class 'type'. Unless 'scopes' were instances of some other metaclass,
the statement has to be true. I understand 'scope' as referring to a
section of code, as opposed to a runtime object.

Class statements introduce a new local namespace used to define the
class. Whether one considers that as introducing a new 'scope' depends,
I suppose, on one's exact definition of scope.

This looks to me like a major wart, on two counts.

It is a 'wart' that a Python object is not a 'scope', whatever that is
to you? I have trouble comprehending that claim.

First, one of the goals of OO is encapsulation, not only at the
level of instances, but also at the level of classes. Your comment
suggests that Python does not fully support class-level encapsulation.

I really do not see how your claim follows from the comment. The irony
of your 'two counts' is that the example of 'count 2' fails precisely
because of the encapsulation that you claim does not exist as 'count 1'.

Second, my example shows that Python puts some peculiar restrictions
on recursion.

I claim it does not. Name-based recursion inherently requires that a
function be able to access itself by name at the time it is called. This
can be a bit tricky, especially in a language with dynamic rather than
static name binding and resolution, as it requires that code within the
function be able to access a scope outside itself in which the function
name is defined. In other words, it requires that function code *not* be
completely encapsulated. It also require that the appropriate name be
used when there is one. Neither of these is a 'peculiar restriction'
imposed by Python.

class Demo(object):
def fact_rec(n):
if n < 2:
return 1
else:
return n * fact_rec(n - 1)

This function is just a function. It is not an instance method. It is
not even a class method. It does not belong here and should not be here.

If you insist on putting it where it does not belong, then you have to
call it by a name that works. If you only call it after the class
statement has finished, then 'Demo.fact_rec' works, as I believe someone
else pointed out.

If you want to call the function during class creation, before (in this
case) Demo exists, then binding it to a local name works. With 3.1

class Demo:
def f1(n):
if n < 2: return 1
else: return n * Demo.f1(n - 1)

def f2(n,f):
if n < 2: return 1
else: return n * f(n - 1, f)

cvar = f2(5, f2)

print(Demo.f1(5), Demo.cvar, Demo.f2(5,Demo.f2))

# prints120 120 120

Recursive functions should be OK wherever functions are OK.

Iteration can and has been viewed as within-frame recursion. When
iterative rather than recursive syntax is used, the naming issue is
avoided.

Is there any good reason (from the point of view of Python's overall
design) for not fixing this?

After reading the above, what, if anything, do you think still needs to
be fixed?

Before 2.2, Python functions were more encapsulated than they are today
in that they could only access local and global namespaces but not outer
function namespaces. It would be possible to further de-encapsulate
them by giving them direct access to lexically surrounding class
namespaces, but this would increase the problem of name clashes without
solving any real problems in proper Python code. It could also break the
intentional design principle that function code should mean the same
thing whether placed within or without a class statement. This principle
allows functions to be added to existing classes as attributes and work
as methods that same as if they had been defined with the class.

Terry Jan Reedy

 

[Ethan Furman]

I have many years of programming experience, and a few languages,
under my belt, but still Python scoping rules remain mysterious to
me. (In fact, Python's scoping behavior is the main reason I gave
up several earlier attempts to learn Python.)

Here's a toy example illustrating what I mean. It's a simplification
of a real-life coding situation, in which I need to initialize a
"private" class variable by using a recursive helper function.

class Demo(object):
def fact(n):
if n < 2:
return 1
else:
return n * fact(n - 1)

_classvar = fact(5)

As has been pretty thoroughly discussed, the issue here is not
recursion, but name lookup.

Going back through the archives I found Arnaud's post with this decorator:

def bindfunc(f):
def boundf(*args, **kwargs):
return f(boundf, *args, **kwargs)
return boundf

If you use it on your fact function like so...

class Demo(object):
@bindfunc
def fact(recurse, n) # recurse can be any name you like
if n < 2:
return 1
else:
return n * recurse(n-1)
_classvar = fact(5)
del fact # no longer needed, and won't work
# once class is created

This should do as you want.

As a side note, if you're going to bother asking questions on this list,
you really should try to understand the answers. I won't gripe at you
too much, though, 'cause I learned a lot from the many responses given
due to your refusal to do so. ;-)

~Ethan~

 

[kj]

http://docs.python.org/reference/executionmodel.html

It is also discussed in the PEP introducing nested scopes to Python:

It's even eluded to in the tutorial:

http://docs.python.org/tutorial/classes.html

Thanks!

kynn

 

[Dave Angel]

This won't work normally. It only worked for you because you made a
typo.


Carl Banks

Sorry about the typo. I was trying out several different versions of
the class in the same module, and forgot to include to change Demo to
Demo9 in the recursive call.

I didn't like that approach anyway, as it smacked of taking advantage of
some implementation accident. The other approaches are more
straightforward.


DaveA

 

[kj]

Going back through the archives I found Arnaud's post with this decorator:

def bindfunc(f):
def boundf(*args, **kwargs):
return f(boundf, *args, **kwargs)
return boundf

If you use it on your fact function like so...

class Demo(object):
@bindfunc
def fact(recurse, n) # recurse can be any name you like
if n < 2:
return 1
else:
return n * recurse(n-1)
_classvar = fact(5)
del fact # no longer needed, and won't work
# once class is created

This should do as you want.

Thanks, this is instructive.

As a side note, if you're going to bother asking questions on this list,
you really should try to understand the answers.

I think I understand the answers well enough. What I *really*
don't understand is why this particular "feature" of Python (i.e.
that functions defined within a class statement are forbidden from
"seeing" other identifiers defined within the class statement) is
generally considered to be perfectly OK. IMO it's a bizarre,
inexplicable blindspot (which, among other things, gives rise to
a certain worry about what other similar craziness lurks under
Python's image of rationality). I have never seen even a half-hearted
justification, from a language design point of view, for why this
particular "feature" is worth having. Maybe some day the BDFL will
deign to give one.

kynn

 

[kj]

But if you insist on the above methodology, you can do this:

class Demo(object):

def fact(n):
def inner(n):
if n < 2:
return 1
else:
return n * inner(n - 1)
return inner(n)

_classvar = fact(5)

This makes inner a *local* variable, which is found.

Thanks for this. I think this is the most straightforward workaround.

kynn

 

[kj]

Let me take a shot at explaining this, maybe it helps that I'm mostly a C++
guy....

Thanks for your reply.

I fully agree that this case is rather vexing to my (and obviously your)
biased brain. I wouldn't call this a bug before fully understanding why
e.g. you can not access a class before its definition is finished.

I understand this, what I don't understand is why do it this way.
I've been trying to understand this particular design point for
*literally* years.

I think
someone mentioned one or two PEPs, which are the design documents that
explain Python including the rationale, I'd use that as a starting point.

I'm reading PEP 227 now, suggested by Steven D'Aprano. In it I
find statements like the following alert:

(Note: If a region is contained within a class definition, the
name bindings that occur in the class block are not visible to
enclosed functions.)

I've seen this before, but never an explanation for why having this
restriction. It's just one of life's mysteries. (Incidentally,
the fact that the author of PEP 227 felt it necessary to add that
parenthetical remark suggests that the expectation it warns against
is not so crazy after all.)

But I'm still not done with PEP 227. Maybe I'll see the light by
the time I'm done.

kynn

 

[Jan Kaliszewski]

class Color:
...

setattrib(Color, "BLACK", Color(0,0,0))

Or simpler:

class Color:
...

Color.BLACK = Color(...)

Then
(Color.BLACK is Color.BLACK.BLACK.BLACK.BLACK) == True


*j

PS. Obviously, that's nothing special (there is no problem with
creating such "recursive" references in Python).

 

[Ethan Furman]

I think I understand the answers well enough. What I *really*
don't understand is why this particular "feature" of Python (i.e.
that functions defined within a class statement are forbidden from
"seeing" other identifiers defined within the class statement) is
generally considered to be perfectly OK. IMO it's a bizarre,
inexplicable blindspot (which, among other things, gives rise to
a certain worry about what other similar craziness lurks under
Python's image of rationality). I have never seen even a half-hearted
justification, from a language design point of view, for why this
particular "feature" is worth having. Maybe some day the BDFL will
deign to give one.

kynn

You keep using that word. I do not think it means what you think it
means.

Keep studying. Listen. Learn how it *is*. Understanding may come later.

~Ethan~

 

[Jan Kaliszewski]

In <[email protected]> Steven D'Aprano



Because, as I've explained elsewhere, it is not a method: it's a
"helper" function, meant to be called only once, within the class
statement itself.

Well, this is not strictly true, because the function is recursive,
so it will also call itself a few times. But still, all these
calls happen (loosely speaking) "within the class statement".

In fact the only reason to use a function for such initialization
work is when one needs recursion; otherwise there's no point in
defining a function that will only be invoked exactly once.

1. I don't understand then... Why do you desire to both define and
run it *within* that class statement as if it was this class's
method?

2. Could you please show me how it could be done in C++ or Java?
(Or you want to say that that languages also are not fully valuable
OO languages?)

3. Python makes function bodies "agnostic" about the context of their
definition -- generally any non-global information must be passed
explicitly to their interior. *It has nothing to do with recursion.*

If you really must both define and use such a function within the
class definition, pass function object to itself explicitly, and
everybody will be happy:


class Demo(object):

def fact(fact, n):
if n < 2:
return 1
else:
return n * fact(fact, n - 1)

fact(fact, 3)


*j