Need help with Python scoping rules

[Hendrik van Rooyen]

As I described at length in another reply, the function in question
is not intended to be "callable outside the class". And yes,

I think this might go to nub of your problem - It might help you to think as
follows:

A Python class, even after it has been executed, does not really exist except
as a kind of template or pattern - it is mostly useless until an instance of
the class is made by calling it with whatever it needs to set up the
instance. And once you have an instance, you can do stuff with that
particular instance. Before that time, the class is mostly just a promise of
things to come.

<aside>
The other thing to bear in mind is that you cannot really hide stuff in python
classes - there are no really private things.

recursive functions in Python *are* restricted in ways that
non-recursive functions aren't. The examples I've posted prove
this point unambiguously.

Yes and no - mostly no - your examples just illustrate the point I tried to
make above.

Pythons object model, and its classes, are different from what you are used
to. A bare class is mostly useless without an instance, which is ultimately
why accessing a function in a class from itself like you are doing, without
reference to an instance, does not work - the function does not exist yet to
a degree that it can be referenced. It is kind of subtle, and different from
other languages.

8<------------- rant against the way things are -----------------------

Welcome to python, and the newsgroup, by the way.

- Hendrik

 

[Hendrik van Rooyen]

In <[email protected]> Steven D'Aprano

Because, as I've explained elsewhere, it is not a method: it's a
"helper" function, meant to be called only once, within the class
statement itself.

If the sole purpose of the function is to be used to define what will become a
constant, why do you not just calculate the constant on your calculator, or
at the interactive prompt, and assign it to the attribute, and be done with
it?

Why waste run time recalculating every time the programme is called?

Well, this is not strictly true, because the function is recursive,
so it will also call itself a few times. But still, all these
calls happen (loosely speaking) "within the class statement".

Why *must* it be there?

In fact the only reason to use a function for such initialization
work is when one needs recursion; otherwise there's no point in
defining a function that will only be invoked exactly once.

Seems no point to me even when there is recursion - a number is a number is a
number. If you know the arguments when you write it, and if the function is
known at time of writing, storing a literal is the best solution.

Conversely, if the arguments are not known at time of writing, then they
should be passed at run time, when an instance is created, and assigned as
part of the __init__ method.

And the same is true if the function is not known at time of writing - then it
must be passed to the constructor of the instance.

Storm in a teacup.

- Hendrik

 

[Terry Reedy]

I think I understand the answers well enough. What I *really*
don't understand is why this particular "feature" of Python (i.e.
that functions defined within a class statement are forbidden from
"seeing" other identifiers defined within the class statement) is
generally considered to be perfectly OK. IMO it's a bizarre,
inexplicable blindspot (which, among other things, gives rise to
a certain worry about what other similar craziness lurks under
Python's image of rationality). I have never seen even a half-hearted
justification, from a language design point of view, for why this
particular "feature" is worth having. Maybe some day the BDFL will
deign to give one.

I attempted to give an explanation in my previous post.

 

[Bruno Desthuilliers]

Ulrich Eckhardt a écrit :
(snip)

Now, what actually causes problems is that 'fact_rec' is not universally
accessible until class 'Demo' is fully defined, but you need this in order
to fully define it. Here comes a drawback of the dynamic nature of Python,
that the declaration phase (compiling) is not separate from the execution.

It is. You can manually compile a module, remove the source .py and
still import (or execute) the compiled .pyc.

The "problem" here is not with compilation, but with the fact that
'class' (and 'def') statements are *executable* statements. IOW, the
class statement is executed when encountered, that is, usually (most
class statements being top-level statements), when the module is first
loaded.

I fully agree that this case is rather vexing to my (and obviously your)
biased brain. I wouldn't call this a bug before fully understanding why
e.g. you can not access a class before its definition is finished.

Nor why it's a GoodThing(tm) for quite a lot of use case - while the
OP's problem is in the splitting-hairs category - only a problem if yoçu
insist in imposing your views on the language instead of learning how to
use it.

Suggestion: someone mentioned that you should make the function a normal
function.

(snip)

Someone (Diez Roggisch IIRC) also mentionned a simple solution that
mostly fulfills the OP's desires: use a nested function.

 

[Bruno Desthuilliers]

kj a écrit :
(snip)

I understand this, what I don't understand is why do it this way.
I've been trying to understand this particular design point for
*literally* years.

It's quitye simple: 'class' is an executable statement that creates a
class object (yes, all this happens at runtime) and bind it in the
current namespace. So until the class statement is executed, the class
object doesn't exist and it's name is not bound.

Now if you don't understand why it's a GoodThing(tm) that all this
happens at runtime, have a look at metaclasses, how they are used in
most major Python frameworks, and how this allows to vastly reduce the
amount of needed boring and erreor-prone boilerplate code one cand find
elsewhere.

I'm reading PEP 227 now, suggested by Steven D'Aprano. In it I
find statements like the following alert:

(Note: If a region is contained within a class definition, the
name bindings that occur in the class block are not visible to
enclosed functions.)

I've seen this before, but never an explanation for why having this
restriction.

You should re-read Carl Bank's answers more carefully then.

It's just one of life's mysteries. (Incidentally,
the fact that the author of PEP 227 felt it necessary to add that
parenthetical remark suggests that the expectation it warns against
is not so crazy after all.)

It's not so crazy for someone coming from a more static language.
Python's object model is indeed a bit peculiar, and it can take some
times getting the big picture, but be sure it has been carefully
designed, and is incredibly powerfull once you start understanding the
whole thing.

But I'm still not done with PEP 227. Maybe I'll see the light by
the time I'm done.

My 2 cents : learn about metaclasses, attributes lookup rules, and the
descriptor protocol (and some practical applications of these features).
Then you might decide that the few resulting restrictions are really
worth the price.

 

[Bruno Desthuilliers]

kj a écrit :

It does: it thwarts encapsulation. The helper function in my
example is one that clearly rightfully belongs nowhere else than
the class itself, i.e. encapsulated within the class.

I don't see what's wrong with having this function at the top-level.
Sorry to have to say it this way, but IMHO you're border on cargo-cult
thinking here. Python is an all-object language (ie : everything is an
object, including functions, classes and modules), but it's by no mean
'pure-object', and module level functions are perfectly ok (and quite
often the right thing).

Now if you really want to "encapsulate" the function and the class
together, you do have simple and and working solutions : using a nested
recursive function (as explained by Diez), or nesting both the class and
function in a factory function, ie:

def Demo():
def fact(n):
if n < 2:
return 1
else:
return n * fact(n - 1)
class Demo(object):
_classvar = fact(5)
return Demo

Demo = Demo()


But anyway: this is really a WTF. Python's notion of encapsulation is
very weak (by design), and you're really wasting time trying to fight
the language instead of learning it. Just put that f... helper function
at the top level (prefixing it's name with a '_' to mark it as
implementation detail), and move on to something else !-)

It is only
this silly prohibition against recursive functions in a class
statement that forces one to put it outside the class statement.

The "prohibition" only happens for recursive functions defined *and*
called in the class statement. And once you get the big picture, it's
certainly not "silly".

 

[Bruno Desthuilliers]

kj a écrit :

Python itself: it already offers a limited form of class encapsulation
(e.g. class variables).

"class variables" - which FWIW include the "methods" - are just
attributes of the class *object* (reminder : Python's classes and
functions *are* objects too). Any name bound at the top level of the
class statement scope becomes an attribute of the class object. IOW,
this "limited form of class encapsulation" is just the ordinary "form of
encapsulation" you'll get with Python objects.

It would be nice if it went all the way
and gave classes their own bona fide scope. (But I hasten to add:
I *still* don't understand the Python scope model,

I think that what you should first understand are Python's execution and
object models.

...except, apparently, when that code is a recursive function, in
which case one's out of luck.

I wrote quite a few recursive functions in Python and never had any
problem.

It's not a restriction on recursion, it's a scoping rule that apply to
any other name. You can't access the class statement's scope from within
a function, period.

If recursion is so evil, and Python so intent in saving us from
shooting ourselves in the foot, why does it allow recursion at all?

Recursion is not evil, and Python doesn't try to prevent anyone from
doing stupid things anyway. FWIW, do you know that you can add / replace
/ remove attributes (including methods) at runtime, both on a
per-instance or per-class basis ? And even dynamically change the class
of an object ?-)

 

[Steven D'Aprano]

I think this might go to nub of your problem - It might help you to
think as follows:

A Python class, even after it has been executed, does not really exist
except as a kind of template or pattern - it is mostly useless until an
instance of the class is made by calling it with whatever it needs to
set up the instance. And once you have an instance, you can do stuff
with that particular instance. Before that time, the class is mostly
just a promise of things to come.

Oh my! I couldn't disagree more strongly! I think the above is totally
incorrect.

Classes and types are first class objects, you can treat them like
anything else in Python. In fact, classes themselves are instances of
type, so you can say:
.... pass
....True

Classes are themselves instances of their metaclass. By default, classes
have a metaclass of type, but you can easily change that. Metaclass
programming is advanced but very powerful.

Because classes are themselves objects, you can (with a bit of metaclass
jiggery-pokery) make them do all sorts of interesting things. For
instance, huge amounts of effort are often put into creating a Singleton
class, a class that has a single instance. Well, okay... but why not just
use the class object itself, instead of an instance? There's already one
of those, and you can't (easily) make a copy of it, and even if you did,
it would be an independent class. Instead of this:

singleton = SingletonClass(args)
do_something_with(singleton)

just do this:

do_something_with(SingletonClass)

Of course SingletonClass needs to be designed to work that way, which
will probably need some metaclass magic. It would be interesting to see
which requires less effort.

When it comes to built-in classes (types), I often use the class object
itself as an object. E.g. I might do something like this:

def convert(seq):
t = type(seq) # remember the original type
result = process(seq) # always produces a list
if t is list:
return result # don't bother making a copy of the result
elif t is str or t is unicode:
empty = t()
return empty.join(result)
else:
return t(result) # return the original type

Yes and no - mostly no - your examples just illustrate the point I
tried to make above.

Completely no. You may have missed the examples I've given, but the
problems the Original Poster were having had nothing to do with recursion.

Pythons object model, and its classes, are different from what you are
used to. A bare class is mostly useless without an instance, which is
ultimately why accessing a function in a class from itself like you are
doing, without reference to an instance, does not work - the function
does not exist yet to a degree that it can be referenced.

That is incorrect. What's going on is more subtle.

.... def function(x):
.... print "Calling function with argument %s" % x
.... function(None)
.... function(1)
.... function(function)
....
Calling function with argument None
Calling function with argument 1
<class __main__.Demo at 0xb7d4e0ec>

 

[Steven D'Aprano]

If the sole purpose of the function is to be used to define what will
become a constant, why do you not just calculate the constant on your
calculator, or at the interactive prompt, and assign it to the
attribute, and be done with it?

Why waste run time recalculating every time the programme is called?

What you are calculating might actually be quite complicated to enter as
a literal. You might have something like:

class C(object):
TABLE = []
for i in xrange(100):
row = []
for j in xrange(100):
row.append((i**3 + 2*i**2 - i*j + 7*j**2 + 9*i*j**2 - j)/3)
TABLE.append(row)

It's a little hard to type out C.TABLE as a literal.

And even if you could, which would you rather see if you were debugging
this class? The above, or:

class C(object):
TABLE = [
[0.0, 2.0, 8.6666666666666661, 20.0, 36.0, ... ],
[1.0, 5.666666666666667, 21.0, 47.0, 83.666666666666671, ...],
[5.333333333333333, 12.666666666666666, 36.666666666666664, ...],
...
[... , 3143161.0, 3201497.6666666665, 3260433.0]
]


For obvious reasons I haven't typed the whole thing out! And imagine
trying to check it for typos!

Clearly this is a made-up example, but the principle is sound. If Python
can calculate values for you, why not let it do so? It is easier for you,
easier to check that you've calculated them correctly, easier to debug
and read, it makes the algorithm clearer (fewer "magic constants"). Yes,
there is a run-time cost, but you only pay it once, when you import the
module, and it's likely to be not that much more expensive than parsing
the literals anyway.

Of course, for something as big and complicated as the above table, I'd
almost certainly put the code to calculate it in a function outside of
the class, but that's a matter of style, and it will work to put it
inside the class.

 

[greg]

No, the fact() function here represents an internal "helper"
function. It is meant to be called only once to help initialize
a class variable that would be inconvenient to initialize otherwise;
this helper function is not meant to be called from outside the
class statement.

That, to me, is an excellent indication that the function
does *not* belong inside the class!

The only things that belong inside the class are things
that users of the class will need to use, or that instances
of the class will need to do their job.

Your helper function belongs outside the class, in the
module where the class is defined.

 

[Jean-Michel Pichavant]

Apart from it being "setattr" and not "setattrib", a simple

Color.BLACK = Color(0,0,0)

Obviously ... I'll remember that.

JM

 

[Jean-Michel Pichavant]

I think I understand the answers well enough. What I *really*
don't understand is why this particular "feature" of Python (i.e.
that functions defined within a class statement are forbidden from
"seeing" other identifiers defined within the class statement) is
generally considered to be perfectly OK. IMO it's a bizarre,
inexplicable blindspot (which, among other things, gives rise to
a certain worry about what other similar craziness lurks under
Python's image of rationality). I have never seen even a half-hearted
justification, from a language design point of view, for why this
particular "feature" is worth having. Maybe some day the BDFL will
deign to give one.

kynn

I think I got your point.
I guess many people may not be receptive to your question, cause guess
what, we're all python fans )

in foo.py:

a = 5
b = a # works fine

class A:
c = 5
d = c # broken
d = A.c # broken either

def foo(self):
e = 5
f = e #works fine


We should all acknowledge that any newcomer to python will not expect
such behavior. There are plenty of good answers to that thread
explaining why the fact that classes are not scopes is much better.
Still this design fails at one point : insight.
It may be solved by creating the class upon the "class" statement. If
the class A object is created, then c is added as a property of that
object, there's no problem accession one object property with A.c.

JM

 

[kj]

I think I got your point.
I guess many people may not be receptive to your question, cause guess
what, we're all python fans )

in foo.py:

a = 5
b = a # works fine

class A:
c = 5
d = c # broken
d = A.c # broken either

def foo(self):
e = 5
f = e #works fine

We should all acknowledge that any newcomer to python will not expect
such behavior. There are plenty of good answers to that thread
explaining why the fact that classes are not scopes is much better.
Still this design fails at one point : insight.
It may be solved by creating the class upon the "class" statement. If
the class A object is created, then c is added as a property of that
object, there's no problem accession one object property with A.c.

Thanks! I was beginning to think I'd gone crazy.

kynn

 

[Steven D'Aprano]

in foo.py:

a = 5
b = a # works fine

class A:
c = 5
d = c # broken

Incorrect. That works fine.
.... c = 5
.... d = c # not actually broken
....5

The class is a scope, and inside the class scope, you can access local
names. What you can't do is access the class scope from inside nested
functions.

 

[kj]

in foo.py:

a = 5
b = a # works fine

def foo(self):
e = 5
f = e #works fine

It may be solved by creating the class upon the "class" statement. If
the class A object is created, then c is added as a property of that
object, there's no problem accession one object property with A.c.

I thought one could implement something like

class A:
c = 5
d = __thisclass__.c

But apparently this is a *huge* deal. I'm not sure if the reason
it is a huge deal is that it is technically difficult to implement,
or that there is an implicit "lie" in having something (__thisclass__)
standing for something else that doesn't yet exist (and the potential
errors that may arise from this "lie" in sufficiently unfortunate
code). Or maybe something else beyond my noobish ken altogether.

kynn

 

[kj]

What you are calculating might actually be quite complicated to enter as
a literal.

Thank you!

kynn

 

[Bruno Desthuilliers]

Jean-Michel Pichavant a écrit :

I think I got your point.
I guess many people may not be receptive to your question, cause guess
what, we're all python fans )

in foo.py:

a = 5
b = a # works fine

class A:
c = 5
d = c # broken

Err... Did you actually tried this ?

.... c = 5
.... d = c
....

d = A.c # broken either

Not "broken" : the class doesn't yet exists, nor is it bound to global
name 'A'. FWIW, *this* works (for some definitions of 'works') juts fine:
.... c = 42
........ d = A.c
....

42




We should all acknowledge that any newcomer to python will not expect
such behavior.

Any newcomer to any language should aknowledge that her expectations
based on previous experience with any other language should be kept
aside or just plain abandoned, and start by learning the new language.

The only thing one is entitled to expect when learning a new language is
that the language's implementation follows the language specs.

There are plenty of good answers to that thread
explaining why the fact that classes are not scopes is much better.
Still this design fails at one point : insight.

Oh, really ?

It may be solved by creating the class upon the "class" statement. If
the class A object is created, then c is added as a property of that
object, there's no problem accession one object property with A.c.

Please, write a pep...

 

[D'Arcy J.M. Cain]

So why didn't you delete it after you were done with it?

Class Demo(object):

That should be "class".

_classvar = fact(5)
del fact()

I suppose you mean "del fact".

 

[Hendrik van Rooyen]

Oh my! I couldn't disagree more strongly! I think the above is totally
incorrect.

It was intended to help the OP out of the mental hole he finds himself in.

Most of the classes I use fall directly into such a classification - they are
useless until an instance is created. And I would be so bold as to say that
_all_ gui classes are like that.

This is the pattern I am talking about:

class thing(object):
def __init__(self,foo,bar):
stuff to do things with foo and bar,
creating or modifying attributes of
the instance.

def somemethod(self,baz,bling):
instance method to do further operations on
the attributes of the instance

Now kindly explain to me how a class like that is usefull before an instance
of it is created, and I might agree with you that what I said is "totally
incorrect"

8< --trivial examples showing that there is something there ------------

Classes are themselves instances of their metaclass. By default, classes
have a metaclass of type, but you can easily change that. Metaclass
programming is advanced but very powerful.

Because classes are themselves objects, you can (with a bit of metaclass
jiggery-pokery) make them do all sorts of interesting things. For
instance, huge amounts of effort are often put into creating a Singleton
class, a class that has a single instance. Well, okay... but why not just
use the class object itself, instead of an instance? There's already one
of those, and you can't (easily) make a copy of it, and even if you did,
it would be an independent class. Instead of this:

singleton = SingletonClass(args)
do_something_with(singleton)

just do this:

do_something_with(SingletonClass)

Of course SingletonClass needs to be designed to work that way, which
will probably need some metaclass magic. It would be interesting to see
which requires less effort.

*nods* yes this would make sense - but it is not quite what either the OP or I
was on about.

When it comes to built-in classes (types), I often use the class object
itself as an object. E.g. I might do something like this:

def convert(seq):
t = type(seq) # remember the original type
result = process(seq) # always produces a list
if t is list:
return result # don't bother making a copy of the result
elif t is str or t is unicode:
empty = t()
return empty.join(result)
else:
return t(result) # return the original type

nice. now try doing it with object:

thing = object()

After that, thing will exist, but it is a bit like write only memory -
completely useless, until you have done some more work with it.

Completely no. You may have missed the examples I've given, but the
problems the Original Poster were having had nothing to do with recursion.

The yes was in there to make the man feel a bit better, and because from where
he stood, it _was_ the recursion that did him in. : - )

That is incorrect. What's going on is more subtle.

Right - I can see that you are reading that to mean that there must be an
instance. That is not what I intended to bring across. I was talking about
the lack of a reference that is his original problem, which he only
encountered with recursion.

... def function(x):
... print "Calling function with argument %s" % x
... function(None)
... function(1)
... function(function)
...
Calling function with argument None
Calling function with argument 1


<class __main__.Demo at 0xb7d4e0ec>

two points:

Of what earthly use is a bare class like that - except possibly as a container
for all the good stuff those functions produced out of nothing? - You might
as well use a list, or better a dict, to keep the results of such functions,
unless the functions are procedures, doing things by side effect - and if
that is the case, why bother putting them in a class?

Secondly:

Now what you have done is carefully avoided the OP's original problem, which
arises when the function in the class is called, and tries to call itself at
the time of execution of the class suite, and fails, for the reasons so
nicely explained by a lot of people. So the OP had a point when he
complained that python does not like recursion, because, until he understands
what is happening, the simplest theory that fits his observed facts is that
the failure is caused by the recursion. - a completely rational conclusion
based on Occam's razor. It just happens to be wrong in this case. This was
the source for my "Yes" above.

This should also work for the OP - I cannot recall if it has been shown:

class Demo(object):
def fact(x,f):
if x < 2:
return 1
else:
return x*f(x-1,f)

thing = fact(42,fact)

I still think it is stupid, though - thing should simply be set to
1405006117752879898543142606244511569936384000000000L
and that would be the end of it for this use case, without the bother of a
function and the concomitant waste of time.

- Hendrik

 

[Hendrik van Rooyen]

What you are calculating might actually be quite complicated to enter as
a literal. You might have something like:

8< ---------- Complicated Table Example -------------------------------

Me? - never! I am just an assembler programmer. I would not touch a thing
like that with a barge pole. Unless of course, I have to.

Clearly this is a made-up example, but the principle is sound. If Python
can calculate values for you, why not let it do so? It is easier for you,
easier to check that you've calculated them correctly, easier to debug
and read, it makes the algorithm clearer (fewer "magic constants"). Yes,
there is a run-time cost, but you only pay it once, when you import the
module, and it's likely to be not that much more expensive than parsing
the literals anyway.

Of course, for something as big and complicated as the above table, I'd
almost certainly put the code to calculate it in a function outside of
the class, but that's a matter of style, and it will work to put it
inside the class.

It is a hell of a thing if it needs recursion to calculate - If it was really
that complex, I would calculate it, check it (if I can), document it and put
it in a module of its own, with "This side up", "Fragile", and other warning
stickers all over it.

- Hendrik