C-style static variables in Python?

[kj]

When coding C I have often found static local variables useful for
doing once-only run-time initializations. For example:

int foo(int x, int y, int z) {

static int first_time = TRUE;
static Mongo *mongo;
if (first_time) {
mongo = heavy_lifting_at_runtime();
first_time = FALSE;
}

return frobnicate(mongo, x, y, z);
}

In this case, the static variable mongo is initialized only once
(at most).

What I like most about this is that it obviates the need for a
global variable to hold the persistent value (I avoid globals like
the plague, especially in Python). It also nicely encapsulates
the logic that determines whether initialization is required.

The best way I've found to achieve a similar effect in (procedural)
Python defines the function as a closure. For example, here's a
function that keeps track of (and prints out) how many times it
has been called:
.... counter = [0]
.... def _():
.... counter[0] += 1
.... print counter[0]
.... return _
.... 3

(Too bad that one can't stick the whole def inside parentheses and
call the function right there, like one can do with JavaScript.)

Another approach would be to stuff the static values in the function's
__dict__. This is less satisfactory than the closure approach
because the "pseudo-static" variable is accessible from outside
the function, but the code is arguably a little more straightforward,
and one does not end up with the now useless one-time closure-generating
function kicking around. Here's another version of the function
above:
.... d = spam.__dict__
.... if not 's' in spam.__dict__:
.... spam.s = 1
.... print spam.s
.... spam.s += 1
.... 3

Besides the external accessibility issue, I don't like explictly
coding the name of the function within the function. Is there any
way to have the function access its own __dict__ without having to
explicitly code its name in its body? E.g., is there some generic
special variable that, within a function, refers to the function
object itself?

I'm sure that there are many other ways to skin this cat, especially
if one starts definining fancy callable classes and whatnot. But
is there a better *simple* way to achieve C-style static locals in
Python that does not require a lot of extra machinery?

TIA!

~K

 

[Chris Rebert]

When coding C I have often found static local variables useful for
doing once-only run-time initializations.

Another approach would be to stuff the static values in the function's
__dict__.  This is less satisfactory than the closure approach
because the "pseudo-static" variable is accessible from outside
the function, but the code is arguably a little more straightforward,
and one does not end up with the now useless one-time closure-generating
function kicking around.  Here's another version of the function
above:

...     d = spam.__dict__
...     if not 's' in spam.__dict__:
...         spam.s = 1
...     print spam.s
...     spam.s += 1
...
3

Besides the external accessibility issue, I don't like explictly
coding the name of the function within the function.  Is there any
way to have the function access its own __dict__ without having to
explicitly code its name in its body?  E.g., is there some generic
special variable that, within a function, refers to the function
object itself?

Nope. It's been proposed in that past
(http://www.python.org/dev/peps/pep-3130/), but that proposal was
rejected.

I'm sure that there are many other ways to skin this cat, especially
if one starts definining fancy callable classes and whatnot.  But
is there a better *simple* way to achieve C-style static locals in
Python that does not require a lot of extra machinery?

You can abuse the default argument value mechanism:

def spam(s_cell=[1]):
s = s_cell[0]
print s
s_cell[0] += 1

It's a bit less ugly when the value itself is mutable, which isn't the
case here with the integer.

Personally, I hate such abuse with a passion; I think a global
variable is clearest.

Cheers,
Chris

 

[Steve Holden]

When coding C I have often found static local variables useful for
doing once-only run-time initializations.

Another approach would be to stuff the static values in the function's
__dict__. This is less satisfactory than the closure approach
because the "pseudo-static" variable is accessible from outside
the function, but the code is arguably a little more straightforward,
and one does not end up with the now useless one-time closure-generating
function kicking around. Here's another version of the function
above:

... d = spam.__dict__
... if not 's' in spam.__dict__:
... spam.s = 1
... print spam.s
... spam.s += 1
...
3

Besides the external accessibility issue, I don't like explictly
coding the name of the function within the function. Is there any
way to have the function access its own __dict__ without having to
explicitly code its name in its body? E.g., is there some generic
special variable that, within a function, refers to the function
object itself?

Nope. It's been proposed in that past
(http://www.python.org/dev/peps/pep-3130/), but that proposal was
rejected.

I'm sure that there are many other ways to skin this cat, especially
if one starts definining fancy callable classes and whatnot. But
is there a better *simple* way to achieve C-style static locals in
Python that does not require a lot of extra machinery?

You can abuse the default argument value mechanism:

def spam(s_cell=[1]):
s = s_cell[0]
print s
s_cell[0] += 1

It's a bit less ugly when the value itself is mutable, which isn't the
case here with the integer.

Personally, I hate such abuse with a passion; I think a global
variable is clearest.

But the real problem is that the OP is insisting on using purely
procedural Python when the problem is screaming for an object-oriented
answer.

If the function were instead a method then the instance namespace would
be the logical place to store the required data.

regards
Steve
regards
Steve

 

[Terry Reedy]

When coding C I have often found static local variables useful for
doing once-only run-time initializations. For example:

int foo(int x, int y, int z) {

static int first_time = TRUE;
static Mongo *mongo;
if (first_time) {
mongo = heavy_lifting_at_runtime();
first_time = FALSE;
}

return frobnicate(mongo, x, y, z);

Global var or class or closure such as below (obviously untested ;=):

make_foo()
mongo = heavy_lifting_at_runtime();
def _(x,y,z):
return frobnicate(mongo, x, y, z)
return _
foo = make_foo
del make_foo # to make sure it is *never* called again ;

Now you only have foo with a hard-to-access private object and no
first_time checks when you call it.

Terry Jan Reedy

 

[Patrick Maupin]

But the real problem is that the OP is insisting on using purely
procedural Python when the problem is screaming for an object-oriented
answer.

If the function were instead a method then the instance namespace would
be the logical place to store the required data.

In some situations I will use either the default parameter
initialization Chris mentioned, or the closure mechanism that the OP
presented, but only on code that I am optimizing for speed (local
variable lookups, even in nested functions, being much faster than
global or instance lookups). If it doesn't need to go fast, globals
or instance variables are the way to go.

Pat

 

[Steve Holden]

Global var or class or closure such as below (obviously untested ;=):

make_foo()
mongo = heavy_lifting_at_runtime();
def _(x,y,z):
return frobnicate(mongo, x, y, z)
return _
foo = make_foo

I suspect you mean

foo = make_foo()

del make_foo # to make sure it is *never* called again ;

Now you only have foo with a hard-to-access private object and no
first_time checks when you call it.

Terry Jan Reedy

I don't think I'd ever want to use such an obscure technique in a
program. You might want to consider using functools.wraps to make sure
that the foo function looks right.

regards
Steve

 

[Alf P. Steinbach]

* kj:

When coding C I have often found static local variables useful for
doing once-only run-time initializations. For example:

int foo(int x, int y, int z) {

static int first_time = TRUE;
static Mongo *mongo;
if (first_time) {
mongo = heavy_lifting_at_runtime();
first_time = FALSE;
}

return frobnicate(mongo, x, y, z);
}

In this case, the static variable mongo is initialized only once
(at most).

What I like most about this is that it obviates the need for a
global variable to hold the persistent value (I avoid globals like
the plague, especially in Python). It also nicely encapsulates
the logic that determines whether initialization is required.

In C++ you just write

int foo( int x, int y, int z )
{
static Mongo* const mongo = heavy_lifting_at_runtime();
return frobnicate( mongo, x, y, z );
}

The best way I've found to achieve a similar effect in (procedural)
Python defines the function as a closure. For example, here's a
function that keeps track of (and prints out) how many times it
has been called:
... counter = [0]
... def _():
... counter[0] += 1
... print counter[0]
... return _
... 3

(Too bad that one can't stick the whole def inside parentheses and
call the function right there, like one can do with JavaScript.)

Off the cuff, Py3:

class Spam:
def __init__( self ):
self._counter = 0

def __call__( self ):
self._counter += 1
print( counter )

spam = Spam()
spam()
spam()
spam()


[snip]

I'm sure that there are many other ways to skin this cat, especially
if one starts definining fancy callable classes and whatnot.

As I see it it's the closure that's fancy, and the class that's simple and direct.

But
is there a better *simple* way to achieve C-style static locals in
Python that does not require a lot of extra machinery?

If you often need this functionality you might consider a general decorator that
supplies the function with a self argument, e.g. like this:


<example>
#Py3

class Object: pass

def static_initialization( init_func ):
def self_aware( f ):
def wrapped( *args, **kwargs ):
return f( f, *args, **kwargs )
init_func( f )
return wrapped
o = Object()
o.body = self_aware
return o


# Example usage:

@static_initialization
def spam( self ):
self.counter = 0

@spam.body
def spam( self ):
self.counter += 1
print( self.counter )

spam()
spam()
spam()
</example>


But as mentioned, a class is (at least IMHO) simpler and more direct.



Cheers & hth.,

- Alf (department of disingenious solutions)

 

[Paul Rubin]

When coding C I have often found static local variables useful for
doing once-only run-time initializations. For example:

int foo(int x, int y, int z) {
static int first_time = TRUE;
static Mongo *mongo;
if (first_time) { ...

Here are some cheesy ways.

1. Put an attribute onto the function name:

def foo(x, y, z):
if foo.first_time:
foo.mongo = heavy_lifting_at_runtime()
foo.first_time = False
...
foo.first_time = True

2. Use a mutable keyword parameter:

def foo(x, y, z, wrapped_mongo=[]):
if len(wrapped_mongo) == 0:
wrapped_mongo.append(heavy_lifting_at_runtime())
mongo = wrapped_mongo[0]
...

3. Streamline the first method a little:

def foo(x, y, z):
if len(foo.wrapped_mongo == 0):
foo.wrapped_mongo.append(heavy_lifting_at_runtime())
mongo = foo.wrapped_mongo[0]
...
foo.wrapped_mongo = []

All of these of course don't give as good encapsulation as one might
like.

 

[kj]

But the real problem is that the OP is insisting on using purely
procedural Python when the problem is screaming for an object-oriented
answer.

My initial reaction to this comment was something like "What? switch
from procedural to OO just to be able to do some one-time initialization
of function-private data???" But then, again, since Python allows
easy mixing of both programming styles, I suppose one could refactor this:

<procedural>
def spam(x, y, z):
try:
mongo = spam.mongo
except AttributeError:
mongo = spam.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)

ham = spam(3, 4, 5)
</procedural>

into this:

<OO>
class _Spam(object):
@classmethod
def _(cls, x, y, z):
try:
mongo = cls.mongo
except AttributeError:
mongo = cls.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)

ham = _Spam._(1, 2, 3)
</OO>


Is this really more natural or more readable? Hmmm.

In any case, the first solution does rely on the fact that functions
are objects, and therefore can have attributes, so even the
"procedural" version relies on Python's OO model.

Other responses advocated for global variables. I avoid them in
general, and doubly so in Python, because I find Python's shenanigans
with globals mystifying (this business of becoming silently local
if assigned to); it's one rare instance in which Python out-Perls
Perl. And yes, I know that the language includes ways to deal with
this (with the global keyword, etc.) but I find the whole scheme
is so much "cutting against the grain".

Thanks for all the replies. There are a lot of good ideas there.
I'm particular, I'm thankful for the pointers to PEP 3130 (initial
reaction: maybe I should learn Dutch) and to functools.wraps, and
for the code snippets.

~K

 

[Paul McGuire]

When coding C I have often found static local variables useful for
doing once-only run-time initializations.  For example:

Here is a decorator to make a function self-aware, giving it a "this"
variable that points to itself, which you could then initialize from
outside with static flags or values:

from functools import wraps

def self_aware(fn):
@wraps(fn)
def fn_(*args):
return fn(*args)
fn_.__globals__["this"] = fn_
return fn_

@self_aware
def foo():
this.counter += 1
print this.counter

foo.counter = 0

foo()
foo()
foo()


Prints:

1
2
3

-- Paul

 

[Mel]

In <[email protected]> Steve Holden


My initial reaction to this comment was something like "What? switch
from procedural to OO just to be able to do some one-time initialization
of function-private data???"

Yeah, actually. If the subject had been "Python-style object attributes in
C?" somebody might have suggested C static variables. An example I wrote
lately

volatile static int random_bit ()
{
static unsigned short lfsr = 0xACE1u; // seeded LFSR
// taps: 16 14 13 11; characteristic polynomial: x^16 + x^14 + x^13 +
x^11 + 1
lfsr = (lfsr >> 1) ^ (-(lfsr & 1u) & 0xB400u);
return lfsr & 1;
} // random_bit

(excuse is: this was written for cheap execution in an 8-bit processor.)

This does OK -- but fails the instant I decide that my program needs more
than one pseudo-random bit stream. Then I have the choice of writing
several different random_bit functions, or extending random_bit to take a
pointer to a seeded LFSR provided by the individual caller.

Refactoring the Python function to a Python class, as you mention later,
solves the static-access problem, but that solution is just as vulnerable to
the need-more-than-just-the-one problem as my C function.

Mel.

 

[Ethan Furman]

<OO>
class _Spam(object):
@classmethod
def _(cls, x, y, z):
try:
mongo = cls.mongo
except AttributeError:
mongo = cls.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)

ham = _Spam._(1, 2, 3)
</OO>


Is this really more natural or more readable? Hmmm.

For this type of situation, my preference would be:

class spam(object):
def __call__(self, x, y, z):
try:
mongo = self.mongo
except AttributeError:
mongo = self.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)
spam = spam()


No extra objects, out-of-place underscores, etc.

~Ethan~

 

[Patrick Maupin]

For this type of situation, my preference would be:

class spam(object):
     def __call__(self, x, y, z):
         try:
             mongo = self.mongo
         except AttributeError:
             mongo = self.mongo = heavy_lifting_at_runtime()
         return frobnicate(x, y, z, mongo)
spam = spam()

No extra objects, out-of-place underscores, etc.

~Ethan~

Well, I'm not a big fan of unnecessary try/except, so I would at least
change it to:

class spam(object):
def __getattr__(self, name):
if name != 'mongo':
raise AttributeError
self.mongo = heavy_lifting_at_runtime()
return self.mongo
def __call__(self, x, y, z):
return frobnicate(x, y, z, self.mongo)
spam = spam()

Regards,
Pat

 

[Steven D'Aprano]

Other responses advocated for global variables. I avoid them in
general,

In general this is wise, but remember that because Python globals are not
globally global, but local to a single module, they're safer than globals
in other languages. Still, it's better to avoid them when possible.

and doubly so in Python, because I find Python's shenanigans
with globals mystifying (this business of becoming silently local if
assigned to);

Globals don't become local when assigned to. You can shadow a global with
a local of the same name, but the global remains untouched:
.... myglobal = 0 # shadow the global with a new local
....42

I find this behaviour perfectly natural, and desirable: it means I can
assign to locals without worrying whether or not I'm about to stomp all
over a global and destroy it. The alternative behaviour would be
disastrous:
.... f = 'spam'
....Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'str' object is not callable

 

[Ethan Furman]

Well, I'm not a big fan of unnecessary try/except, so I would at least
change it to:

class spam(object):
def __getattr__(self, name):
if name != 'mongo':
raise AttributeError
self.mongo = heavy_lifting_at_runtime()
return self.mongo
def __call__(self, x, y, z):
return frobnicate(x, y, z, self.mongo)
spam = spam()

Regards,
Pat

Sounds like a personal preference issue, rather than a necessary /
unnecessary issue -- after all, if you call that function a thousand
times, only once is mongo not defined... clearly the exception.

~Ethan~

 

[Patrick Maupin]

Sounds like a personal preference issue, rather than a necessary /
unnecessary issue -- after all, if you call that function a thousand
times, only once is mongo not defined... clearly the exception.  

~Ethan~

Well, I think the whole discussion has basically been about personal
preference. OTOH, but if you call the function a few million times,
you might find the cost of try/except to be something that you would
rather not incur -- it might become a performance issue rather than a
personal choice issue. On the other OTHER hand, if you call the
function a few billion times, performance weighs more heavily in favor
of the closure approach rather than the object approach, since local
variable lookup is so much cheaper.

Regards,
Pat

 

[Ethan Furman]

Patrick Maupin wrote:

[snippage]

Well, I think the whole discussion has basically been about personal
preference. OTOH, but if you call the function a few million times,
you might find the cost of try/except to be something that you would
rather not incur -- it might become a performance issue rather than a
personal choice issue. On the other OTHER hand, if you call the
function a few billion times, performance weighs more heavily in favor
of the closure approach rather than the object approach, since local
variable lookup is so much cheaper.

Indeed. I was surprised to find your __getattr__ approach faster than
the try/except approach (about 20% on my machine). I'll have to think
about that for future situations like this.

My main point, though, was using __call__, and not some weird _ method.

~Ethan~

 

[Patrick Maupin]

My main point, though, was using __call__, and not some weird _ method.  

Yes, __call__ is good. In general, not naming things that don't need
to be named is good (but if you have too many of them to keep track
of, then, obviously, they need to be named, hence named tuples).

But I didn't need to address that, since you already did

 

[kj]

class Spam(object):
mongo = None
def __call__(self, x, y, z):
if self.mongo is None:
self.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, self.mongo)
spam = Spam()

ham = spam(1, 2, 3)

I really like this. Thanks.

That's natural and readable.

From reading this thread, and the "(a==b) ? 'Yes' : 'No'" one, the
inescapable conclusion is that "readability" (like beauty) is very
much in the eye of the beholder, or, in this case, in the eye of
Guido.

~K

 

[Steven D'Aprano]

Well, I think the whole discussion has basically been about personal
preference. OTOH, but if you call the function a few million times, you
might find the cost of try/except to be something that you would rather
not incur -- it might become a performance issue rather than a personal
choice issue.

The cost of a try...except is *very* low -- about the same as a pass
statement:
1.9227982449955801


Actually catching the exception, on the other hand, is quite expensive:
10.598482743564809


The heuristic I use is, if I expect the try block to raise an exception
more than about one time in ten, I change to an explicit test. In this
case, since the exception should only be raised once, and then never
again, I would use a try...except block.