Question about object lifetime and access

[Asaf Las]

Hi community

i am beginner in Python and have possibly silly questions i could not figure out answers for.

Below is the test application working with uwsgi to test json-rpc.
------------------------------------------------------------------------
from multiprocessing import Process
from werkzeug.wrappers import Request, Response
from werkzeug.serving import run_simple

from jsonrpc import JSONRPCResponseManager, dispatcher

p = "module is loaded" <------ (3)
print(p)
print(id(p))

@Request.application
def application(request):
print("server started")

print(id(p))

# Dispatcher is dictionary {<method_name>: callable}
dispatcher["echo"] = lambda s: s <---- (1)
dispatcher["add"] = lambda a, b: a + b <---- (2)

print("request data ==> ", request.data)
response = JSONRPCResponseManager.handle(request.data, dispatcher)
return Response(response.json, mimetype='application/json')
------------------------------------------------------------------------

As program will grow new rpc method dispatchers will be added so there is idea to reduce initialization code at steps 1 and 2 by making them global objects created at module loading, like string p at step 3.

Multithreading will be enabled in uwsgi and 'p' will be used for read only.

Questions are:
- what is the lifetime for global object (p in this example).
- will the p always have value it got during module loading
- if new thread will be created will p be accessible to it
- if p is accessible to new thread will new thread initialize p value again?
- is it guaranteed to have valid p content (set to "module is loaded") whenever application() function is called.
- under what condition p is cleaned by gc.

The rationale behind these question is to avoid object creation within application() whose content is same and do not change between requests calling application() function and thus to reduce script response time.

Thanks in advance!

 

[Ned Batchelder]

Hi community

i am beginner in Python and have possibly silly questions i could not figure out answers for.

Below is the test application working with uwsgi to test json-rpc.
------------------------------------------------------------------------
from multiprocessing import Process
from werkzeug.wrappers import Request, Response
from werkzeug.serving import run_simple

from jsonrpc import JSONRPCResponseManager, dispatcher

p = "module is loaded" <------ (3)
print(p)
print(id(p))

@Request.application
def application(request):
print("server started")

print(id(p))

# Dispatcher is dictionary {<method_name>: callable}
dispatcher["echo"] = lambda s: s <---- (1)
dispatcher["add"] = lambda a, b: a + b <---- (2)

print("request data ==> ", request.data)
response = JSONRPCResponseManager.handle(request.data, dispatcher)
return Response(response.json, mimetype='application/json')
------------------------------------------------------------------------

As program will grow new rpc method dispatchers will be added so there is idea to reduce initialization code at steps 1 and 2 by making them global objects created at module loading, like string p at step 3.

Multithreading will be enabled in uwsgi and 'p' will be used for read only.

The important concepts to understand are names and values. All values in
Python work the same way: they live until no name refers to them. Also,
any name can be assigned to (rebound) after it has been defined.

This covers the details in more depth:
http://nedbatchelder.com/text/names.html

Questions are:
- what is the lifetime for global object (p in this example).

The name p is visible in this module for as long as the program is
running. The object you've assigned to p can be shorter-lived if p is
reassigned.

- will the p always have value it got during module loading

Depends if you reassign it.

- if new thread will be created will p be accessible to it

If the thread is running code defined in this module, yes, that code
will be able to access p in that thread.

- if p is accessible to new thread will new thread initialize p value again?

No, the module is only imported once, so the statements at the top level
of the module are only executed once.

- is it guaranteed to have valid p content (set to "module is loaded") whenever application() function is called.

Yes, unless you reassign p.

- under what condition p is cleaned by gc.

Names are not reclaimed by the garbage collector, values are. The value
assigned to p can be reclaimed if you reassign the name p, and nothing
else is referring to the value.

The rationale behind these question is to avoid object creation within application() whose content is same and do not change between requests calling application() function and thus to reduce script response time.

Thanks in advance!

Welcome.

 

[Chris Angelico]

Questions are:
- what is the lifetime for global object (p in this example).
- will the p always have value it got during module loading
- if new thread will be created will p be accessible to it
- if p is accessible to new thread will new thread initialize p value again?
- is it guaranteed to have valid p content (set to "module is loaded") whenever application() function is called.
- under what condition p is cleaned by gc.

Your global p is actually exactly the same as the things you imported.
In both cases, you have a module-level name bound to some object. So
long as that name references that object, the object won't be garbage
collected, and from anywhere in the module, you can reference that
name and you'll get that object. (Unless you have a local that shadows
it. I'll assume you're not doing that.)

How do you go about creating threads? Is it after initializing the
module? If so, they'll share the same p and the same object that it's
pointing to - nothing will be reinitialized.

As long as you don't change what's in p, it'll have the same value
([1] - handwave) whenever application() is called. That's a guarantee.

For your lambda functions, you could simply make them module-level
functions. You could then give them useful names, too. But decide
based on code readability rather than questions of performance. At
this stage, you have no idea what's going to be fast or slow - wait
till you have a program that's not fast enough, and then *profile it*
to find the slow bits. Unless you're doing that, you're completely
wasting your time trying to make something faster. Start with
readable, idiomatic code, code that you could come back to in six
months and be confident of understanding. Do whatever it takes to
ensure that, and let performance take care of itself. Nine times out
of ten, you won't even have a problem. In the past twelve months, I
can think of exactly *one* time when I needed to improve an app's
performance after I'd coded it the readable way, and there was just
one part of the code that needed to be tweaked. (And it was more of an
algorithmic change than anything else, so it didn't much hurt
readability.) Remember the two rules of code optimization:

1. Don't.
2. (For experts only) Don't yet.

Follow those and you'll save more time than you would gain by
micro-optimizing. And your time is worth more than the computer's.

ChrisA

[1] Technically p doesn't "have a value" at all. It's a name that's
bound to some object. You can rebind it to another object, you can
mutate the object it's bound to (except that you've bound it to a
string, which is immutable), or you can sever the connection (with
'del p'), but in simple terms, it's generally "near enough" to say
that p has a value.

 

[Asaf Las]

Thanks a lot for detailed answer!

i plan to assign object to name only when module loads, that means outside of function or class method. Then object will be accessed from functions only for read purpose.

I have read somewhere that global objects are referenced from module namespace will never have reference count down to 0 even if they are not referenced from functions or class methods. Is this true? Does it mean that global objects are destroyed when interpreter exits or thread where it runs is terminated?

 

[Asaf Las]

Thanks!

Questions are:

- what is the lifetime for global object (p in this example).

- will the p always have value it got during module loading

- if new thread will be created will p be accessible to it

- if p is accessible to new thread will new thread initialize p value again?

- is it guaranteed to have valid p content (set to "module is loaded") whenever application() function is called.

- under what condition p is cleaned by gc.

Your global p is actually exactly the same as the things you imported.

In both cases, you have a module-level name bound to some object. So

long as that name references that object, the object won't be garbage

collected, and from anywhere in the module, you can reference that

name and you'll get that object. (Unless you have a local that shadows

it. I'll assume you're not doing that.)



How do you go about creating threads? Is it after initializing the

module? If so, they'll share the same p and the same object that it's

pointing to - nothing will be reinitialized.



As long as you don't change what's in p, it'll have the same value

([1] - handwave) whenever application() is called. That's a guarantee.



For your lambda functions, you could simply make them module-level

functions. You could then give them useful names, too. But decide

based on code readability rather than questions of performance. At

this stage, you have no idea what's going to be fast or slow - wait

till you have a program that's not fast enough, and then *profile it*

to find the slow bits. Unless you're doing that, you're completely

wasting your time trying to make something faster. Start with

readable, idiomatic code, code that you could come back to in six

months and be confident of understanding. Do whatever it takes to

ensure that, and let performance take care of itself. Nine times out

of ten, you won't even have a problem. In the past twelve months, I

can think of exactly *one* time when I needed to improve an app's

performance after I'd coded it the readable way, and there was just

one part of the code that needed to be tweaked. (And it was more of an

algorithmic change than anything else, so it didn't much hurt

readability.) Remember the two rules of code optimization:



1. Don't.

2. (For experts only) Don't yet.



Follow those and you'll save more time than you would gain by

micro-optimizing. And your time is worth more than the computer's.



ChrisA



[1] Technically p doesn't "have a value" at all. It's a name that's

bound to some object. You can rebind it to another object, you can

mutate the object it's bound to (except that you've bound it to a

string, which is immutable), or you can sever the connection (with

'del p'), but in simple terms, it's generally "near enough" to say

that p has a value.

 

[Dave Angel]

Hi community
Welcome.


Multithreading will be enabled in uwsgi and 'p' will be used for read only.

Questions are:
- what is the lifetime for global object (p in this example).

The name will be visible in this module until the application
shuts down or till you use del.

- will the p always have value it got during module loading

The (str) object that you bind to it will survive till you del or
reassign p. Reassignment can happen with a simple assignment
statement or via an 'as' clause. The value of such a str object
will never change because it's an immutable type.

Convention is to use names that are all uppercase. And long,
descriptive names are preferred over one-letter names, especially
for long-lived ones. Don't worry, long names do not take longer.

- if new thread will be created will p be accessible to it

It is accessible to all threads in the same process.

It is also available to other modules via the import mechanism.
But watch out for circular imports, which frequently cause bugs.

- if p is accessible to new thread will new thread initialize p value again?

Module level code runs only once per process.

- is it guaranteed to have valid p content (set to "module is loaded") whenever application() function is called.

Except with circular imports.

- under what condition p is cleaned by gc.

Names are never garbage collected. See above for objects.

The rationale behind these question is to avoid object creation within application() whose content is same and do not change between requests calling application() function and thus to reduce script response time.

Highly unlikely to matter, and it might slow down a program
slightly rather than speed it up slightly. Get your program
readable so you have a chance of catching bugs, pick your
algorithms reasonably, and if it's not fast enough, measure,
don't guess.

Thanks in advance!

--
DaveA



----Android NewsGroup Reader----
http://www.piaohong.tk/newsgroup

 

[Steven D'Aprano]

I have read somewhere that global objects are referenced from module
namespace will never have reference count down to 0 even if they are not
referenced from functions or class methods. Is this true?

Correct. The global name is a reference, so the reference count will be
at least 1. In fact, referencing the name from a function or method
doesn't increase the ref count:

instance = 123.456789 # ref count of float is 1

def test():
print(instance) # This refers to the *name* "instance", not the float

So the test() function cannot keep the float alive. If you reassign
global instance, test() will see the new value, not the old, and
123.456789 is free to be garbage collected.

This sounds more complicated than it actually is. In practice it works
exactly as you expect global variables to work:

py> test()
123.456789
py> instance = 98765.4321
py> test()
98765.4321

Does it mean
that global objects are destroyed when interpreter exits or thread where
it runs is terminated?

Certainly not! Global objects are no different from any other object.
They are destroyed when their reference count falls to zero. In the case
of global objects, that is *often* not until the interpreter exits, but
it can be before hand.

So long as the object is in use, it will be kept. When it is no longer in
use, the garbage collector is free to destroy it. So long as *some*
object or name holds a reference to it, it is considered to be in use.

value = instance = 1.23456 # ref count 2
alist = [1, 2, 3, 4, 5, value] # ref count now 3
mydict = {"Key": alist} # ref count now 4
value = 42 # rebind a name, ref count of float now 3
mydict.clear() # ref count now 2
del instance # delete the name, ref count now 1
assert alist[5] == 1.23456
alist[5] = 0 # final reference gone, ref count is now 0

At this point the global object 1.23456 is free to be destroyed.

(Note: some Python implementations don't do reference counting, e.g.
Jython and IronPython use the Java and .Net garbage collectors
respectively. In their case, the same rule applies: where there are no
longer any references to an object, it will be garbage collected. The
only difference is in how soon that occurs: in CPython, it will be
immediate, in Jython or IronPython it will occur when the garbage
collector runs.)

 

[Asaf Las]

First of all many thanks to all for their detailed answers on subject.
I really appreciate it!

Correct. The global name is a reference, so the reference count will be

at least 1. In fact, referencing the name from a function or method
doesn't increase the ref count:

i have tried some tests, though accessing object from functions increase refcount but only temporary and only if object is used within function. i guess as soon as k is bound to string object latter's reference count will increase and function return results in unbound of k from object (according tooutput below).

What is interesting if module namespace can be held accountable for 1 reference count what are remaining 3 references counted on string object referenced by 'p'? (CPython 3.3.2, windows 7, run from within eclipse/PyDev and same output on centos 6.5 for python v3.3.3)

from sys import getrefcount

p = "test script"
print("refcnt before func() ", getrefcount(p))

def access_p1():
global p
print("refcnt inside func1()", getrefcount(p))

def access_p2():
global p
k = p
print("refcnt inside func2()", getrefcount(p))

access_p1()
access_p2()

print("refcnt after func() ", getrefcount(p))

--------------------------------------------------------------
Output:

refcnt before func() 4
refcnt inside func1() 4
refcnt inside func2() 5
refcnt after func() 4