Alex said:
What? When I add/del an item to a dict or list, this is not an atomic
thread-safe operation?
Exactly: there is no such guarantee in the Python language.
E.g.:
One thread does things like d['x']='y'
Another thread reads d['z'] or sets d['z']='w' or dels something.
If those operations are not atomic, then you'd have to use locks all the
time to not get RuntimeErrors and worse !?
If you want to be writing correct Python, yes. A preferred approach is
to simply avoid sharing objects among threads, except for objects
designed to be thread-safe (chiefly Queue.Queue).
I don't know the Python language (non?-)definition about this. But the
day, that will be a requirement in the Python implementation, I'll put
the last Python in a safe
( ..and rethink my bad opinion about Ruby )
For example hundreds of things like sre._cache and tenthousands of
common global variables are shared "thread safe" in the standard lib
whithout locks.
;-) They never will change any Python implementation and do the work to
put millions of lock.acquire()'s into the standard lib...
You're relying on an accident of a specific, particular implementation;
if any Python-coded standard library module does likewise, and I'm not
aware of any, that's somewhat different (since that module is PART of
the implementation, it may rely on all kinds of implementation details,
correctly if maybe not wisely). The situation is quite different for
C-coded modules in the CPython implementation, Java-coded ones in the
Jython one, C#-coded one in the IronPython one; each of these is subject
to specific constraints that it's perfectly wise to rely on (since each
implementation, as the language specification fully allows it to do,
adopts a different locking strategy at these low levels).
the other implementations whould also have a hard time to rewrite the
standard lib.
Python byte code is kind of "defined" and always interpreted similar and..
.... print d['a']
.... print d.keys()
.... d['b']=2
.... 2 0 LOAD_GLOBAL 0 (d)
3 LOAD_CONST 1 ('a')
6 BINARY_SUBSCR
7 PRINT_ITEM
8 PRINT_NEWLINE
3 9 LOAD_GLOBAL 0 (d)
12 LOAD_ATTR 1 (keys)
15 CALL_FUNCTION 0
18 PRINT_ITEM
19 PRINT_NEWLINE
4 20 LOAD_CONST 2 (2)
23 LOAD_GLOBAL 0 (d)
26 LOAD_CONST 3 ('b')
29 STORE_SUBSCR
30 LOAD_CONST 0 (None)
33 RETURN_VALUE
...things like LOAD_CONST / STORE_SUBSCR will be atomic as long as there
is a GIL or at least a GIL during execution of one byte code. No
threaded script language can reasonably afford to have thread-switching
open down to native microprocessor execution atoms.
It's not forced to by language specification, but neither is it
forbidden. It would be an absurd implementation strategy to waste time
and space to extract a dict's keys() first, as it would NOT buy
"atomicity" anyway -- what if some other thread deletes keys while
you're looping, or calls any nonatomic method on the very value you're
in the process of serializing?!
First I look at the practical requirement: I have the threads and the
fast object tree and the need to autosave in this process. And don't
want to lock everywhere any dict-access to the tree just because of the
autosave (for app-reasons I need only a handful of locks so far). And I
don't want to use a slow & overkill method like ZODB.
One clean method (A) in order to stay practical would be possible if a
global lock for Python treading would be offered by the thread-lib as
described in <
[email protected]>
The other method (B), (that I believed to use so far) is to have a
little discipline, but no need for massive trivial locking:
* add complex objects to the hot tree only if the objects are
complete/consistent.
* don't do nasty things on removed complex objects; just forget them as
usual
* most attribute changes are already app-atomic; the overall majority of
operations is read-only access anyway - both regarding the number of
executions and the amount of source code (that is 99% the reason for
using this "disciplined" method in threaded apps)
* ..in case of non-atomic attribute changes, create a new appropriate
top object with compound changes (same logic as in fact you have, when
you use ZODB) and replace the whole top object in one step. Or use locks
in rare cases.
Now a practical "autosave" method in my app could fit to such practical
disciplined method ( or not :-( ).
And there is reason, why the Python standard lib should at least offer
one "disciplined" method to sample-copy such object tree, despite of
threads. There is no need get an "arbitray-Python-atomic" copy of the
whole tree (which would require (A)). The app itself respects the
discipline for "app-atomicity" if I use things in this way.
Its a true practical standard requirement: Just a method with no
"RuntimeError".
Thus it is useful to have a deepcopy and/or cPickle.dump which does not
break.
Is current dict.copy() exposed to this Runtime Error? AFAIK: not.
In that case copy.copy() respects "the disciplin", but deepcopy & dump
not, because of use of .iteritems(). And most probably it worked before
py2.2 as I got no such errors with that app in those times.
The argument for the cost of dict.keys() - I guess it does nearly not
even pay off in terms of speed. It could better stay disciplined in
those few critical locations like in dump and deepcopy which are
supposed to double much more and expensive things anyway. Its a weak
O(1) issue.
So far, its simple: I'd have to write my own dump or deepcopy after
Python changes, because of new "fun" with RuntimeErrors !
At least an alternate Runtime-Save version of deepcopy/dump would fit
into a practical Python, if the defaults cannot be keept flat.
In some Python implementations, a C-coded module may count on some
atomicity as long as it doesn't explicitly allow other threads nor ever
call back into ANY python-coded part, but obviously cpickle cannot avoid
doing that, so even in those implementations it will never be atomic.
The degree of atomicity defines the degree of usability for programming
ideas. And that should not be lowered as it makes thread programming in
VHL script languages so practical, when you just can do most things like
d['a']='b' without thread worries.
There is no theoretical treshold in a practical (=connected) world:
Each OS and ASM/C level relies on CPU-time- & memory-atoms. In fact,
otherwise, without such atoms, digital "platonic" computers could not
interfer with the "threaded" reality at all. That is a
_natural_requirement_ of "ideas" to _definitely_ be "atomic".
The CPU bit x clocktick is defined now by ~1.8eV/kT in the real world =>
no computing error within 10^20 years.
( Only something like biological neuro-brains or analogical computers
can offer more of "free threading". But believe me, even the neuro- and
quantum-threading respects the Plank-h-quantum: the time-resolution of
"thread-interaction" is limited by energy restrictions and light speed )
If Python really has not yet defined its time-atoms, that should go on
the To-Do list ASAP. At worst, its atoms are that of ASM - I hope its
better...
Whats does copy/deepcopy/[:] ?
Roughly the same situation.
If as you indicate you want to stick with a Python-like language but do
not want to change your style to make it correct Python, you could
perhaps fork the implementation into an "AtomicPython" in which you
somehow fix all nonatomicities (not sure how that would even be possible
if pickling, deepcopying or anything else ever needs to fork into Python
coded parts, but perhaps you might make the GIL into a reentrant lock
and somehow hack it to work, with some constraints). Or perhaps you
might be able to write an extension containing atomicset, atomicget,
atomicpickle, and other operations you feel you need to be atomic (again
barring the difficulties due to possible callbacks into Python) and use
those instead of bare Python primitives.
Each Python _is_ an AtomicPython. And that sets its VHL value. Maybe, it
just doesn't know itself sofar?
I asked maybe for a _big_ practical atom, and now have to make it
myself, because Python became smaller
The practical problem is: You must rely on and know deeply the real
atoms of Python and the lib in order to know how much atoms/locks you
have to ensure on your own.
At a certain level, Python would loose its value. This
disciplined-threading/dump/deepcopy issue here is of course somewhat
discussable on the upper region of VHL. (I certainly like that direction)
But, what you sketch here on the far other side about
take-care-about-all-dicts-in-future-Python-threads is the door to the
underworld and is certainly a plan for a Non-Python.
"take care for everything" is in fact an excuse for heading towards low
level in the future. (Ruby for example has in many ways more "greed"
towards the low level and to "aritrary fun" - too much for me - and
without offering more real power)
Python evolution was mostly ok and careful. But it should be taken care
that not too much negative ghosts break VHL atoms:
One such ghost went into _deepcopy_dict/dump (for little LL speed
issues). Other ghosts broke rexec ( a nice "Atom" which i miss very much
now ), ...
A function for locking python threading globally in the thread module
would be kind of a practical "hammer" to maintain VHL code in deliberate
key cases. (Similar as you do 'cli' in device driver code, but for a
python process.)
For example: "I know as programmer, if I can afford to lock app threads
during a small multi-attribute change or even during .deepcopy etc.; The
cost for this is less than the need to spread individual locks massively
over the whole treaded app."
For example socket.getaddrinfo does this unforeseeable for all
app-thread for minutes (!) in bad cases internally on OS-level anyway.
Python should name its Atoms.
Robert
