thread-local data

[Emanuele D'Arrigo]

Hi everybody,

Assuming a snippet such as:

threadLocalData = threading.local()
threadLocalData.myDictionary = self.myDictionary

is it correct to say that threadLocalData.myDictionary is NOT a thread-
local -copy- of self.myDictionary but it's actually pointing to the
same object?

If that's the case, and assuming I want to iterate over the dictionary
without it changing under my nose while I'm in the loop, would it be
better to encase the whole loop in lock-protected section or would it
be better to make a copy of the dictionary first and then iterate over
that one? Given that in this particular thread I do not want to modify
the dictionary, conceptually a copy would work. But would making
thread-local copy be just as slow as making the whole loop thread
safe?

Thanks for your help!

Manu

 

[MRAB]

> Hi everybody,
>
> Assuming a snippet such as:
>
> threadLocalData = threading.local()
> threadLocalData.myDictionary = self.myDictionary
>
> is it correct to say that threadLocalData.myDictionary is NOT a thread-
> local -copy- of self.myDictionary but it's actually pointing to the
> same object?
>
> If that's the case, and assuming I want to iterate over the dictionary
> without it changing under my nose while I'm in the loop, would it be
> better to encase the whole loop in lock-protected section or would it
> be better to make a copy of the dictionary first and then iterate over
> that one? Given that in this particular thread I do not want to modify
> the dictionary, conceptually a copy would work. But would making
> thread-local copy be just as slow as making the whole loop thread
> safe?
>

It depends on how long it takes to iterate over the dict compared with
how long it takes to copy the dict. Other threads will/should be denied
access during the iteration/copying and therefore block, reducing
throughput.

I'd probably use the snapshot approach (take a copy).

 

[Diez B. Roggisch]

Hi everybody,

Assuming a snippet such as:

threadLocalData = threading.local()
threadLocalData.myDictionary = self.myDictionary

is it correct to say that threadLocalData.myDictionary is NOT a thread-
local -copy- of self.myDictionary but it's actually pointing to the
same object?

Yes, it's pointing to the same object, and it thus makes the whole
purpose of threadlocal moot. Use a global.

> If that's the case, and assuming I want to iterate over the dictionary
> without it changing under my nose while I'm in the loop, would it be
> better to encase the whole loop in lock-protected section or would it
> be better to make a copy of the dictionary first and then iterate over
> that one? Given that in this particular thread I do not want to modify
> the dictionary, conceptually a copy would work. But would making
> thread-local copy be just as slow as making the whole loop thread
> safe?

As MRAB pointed out - it depends. My gut feeling on this is that the
copy-approach is fastest. Or even faster, if you can cope with keys
getting lost while processing them:


for key in self.myDictionary.keys():
try:
value = self.myDictionary[key]
except KeyError:
pass


Diez

 

[Diez B. Roggisch]

Yes, it's pointing to the same object, and it thus makes the whole
purpose of threadlocal moot. Use a global.

Scratch the "use a global" - that's of course nonsense. Use
self.myDictionary directly, or wherever the dict comes from.

Diez

 

[Emanuele D'Arrigo]

Thank you both MRAB and Diez.

I think I'll stick to making copies inside a thread-protected section
unless I need to speed up things, at which point I might go for the
key exception path.

Thank you again!

Manu