K
k3xji
Hi all,
I have written a simple multithreaded profiler using decorators. Below
is how it works:
1) Iterate all modules in sys.modules and iterate each function/ class
methods inside them, means all of them including built-in methods.
2) Decorate the methods and functions to a global function.
3) This global function acquires a global lock and updates callCount,
timeElapsed values, the list of running threads and the function that
the thread is currently executing.
4) We have a result(..) function which acquires the global lock again
and iterates the above info with showing it in the console.
The problem arises when following happens:
- we call result(..) function from Thread 1
- result(..) acquires the global lock to read the information
- Then, somehow in the result __repr__() builtin function is being
called and as we also have decorated this function, this leads to a
deadlock.
Above took me 3 hours of debugging effort, and the worse part is
__repr__() is NOT being called in the same place always. It is
sometimes called in somestring processing in the top of the function
sometimes in the below.
So, I ended up avoiding __repr__() profiling in the profiler .
However, this situation puts me in a position that there may be other
built-in calls that may end up with a deadlock.
I can afford to avoid profiling of all built-in functions but how to
do that? Or is there any idea on this?
IMPORTANT: I am not sure, but this problem may be the real cause of
why we don't see realtime profile results on cProfile or Profile
libraries? All of these profilers are have to be stopped before
showing you the results, that way I can avoid the necessity of locking
() in the result(..) function.
Regards,
I have written a simple multithreaded profiler using decorators. Below
is how it works:
1) Iterate all modules in sys.modules and iterate each function/ class
methods inside them, means all of them including built-in methods.
2) Decorate the methods and functions to a global function.
3) This global function acquires a global lock and updates callCount,
timeElapsed values, the list of running threads and the function that
the thread is currently executing.
4) We have a result(..) function which acquires the global lock again
and iterates the above info with showing it in the console.
The problem arises when following happens:
- we call result(..) function from Thread 1
- result(..) acquires the global lock to read the information
- Then, somehow in the result __repr__() builtin function is being
called and as we also have decorated this function, this leads to a
deadlock.
Above took me 3 hours of debugging effort, and the worse part is
__repr__() is NOT being called in the same place always. It is
sometimes called in somestring processing in the top of the function
sometimes in the below.
So, I ended up avoiding __repr__() profiling in the profiler .
However, this situation puts me in a position that there may be other
built-in calls that may end up with a deadlock.
I can afford to avoid profiling of all built-in functions but how to
do that? Or is there any idea on this?
IMPORTANT: I am not sure, but this problem may be the real cause of
why we don't see realtime profile results on cProfile or Profile
libraries? All of these profilers are have to be stopped before
showing you the results, that way I can avoid the necessity of locking
() in the result(..) function.
Regards,