some profiler questions

[Mac]

I'm writing a bunch of algorithms in Python for solving a certain class
of problems, and I was interested in doing some comparisons between
them and would like to collect various high-level runtime stats, such
as the number of invocations of certain key subroutines. I have my own
code to do this. Then just earlier today it hit me that I'm really
just profiling, albeit in a more restricted way than the "profile"
module. This got me thinking and looking deeper at "profile.py", and
I've got some questions:

1) I'd still like to run my whole app (i.e., using main()), but I'd
like to limit the profiling to only the select few subroutines. That
is, say I have a set of such fns in mind, call it "key_fns", and I
would like to only profile # of invocations of these fns from key_fns,
as well as by whom they were called, and how much cumulative time was
spent in them. Is such lower-level control possible? The main reason
I want this is that I do not want to profile most of the low-level
routines, like vector addition, at least not yet... I don't want to
slow down code execution any more than is necessary, as the statistics
gathering should occur during "normal" runs (i.e., during normal
operation).

2) I've only just discovered that pstats has "print_callers()"! That's
very useful info I wasn't aware was available! What I'm really looking
for now is profiler output in the style generated by "gprof", the GNU
profiler, as I have found that format terribly useful (a section for
each fn, with the fn's callers and callees interwoven in each section).
Does anyone know of a utility which would format the Python profiling
info in that format, or something very similar? I haven't actually
seen any output from print_callers (can't find any samples on Net, and
my app is currently half-busted, mid-refactor), so if that's what it
precisely does, ignore this question.

3) assuming the above-mentioned fine control of (1) is not yet
possible, I will muddle on with my own "selective" profiling code; the
question I have then is, what is the cleanest way to override a class
instance's method at runtime? What my profiler is doing is overriding
the key fns/methods of an instance with a stat-gatherer-instrumented
version, which ends up calling the original method. I tried reading
profile.py and pstats.py for ideas, but they are a little too
complicated for me, for now; I doubt that's how they do their profiling
anyways. Any way to do this in an automated fashion, given that I have
the list of method names I want to instrument in a list variable?

 

[Dieter Maurer]

....
1) I'd still like to run my whole app (i.e., using main()), but I'd
like to limit the profiling to only the select few subroutines. That
is, say I have a set of such fns in mind, call it "key_fns", and I
would like to only profile # of invocations of these fns from key_fns,
as well as by whom they were called, and how much cumulative time was
spent in them. Is such lower-level control possible? The main reason
I want this is that I do not want to profile most of the low-level
routines, like vector addition, at least not yet... I don't want to
slow down code execution any more than is necessary, as the statistics
gathering should occur during "normal" runs (i.e., during normal
operation).

Please read the "How it works section".

You find an example how the profiler can be customized
in my "ZopeProfiler" product (the "HLProfiler.py").
It makes profiling slower (as it derives high level profiles
in addition to the low level ones) but might nevertheless
help you as an example.

2) I've only just discovered that pstats has "print_callers()"! That's
very useful info I wasn't aware was available! What I'm really looking
for now is profiler output in the style generated by "gprof", the GNU
profiler, as I have found that format terribly useful (a section for
each fn, with the fn's callers and callees interwoven in each section).
Does anyone know of a utility which would format the Python profiling
info in that format, or something very similar? I haven't actually
seen any output from print_callers (can't find any samples on Net, and
my app is currently half-busted, mid-refactor), so if that's what it
precisely does, ignore this question.

Thus, you look at the internal representation of "pstats"
and format it in the way you like...

3) assuming the above-mentioned fine control of (1) is not yet
possible, I will muddle on with my own "selective" profiling code; the
question I have then is, what is the cleanest way to override a class
instance's method at runtime?

class_.f = some_wrapper(class_.f)

What my profiler is doing is overriding
the key fns/methods of an instance with a stat-gatherer-instrumented
version, which ends up calling the original method. I tried reading
profile.py and pstats.py for ideas, but they are a little too
complicated for me, for now; I doubt that's how they do their profiling
anyways.

You should read the "How it works section".
You will find out that your doubt is justified...


Dieter