Tracing the execution of scripts?

[Michael B. Trausch]

Alright, I seem to be at a loss for what I am looking for, and I am not
even really all that sure if it is possible or not. I found the 'pdb'
debugger, but I was wondering if there was something that would trace or
log the order of line execution for a multi-module Python program. I am
having a little bit of a problem tracking down a problem that I
mentioned earlier
(http://groups.google.com/group/comp.lang.python/msg/9c759fc888b365be),
and I am hoping that maybe seeing how the statements are executed will
shed some light on the entire thing for me.

The debugger isn't working, though, because I have a timer set up to
fire off every 20ms to check for incoming network traffic from the
server, and that timer firing off makes stepping through with the
debugger seemingly impossible to get any useful data.

Basically, is there something that will log every line of Python code
executed, in its order of execution, to a text file so that I can see
what is (or isn't) happening that I am expecting? I know that the
server is sending and receiving properly, because I can connect to it
with a regular telnet client and it works -- but at the same time,
everything I can think of to check my program's functionality checks out
just fine (e.g., it reports that it is successfully sending what I am
typing to the server, and it says that the server is not sending
anything back to be read from the socket).

If it makes any difference, I am using wxWidgets' Python bindings for
the UI code (and the timer), and using Python's socket library for the
network stuff.

-- Mike

 

[Ben Finney]

1) Write unit tests for your code. Keep writing unit tests until
you have some that _don't pass_. Then fix your code so that they
do. When you do further development, write the tests first, then
implement the code that makes them pass.

Hear hear.

Be advised, though, that attempting to apply unit tests to code that
wasn't designed with testing in mind may very quickly reveal a poor
design. [0] If you can't easily test pieces of the code independently,
you probably haven't written those pieces to be loosely coupled and
well-defined.

The moral? Writing unit tests *along with* the functional code will
result in a design that is loosely coupled, and probably
better-defined. Also, hopefully, easy to test


[0] I have no idea whether this is the case for the OP. It's a very
common symptom that arises from people who are first advised to
introduce tests to their code, though.

 

[Stephan Kuhagen]

Basically, is there something that will log every line of Python code
executed, in its order of execution, to a text file so that I can see
what is (or isn't) happening that I am expecting?

Python itself can do this for you. A __VERY__ simple approach:

|def myCallTrace (frame, event, arg):
| tracefile.write('localTrace\n frame: '+str(frame)+'\n event:
|'+str(event)+'\n arg: '+str(arg)+'\n')
| tracefile.write(' '+str(inspect.getframeinfo(frame))+'\n')
|
|def myTrace (frame, event, arg):
| tracefile.write('globalTrace\n frame: '+str(frame)+'\n event:
|'+str(event)+'\n arg: '+str(arg)+'\n')
| if event == 'call':
| return myCallTrace
|
|import inspect
|import sys
|
|sys.settrace(myTrace)
|tracefile = file('trace.txt', 'w')

Insert this in you program and you get a trace of every line of Python-Code
executed in the file trace.txt. You must read the documentation of the
module inspect and of sys.settrace() to understand, what happens and what
it means. Additionally, if it should work with threads, you must take care
that every thread gets its own output file. But as a first step to get a
trace of execution, this should do it.

HTH, Regards
Stephan

 

[R. Bernstein]

pydb (http://bashdb.sf.net/pydb) has a both the ability to trace lines
as they are executed as well as an --output option to have this sent
to a file rather than stdout. If your program has threads it would be
good to use the --threading option. (The best threading support is if
your program uses the threading module for threads rather than the
lower-level thread module.)

A demo of the debugger including showing line tracing and output to a file is
here:
http://showmedo.com/videos/video?name=pythonBernsteinPydbIntro&fromSeriesID=28

 

[Michael B. Trausch]

1) Write unit tests for your code. Keep writing unit tests until
you have some that _don't pass_. Then fix your code so that they
do. When you do further development, write the tests first, then
implement the code that makes them pass.

Hear hear.

Be advised, though, that attempting to apply unit tests to code that
wasn't designed with testing in mind may very quickly reveal a poor
design. [0] If you can't easily test pieces of the code independently,
you probably haven't written those pieces to be loosely coupled and
well-defined.

I will whole-heartedly admit that my code is probably poorly designed.
*shrug* Though, I *do* try to write everything in such a way as to be
able to easily re-use it later. When I wrote little things just to help
me sysadmin, I learned that was really key to making my life easier.

Besides, I am lazy... reuse certainly serves my needs there!

The moral? Writing unit tests *along with* the functional code will
result in a design that is loosely coupled, and probably
better-defined. Also, hopefully, easy to test

I think I have more to learn on the concept of unit-testing. I have
never seen a group push the idea so hard. I have read about unit
testing before, and even written tests (in other languages) to test
modules of library code that I put together once, but that was about it.
I need to, I think, get more "into" the concept, though. It isn't
something that I am able to just whip out and go "Hey, a widget unit
test! And it works!" probably because of my own lack of practice.

[0] I have no idea whether this is the case for the OP. It's a very
common symptom that arises from people who are first advised to
introduce tests to their code, though.

It is very likely that it is the case. I am, by all means, what I could
consider to be a novice programmer when it comes to anything outside of
my little world of PHP. I can write web applications in PHP until the
cows come home with little to no problem, and I can play with SQL the
same way. But, I seem to have problems when I step out of that little
box, and so, that tells me that I need to work harder at it.

-- Mike

 

[Michael B. Trausch]

Basically, is there something that will log every line of Python code
executed, in its order of execution, to a text file so that I can see
what is (or isn't) happening that I am expecting?

Python itself can do this for you. A __VERY__ simple approach:
[snip]

Insert this in you program and you get a trace of every line of Python-Code
executed in the file trace.txt. You must read the documentation of the
module inspect and of sys.settrace() to understand, what happens and what
it means. Additionally, if it should work with threads, you must take care
that every thread gets its own output file. But as a first step to get a
trace of execution, this should do it.

I will need to keep this around to look at a little later. It looks
like it would be something particularly useful when line-tracing by
itself fails. I am constantly finding myself amazed at Python's
capabilities.

-- Mike

 

[Fulvio]

***********************
Your mail has been scanned by InterScan MSS.
***********************

pydb (http://bashdb.sf.net/pydb) has a both the ability to trace lines

I faced several time that pydb stuck without sign of errors. In the other hand
Pdb doesn't appear that problem.
Mostly pydb freeze on long loops.
It might be some problem on my setup, I'll check it up...

F

 

[Larry Bates]

I just thought I would put my 2 cents in on this issue. Others
have suggested that unit tests are an excellent way of debugging
your code and I agree. I also find that writing code from the
outset using a logging class (there is one in the standard
library) that allows you to create log files of information as
you run your application is a good idea. Using increasingly
detailed logs (I use a debug mode of 1,2,3,4 for more detail)
that dump where you are and intermediate variable values works
EXTREMELY well for me. I leave this code in the application
so that I can have customers (or myself) run the application
in debug mode should I have a hard to find problem. This is
especially true for long running or lights-out batch apps
that have no UI making debugging even more difficult. I find
that the overhead of testing if I'm in debug mode and logging
results is almost non-existent to the overall execution speed
of my scripts, but then I don't have very many really speed
sensitive scripts so your mileage might vary.

Hope the information helps.

-Larry

 

[fumanchu]

Python itself can do this for you. A __VERY__ simple approach:
...
Additionally, if it should work with threads, you must take care
that every thread gets its own output file.

Or be differentiated *somehow*. My pyconquer module tabs each thread's
activity into swimlanes, so you can even see when each thread starts,
obtains/releases the GIL, and stops. See the longer example here:
http://projects.amor.org/misc/wiki/PyConquer


Robert Brewer
System Architect
Amor Ministries
(e-mail address removed)

 

[R. Bernstein]

***********************
Your mail has been scanned by InterScan MSS.
***********************

Delighted to know that.

I faced several time that pydb stuck without sign of errors. In the other hand
Pdb doesn't appear that problem.
Mostly pydb freeze on long loops.

In version 1.19 (released today), I extended signal handling so that
you can send the program a signal and it will print a stack trace of
where it is and continue. But even before this release, you could
arrange to enter the debugger by sending it a signal. It's possible
something like this might be used help track down the problem in
either pydb or another Python program that seems to be not
responding. On the other hand, use at your own risk - I don't
guarentee this will work for you.

And.. before you ask for more details, I'll repeat what someone else
posted in response to another of your questions:

I guess you've been told to read this here, but just in case it
wasn't, or you didn't bother to read it:

http://catb.org/esr/faqs/smart-questions.html

It might be some problem on my setup, I'll check it up...

Given your other posts, that's quite possible. If it's not, submit a
bug report. (Posting to c.l.r isn't the same as submitting a bug
report). Thanks.

 

[Ben Finney]

1) Write unit tests for your code. Keep writing unit tests until
you have some that _don't pass_. Then fix your code so that they
do. When you do further development, write the tests first, then
implement the code that makes them pass.

Hear hear.

Be advised, though, that attempting to apply unit tests to code
that wasn't designed with testing in mind may very quickly reveal
a poor design. [0] If you can't easily test pieces of the code
independently, you probably haven't written those pieces to be
loosely coupled and well-defined.

I will whole-heartedly admit that my code is probably poorly
designed. *shrug* Though, I *do* try to write everything in such a
way as to be able to easily re-use it later.

The trick is to write pieces that are small and loosely-coupled, so
that small pieces of code (not just entire modules) can be re-used
without being re-written.

Test-driven development helps with this, because a small,
loosely-coupled piece of code is easy to test, so you will gravitate
toward that style.

Besides, I am lazy... reuse certainly serves my needs there!

The programming virtues espoused by Larry Wall continue to hold true

I think I have more to learn on the concept of unit-testing.

I have never seen a group push the idea so hard.

Writing each test before the functional code is not a new idea, but
Python makes it trivially easy to do, so the excuses for not doing it
are much weaker and easier to ridicule

I have read about unit testing before, and even written tests (in
other languages) to test modules of library code that I put together
once, but that was about it.

Here's an article that goes through the concepts, and happens to use
Python.

I need to, I think, get more "into" the concept, though. It isn't
something that I am able to just whip out and go "Hey, a widget unit
test! And it works!" probably because of my own lack of practice.

One thing to note is that UI widgets shouldn't be the main thing
you're testing. You need to write code that can be easily tested
independent of everything else, because that way you can know that it
works and move on to other things quickly, building on it as a
foundation.

UI widgets should be as dumb as possible, because they don't make a
whole lot of sense by themselves except in the dumbest sense. Every
part of your program that actually does something interesting with
data or resources should be utterly independent of any UI
widgets. *That* is the code that you need to get right, so that should
be riddled with tests.

Your widgets should be plugging into code that dumbly passes the data
to the well-tested, UI-independent code that does work with that data;
and the widgest should be getting their data from such UI-independent
outputs. This is one aspect of "loose coupling": the code talks along
a very narrow, well-defined, simple interface to other parts of the
code.

UI code especially is prone to change entirely independent of the
desired functionality. for this reason, it's vital that you *can*
change the functionality and UI code each without needing to change
the other. The same principle applies along any interface between
parts of code that can be expected to change independent of each
other: loosely couple them together, with narrow well-defined
interfaces.

You may have heard of the "Model, View, Controller" pattern. This is a
design pattern that emphasises a loose coupling between the UI (the
"controller") and the data model, by abstracting each to the other via
a simple "view" interface module. That way, each of the Controller and
the Model can be oblivious to the most common changes in the other,
even when those changes make for complex code, because the View they
interact with stays simple and doesn't need to change.

<URL:http://en.wikipedia.org/wiki/Model-view-controller>

The reason test-driven development is so useful here is that, in order
to write a test for something *before* it exists, you must think about
how that piece of code will interact with the rest of the code. You
must, in other words, design its interface before writing its
code. This is excellent, because it means you're focussing on exactly
the thing that will help to keep your overall design loosely coupled.

Then, after you have a test that interacts with this interface, all
you need to do is write the simplest code to satisfy that
interface. The temptation to write clever, complex code is reduced,
because you're now interested in satisfying the external test, not in
treating this piece of code as an awesome complex machine ignoring the
rest of the code.

Good luck with getting test infected

<URL:http://c2.com/cgi/wiki?TestInfected>