BUG trace realtime

[malv]

How would you approach the following?
In a multithreaded realtime data acquisition system (all python v2.4),
after hours of running without a snag, without warning python hangs at
once without leaving any error message or error traceback at all. After
the incident, the OS itself (linux) appears to remain fully functional.
Thank you!
malv

 

[Peter Hansen]

How would you approach the following?
In a multithreaded realtime data acquisition system (all python v2.4),
after hours of running without a snag, without warning python hangs at
once without leaving any error message or error traceback at all. After
the incident, the OS itself (linux) appears to remain fully functional.

1. Define "hangs" more precisely. It's rare that an app can hang
without any behaviour that can tell you something more about it. For
example, can you Ctrl-C out of it? Does it have sockets open still? Is
it using any CPU time? etc... While you're at it, define "realtime"
too, since many people use it incorrectly and it might mean something
different to you than it does to everyone else reading it...

2. Use logging. "import logging" and proceed from there... without
logging, you have no idea where the problem occurred, and are going to
spend lots of time guessing where the cause is.

3. Make sure you are using Queues exclusively for inter-thread
communication.

4. Consider whether your external source of data (assuming there is one)
is the cause. How are you interfacing to it? Serial port? ctypes?
Something SWIGged for Python? A true "hang" is much more likely in an
external package than in Python code.

5. Tell us more about the platform. Which Linux (including version),
does it have a GUI, what's the basic architecture of the app as it
relates to the threading and data acquisition stuff, etc. (Of course,
the problem might not come from threads or data acquisition, but until
we learn more who could say?)

-Peter

 

[malv]

Hi Peter,
Thank you for your extensive reply.
In the meantime, I solved the problem. I had started out to trace the
path of the randomly arriving external data through the different
threaded processing stages. After one of those 'hangs', with some luck,
I was able to spot the cause of the trouble: a piece of old code that
wasn't supposed to be out there! Really dumb.
FYI, to my surprise, Python really quit. No more CPU time, no message,
nothing. I can't recall having seen this before. You are quite right
about queues. I use these extensively and carefully. This kind of
enabled me to trace the path of the externally triggered events without
too much of a hassle.
Thanks again,
malv

 

[Peter Hansen]

After one of those 'hangs', with some luck,
I was able to spot the cause of the trouble: a piece of old code that
wasn't supposed to be out there! Really dumb.
FYI, to my surprise, Python really quit. No more CPU time, no message,
nothing. I can't recall having seen this before.

If you post the relevant "old code" it's very likely someone could point
out exactly why it failed, and how it wasn't *Python* that "quit", but
merely something in your code that was blocking. (In other words,
Python probably did exactly what you told it to do.)

In general, Python does not "quit". It's very robust. When you get a
failure like that, always suspect your own code structure.

-Peter

 

[malv]

What happened in fact was that a graphics plotting function for
monitoring got called twice frorm a different thread, due to an 'old'
instruction inadvertently left. Due to the quasi random nature of the
incoming data streams, this happened rather very sparingly. As the
plotting itself involves quite a bit of code, there are probably
several if not many reasons why problems could arise. It was never my
intention to reenter this function and make it thread safe. Problem
solved.

I was indeed a surprise for me to see Python quit this way.
Thanks again for your commentary.
malv