help with threads

[Michael Mossey]

Hello,

I have a simple application that needs one thread to manage networking
in addition to the main "thread" that does the main job. It's not
working right. I know hardly anything about threads, so I was hoping
someone could point me in the right direction to research this.

Basically, I have a program that does some computational work, and
also conveys its status to a monitor program elsewhere on the network
via sockets. I wanted to use a thread to manage the networking so that
the main program can run without regard to networking (i.e. they would
be asynchronous). So the network thread loops and calls select.

My problem is that in some cases, the network thread appears to stop,
while the main thread is doing a long computation.

I'm hoping someone can give me a general idea what to read about. For
example, under what conditions does a thread stop running? Can other
threads "take priority"? Are there certain operations that block other
threads (such as disk access)?

Thanks,
Mike

 

[Loïc Domaigné]

Hallo Michael,

I have a simple application that needs one thread to manage networking
in addition to the main "thread" that does the main job. It's not
working right. I know hardly anything about threads, so I was hoping
someone could point me in the right direction to research this.

Basically, I have a program that does some computational work, and
also conveys its status to a monitor program elsewhere on the network
via sockets. I wanted to use a thread to manage the networking so that
the main program can run without regard to networking (i.e. they would
be asynchronous). So the network thread loops and calls select.

My problem is that in some cases, the network thread appears to stop,
while the main thread is doing a long computation.

I'm hoping someone can give me a general idea what to read about. For
example, under what conditions does a thread stop running? Can other
threads "take priority"? Are there certain operations that block other
threads (such as disk access)?

May I suggest:
http://www.dabeaz.com/python/GIL.pdf

HTH,
Loïc

 

[Michael Mossey]

Ah yes, that explains it. Some of these long computations are done in
pure C, so I'm sure the GIL is not being released.
Thanks.

 

[Hendrik van Rooyen]

Hello,

My problem is that in some cases, the network thread appears to stop,
while the main thread is doing a long computation.

Is this computation done in pure python or are you calling some
underlying thing in C?

I would be surprised if a pure python computation thread were to hog
the cpu - have you been able to figure out what the computation thread is
actually doing when the hogging occurs?

I'm hoping someone can give me a general idea what to read about. For
example, under what conditions does a thread stop running? Can other
threads "take priority"? Are there certain operations that block other
threads (such as disk access)?

AFAIK python threads are all at the same level of priority, and the running
thread is interrupted every N python bytecode instructions, so what you are
experiencing should not happen.

Try to figure out, if you can, where the computation thread is spending
its time.

- Hendrik

 

[Jean-Michel Pichavant]

Hello,

I have a simple application that needs one thread to manage networking
in addition to the main "thread" that does the main job. It's not
working right. I know hardly anything about threads, so I was hoping
someone could point me in the right direction to research this.

Basically, I have a program that does some computational work, and
also conveys its status to a monitor program elsewhere on the network
via sockets. I wanted to use a thread to manage the networking so that
the main program can run without regard to networking (i.e. they would
be asynchronous). So the network thread loops and calls select.

My problem is that in some cases, the network thread appears to stop,
while the main thread is doing a long computation.

I'm hoping someone can give me a general idea what to read about. For
example, under what conditions does a thread stop running? Can other
threads "take priority"? Are there certain operations that block other
threads (such as disk access)?

Thanks,
Mike

I may be a little out of subject, anyway *if* you are writing also the
server side program, you should really take a look at
http://docs.python.org/library/xmlrpclib.html
or
http://pyro.sourceforge.net/

.... and forget about network coding )

To help you for your initial question, exceptions in thread are not
propagated to the main thread, so when an unhanded exception occurs in
your thread, it will just stop. Maybe this is what is happening.

A quick workaround is to embed all you threaded code in a try except
clause and log the exception before re-raising it. You should only do
that for debugging purpose.

JM

 

[Piet van Oostrum]

MM> Ah yes, that explains it. Some of these long computations are done in
MM> pure C, so I'm sure the GIL is not being released.

Is that C code under your own control? Or at least the glue from Python
to C? In that case, and if the C code is not manipulating Python objects
or something in the Python interpreter, it could be changed to release
the GIL during the computation. That's also how Numpy does it, IIRC.

 

[Michael Mossey]

Is that C code under your own control? Or at least the glue from Python
to C? In that case, and if the C code is not manipulating Python objects
or something in the Python interpreter, it could be changed to release
the GIL during the computation. That's also how Numpy does it, IIRC.

I don't have any control over this code, and it's easier to solve my
problem in other ways. I just put a sleep() call between calls to the
C library, and that gives the network thread enough responsiveness for
my particular task. I am grateful, anyway, to understand why this kind
of thing happens.

 

[Aahz]

I have a simple application that needs one thread to manage networking
in addition to the main "thread" that does the main job. It's not
working right. I know hardly anything about threads, so I was hoping
someone could point me in the right direction to research this.

You might also consider the multiprocessing module.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"...string iteration isn't about treating strings as sequences of strings,
it's about treating strings as sequences of characters. The fact that
characters are also strings is the reason we have problems, but characters
are strings for other good reasons." --Aahz