Logging thread with Queue and multiple threads to log messages

[scriptlearner]

I am trying to put up a queue (through a logging thread) so that all
worker threads can ask it to log messages. However, the problem I am
facing is that, well, the logging thread itself is running forever.
It does not know when it should exit. Any suggestion? None of the
worker threads knows for sure when the logging thread should exit
since they don't know what other worker threads are doing. I also try
to set a time limit for the logging thread, but it does not work.


# mylogging.py - used by everyone else that needs to log messages
import threading
import Queue
import time

# This is the queue object that's used to hold log events.
# LoggingThread knows about it, and the log() function below
# knows about it. No one else is allowed to see it.
_log_queue = Queue.Queue()

class LoggingThread(threading.Thread):
def __init__(self,logfile,duration):
threading.Thread.__init__(self)
self.logfile = logfile
self.duration = duration

def run(self):
f=open(self.logfile, 'w')
#it's probably not the right way to set an end time here
#since the logging thread should wait until all worker threads
are done
end = time.time() + self.duration + 10
while(end > time.time()):
message = _log_queue.get()
f.write(message + '\n') # (or whatever)

f.close()

# Kick off the logging thread.
LoggingThread("logs/myapp.log",20).start()

def log(msg):
_log_queue.put(msg)




class Worker(threading.Thread):
def __init__(self,no,duration):
threading.Thread.__init__(self)
self.no = no
self.duration = duration

def run(self):
end = time.time() + self.duration

while(end > time.time()):
log('Thread Object (%d)%d), Time:%s in seconds %d'%
(self.no,self.duration,time.ctime(),time.time()))
time.sleep(10)

print 'Done with worker (%d) at %s'%(self.no,time.ctime())



def main():
children = []
args = parseArgs()

for i in range(args.threads):
log('i=%d'%(i))
children.append(Worker(i,args.duration))
children.start()
time.sleep(0.1)

 

[Vinay Sajip]

I am trying to put up a queue (through aloggingthread) so that all
worker threads can ask it to log messages. However, the problem I am
facing is that, well, theloggingthread itself is running forever.
It does not know when it should exit. Any suggestion? None of the
worker threads knows for sure when theloggingthread should exit
since they don't know what other worker threads are doing. I also try
to set a time limit for theloggingthread, but it does not work.

# mylogging.py - used by everyone else that needs to log messages
import threading
import Queue
import time

# This is the queue object that's used to hold log events.
# LoggingThread knows about it, and the log() function below
# knows about it. No one else is allowed to see it.
_log_queue = Queue.Queue()

class LoggingThread(threading.Thread):
def __init__(self,logfile,duration):
threading.Thread.__init__(self)
self.logfile = logfile
self.duration = duration

def run(self):
f=open(self.logfile, 'w')
#it's probably not the right way to set an end time here
#since theloggingthread should wait until all worker threads
are done
end = time.time() + self.duration + 10
while(end > time.time()):
message = _log_queue.get()
f.write(message + '\n') # (or whatever)

f.close()

# Kick off theloggingthread.
LoggingThread("logs/myapp.log",20).start()

def log(msg):
_log_queue.put(msg)

class Worker(threading.Thread):
def __init__(self,no,duration):
threading.Thread.__init__(self)
self.no = no
self.duration = duration

def run(self):
end = time.time() + self.duration

while(end > time.time()):
log('Thread Object (%d)%d), Time:%s in seconds %d'%
(self.no,self.duration,time.ctime(),time.time()))
time.sleep(10)

print 'Done with worker (%d) at %s'%(self.no,time.ctime())

def main():
children = []
args = parseArgs()

for i in range(args.threads):
log('i=%d'%(i))
children.append(Worker(i,args.duration))
children.start()
time.sleep(0.1)

Take a look at this example test script to see how to use logging in a
multi-threaded environment:

http://dpaste.com/hold/89734/

Regards,

Vinay Sajip

 

[Floris Bruynooghe]

Hi

I am trying to put up a queue (through a logging thread) so that all
worker threads can ask it to log messages.

There is no need to do anything like this, the logging module is
thread safe and you can happily just create loggers in a thread and
use them, you can even use loggers that where created outside of the
thread. We use logging from threads all the time and it works
flawlessly.

As mentioned by Vinay in your other thread the problem you have is
that your main thread exists before the worker threads, which makes
atexit run the exithandlers and logging registers the
logging.shutdown() function which does flush all logging buffers and
closes all handles (i.e. closes the logfile). So as said before by
simply calling .join() on all the worker threads inside the main
thread your problem will be solved. You might get away with making
your threads daemonic, but I can't guarentee you won't run into race
conditions in that case. If you want to be really evil you could get
into muddling with atexit...

Regards
Floris

 

[scriptlearner]

Thank you guys, indeed, calling join() for each thread actually solved
the problem.
I misunderstood it and call join() right after start() so it didn't
work the way I wanted.

for i in range(args.threads):
children.join()