Logging from a multiprocess application

[Joseph L. Casale]

I have a module that has one operation that benefits greatly from being multiprocessed.
Its a console based module and as such I have a stream handler and filter associated to
the console, obviously the mp based instances need special handling, so I have been
experimenting with a socket server in a thread in order for the remaining application to
carry on.

How have others tackled this problem? The portion of the code made to use multiprocessing
can not be switched to threading as it performs worse than simply serializing each task.

Thanks,
jlc

 

[Mark Betz]

I have a module that has one operation that benefits greatly from being multiprocessed.
Its a console based module and as such I have a stream handler and filterassociated to
the console, obviously the mp based instances need special handling, so Ihave been
experimenting with a socket server in a thread in order for the remainingapplication to
carry on.

How have others tackled this problem? The portion of the code made to usemultiprocessing
can not be switched to threading as it performs worse than simply serializing each task.

Thanks,
jlc

Maybe check out logstash (http://logstash.net/). It's part of elastic search now. Using the python-logstash module (https://pypi.python.org/pypi/python-logstash) you can either log directly to the logstash daemon using the provided event formatter and http, or you can do as I did and set up a redis server to host a list. Logstash has an included redis input plugin that will blpop a list to get events. I use the formatter from the logstash module to format the events and then write them into redis as json. Logstash picksthem up from redis and indexes them into its embedded elastic search instance. You can then connect to the embedded kibana web server to view a dashboard with event information. It's pretty cool and doesn't take long to set up.

 

[Joseph L. Casale]

Maybe check out logstash (http://logstash.net/).

That looks pretty slick, I am constrained to using something provided by the packaged modules
in this scenario.

I think I have it pretty close except for the fact that the LogRecordStreamHandler from the cookbook
excepts when the sending process ends, aside from the obvious, I am sure there is a more robust
performance oriented approach to this. I've yet to work with the SocketServer module so I am lacking
any experience.

Thanks for the pointer,
jlc