Modification to asyncore.py to support threaded event loop

[Anand Pillai]

This is for folks who are familiar with asynchronous event handling in
Python using the asyncore module.

If you have ever used the asyncore module, you will realize that it's
event loop does not have a programmable exit condition. It keeps
looping till the channels in its socket map (a dictionary) are closed
and don't have any pending reads/writes.

If you are using Python threads in your application, by using either
the threading or thread modules, adding asyncore capability on top of
it is a bit tricky. The moment you call asyncore.loop after
initializing your asyncore based client/server, it takes over the main
thread. Hence, if you want to mix threaded classes (derived from
threading.Thread) and asyncore in one program, the only recourse is to
create and initialize all your thread objects first and call
asyncore.loop as the last function in your main thread.

However, with a small hack it is possible to get asyncore event loop
running in its own thread, as shown below.

The following is a mixin class, which derives from threading.Thread
and also overrides asyncore.loop in its 'run()' method.

-==========================================================================-
import threading

class AsyncoreThread(threading.Thread):
""" Asyncore thread class """

def __init__(self, timeout=30.0, use_poll=0,map=None):
self.flag=True
self.timeout=30.0
self.use_poll=use_poll
self.map=map
threading.Thread.__init__(self, None, None, 'asyncore thread')

def run(self):

self.loop()

def loop(self):

# print 'In AsyncoreThread:loop...'
if self.map is None:
self.map = asyncore.socket_map

if self.use_poll:
if hasattr(select, 'poll'):
poll_fun = asyncore.poll3
else:
poll_fun = asyncore.poll2
else:
# print 'Using asyncore.poll...'
poll_fun=asyncore.poll

while self.map and self.flag:
poll_fun(self.timeout,self.map)

def end(self):
print 'Ending asyncore loop...'
self.flag=False
self.map=None

-============================================================================-
Now you can initialize your asyncore based clients/servers and instead
of calling 'asyncore.loop', do it as follows,

t=AsyncoreThread(timeout=mytimeout,...)
t.start()

Since the 'run()' method of this class calls 'loop()' which is exactly
the same as asyncore.loop, only that it gets the asyncore event
handler running in a separate thread, without monopolizing the main
thread of your Python app.

I have used this model successfully in my program, HarvestMan. I wrote
to Sam Rushing, author of asyncore module and here is what he said.

<QUOTE>
"I would recommend separating out your event loop and sending it to
the
Python folks for possible inclusion in the asyncore module. I had
always intended the loop() function in asyncore.py to be a 'field
replaceable item', and in fact have over the years made many different
versions of it. Though this isn't really talked about in asyncore.py,
it's clear if you look at Medusa, which is where the module came
from."
</QUOTE>

Perhaps this is worth a PEP for inclusion in Python 2.4?

Thanks & Regards

-Anand

 

[Josiah Carlson]

If you have ever used the asyncore module, you will realize that it's
event loop does not have a programmable exit condition. It keeps
looping till the channels in its socket map (a dictionary) are closed
and don't have any pending reads/writes.

With the asyncore that will be included with Python 2.4, loop() has a
'count' argument. Your 'programmable exit condition' is simple...

while not exit_condition_satisfied:
asyncore.loop(count=1)

If you are using Python threads in your application, by using either
the threading or thread modules, adding asyncore capability on top of
it is a bit tricky. The moment you call asyncore.loop after
initializing your asyncore based client/server, it takes over the main
thread. Hence, if you want to mix threaded classes (derived from
threading.Thread) and asyncore in one program, the only recourse is to
create and initialize all your thread objects first and call
asyncore.loop as the last function in your main thread.

However, with a small hack it is possible to get asyncore event loop
running in its own thread, as shown below.

Here's an easier way to get asyncore running in its own thread, with
nearly all the functionality you mention, without having to subclass
Thread.

def asyncore_loop(exit_condition, timeout=30.0,
use_poll=False, map=None):
if map is None:
map = asyncore.map
while map and not exit_condition:
asyncore.loop(timeout=30.0, use_poll=False, map=map, count=1)

exit_condition = []

threading.Thread(target=asyncore_loop, args=(exit_condition,)).start()

#to kill the thread...
exit_condition.append(None)

The following is a mixin class, which derives from threading.Thread
and also overrides asyncore.loop in its 'run()' method.

[snip unnecessary code]

Now you can initialize your asyncore based clients/servers and instead
of calling 'asyncore.loop', do it as follows,

See the snippet I provide above.

Perhaps this is worth a PEP for inclusion in Python 2.4?

Module and/or functionality additions do not require a PEP (from what I
understand). Changing the behavior of the language, adding a builtin,
etc., that requires a PEP.

It is not going to make it into Python 2.4. How would I know this?
Because "no new functionality after the beta". We're on beta 2, and
even small functionality additions that were pre-approved (but not
implemented) are not making it into 2.4.

What about for Python 2.5? I doubt it would be added there either. The
functionality is very simple to implement if you know what you are doing,
and doesn't require copying the internals of asyncore.loop to make it
happen.

- Josiah

 

[Anand Pillai]

I tried your code and this is the result...


Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.3/threading.py", line 436, in __bootstrap
self.run()
File "/usr/lib/python2.3/threading.py", line 416, in run
self.__target(*self.__args, **self.__kwargs)
File "urlserver.py", line 205, in asyncore_loop
map = asyncore.map
AttributeError: 'module' object has no attribute 'map'

asyncore does not have an attribute named 'map', only socket_map.
'map' is an argument to ascynore.loop function, again not its
attribute.

Did you try this code out before posting it here?

Also, asyncore.loop has a while loop inside it, and you are adding
another while loop on top. Clearly your exit condition, if it works is
going to only exit the outer while loop, not the inner one. I dont
think it is going to work.


-Anand

If you have ever used the asyncore module, you will realize that it's
event loop does not have a programmable exit condition. It keeps
looping till the channels in its socket map (a dictionary) are closed
and don't have any pending reads/writes.

With the asyncore that will be included with Python 2.4, loop() has a
'count' argument. Your 'programmable exit condition' is simple...

while not exit_condition_satisfied:
asyncore.loop(count=1)

If you are using Python threads in your application, by using either
the threading or thread modules, adding asyncore capability on top of
it is a bit tricky. The moment you call asyncore.loop after
initializing your asyncore based client/server, it takes over the main
thread. Hence, if you want to mix threaded classes (derived from
threading.Thread) and asyncore in one program, the only recourse is to
create and initialize all your thread objects first and call
asyncore.loop as the last function in your main thread.

However, with a small hack it is possible to get asyncore event loop
running in its own thread, as shown below.

Here's an easier way to get asyncore running in its own thread, with
nearly all the functionality you mention, without having to subclass
Thread.

def asyncore_loop(exit_condition, timeout=30.0,
use_poll=False, map=None):
if map is None:
map = asyncore.map
while map and not exit_condition:
asyncore.loop(timeout=30.0, use_poll=False, map=map, count=1)

exit_condition = []

threading.Thread(target=asyncore_loop, args=(exit_condition,)).start()

#to kill the thread...
exit_condition.append(None)

The following is a mixin class, which derives from threading.Thread
and also overrides asyncore.loop in its 'run()' method.

[snip unnecessary code]

Now you can initialize your asyncore based clients/servers and instead
of calling 'asyncore.loop', do it as follows,

See the snippet I provide above.

Perhaps this is worth a PEP for inclusion in Python 2.4?

Module and/or functionality additions do not require a PEP (from what I
understand). Changing the behavior of the language, adding a builtin,
etc., that requires a PEP.

It is not going to make it into Python 2.4. How would I know this?
Because "no new functionality after the beta". We're on beta 2, and
even small functionality additions that were pre-approved (but not
implemented) are not making it into 2.4.

What about for Python 2.5? I doubt it would be added there either. The
functionality is very simple to implement if you know what you are doing,
and doesn't require copying the internals of asyncore.loop to make it
happen.

- Josiah

 

[Josiah Carlson]

I tried your code and this is the result...

[snip exception]

asyncore does not have an attribute named 'map', only socket_map.
'map' is an argument to ascynore.loop function, again not its
attribute.

socket_map then, whatever.

Did you try this code out before posting it here?

No, I typed it out in my email client. But my statement still holds,
the functionality is trivial to implement if you know what you are doing,
and shouldn't be in the asyncore module.

Also, asyncore.loop has a while loop inside it, and you are adding
another while loop on top. Clearly your exit condition, if it works is
going to only exit the outer while loop, not the inner one. I dont
think it is going to work.

If you check the source code for asyncore.py in Python 2.4 (available
from Python CVS), you will discover that you can provide an optional
'count' argument to asyncore.loop. You'll notice that I both mention
it:

and use it:

In this case, the 'inner loop' only runs for one pass. Certainly in
Python <2.4, it would run without stopping (unless the socket map became
empty), but I was referencing 2.4, not 2.3 or prior.


- Josiah