Twisted 100% event driven or hybrid?

[jacopo]

I am diving into Twisted and Perspective Broker (PB) in particular. I
am designing a system having several models running on different
machines, they need to be recalculated periodically, I have to collect
the results, process them and start again from the beginning.

It is not clear to me if I can blend some event driven programming
with a more traditional one where the flow would be deterministic.
In my case I have to iterate on a list of models and send the request
of recalculation to the specific machine where the model resides. I
don’t want to wait for each single result but I want to sent all the
requests in one go. In this phase I am happy to have an event driven
framework with callbacks. Then I have to stop and wait for all the
results to be ready, I collect and process them. From now on I don’t
need a the system to be event drive any more, the processing should
occur only on the master machine, following a deterministic flow.
As soon as finished I am ready to start again to resubmit the models
for recalculation and so on. This should go on forever.

Is it possible to have an hybrid system like this? If I call
reactor.spot() at the certain point of the execution where does the
execution continue from? Would this be a good design or in general is
better to keep a 100% event drive system even if I don’t actually need
to handle asynchronicity for big chunks of the code.

I would appreciate any suggestion and for the time being I recommend
this doc, one of the clearest i have read so far.
http://www.artima.com/weblogs/viewpost.jsp?thread=230001
Grazie,
Jacopo

 

[exarkun]

I am diving into Twisted and Perspective Broker (PB) in particular. I
am designing a system having several models running on different
machines, they need to be recalculated periodically, I have to collect
the results, process them and start again from the beginning.

It is not clear to me if I can blend some event driven programming
with a more traditional one where the flow would be deterministic.
In my case I have to iterate on a list of models and send the request
of recalculation to the specific machine where the model resides. I
don 19t want to wait for each single result but I want to sent all the
requests in one go. In this phase I am happy to have an event driven
framework with callbacks. Then I have to stop and wait for all the
results to be ready, I collect and process them. From now on I don 19t
need a the system to be event drive any more, the processing should
occur only on the master machine, following a deterministic flow.
As soon as finished I am ready to start again to resubmit the models
for recalculation and so on. This should go on forever.

Is it possible to have an hybrid system like this? If I call
reactor.spot() at the certain point of the execution where does the
execution continue from? Would this be a good design or in general is
better to keep a 100% event drive system even if I don 19t actually need
to handle asynchronicity for big chunks of the code.

If you're happy to block event processing, then there's no reason you
can't do just that - once you have your results, start processing them
in a blocking manner. Twisted will not service events while you're
doing this, but as long as you're happy with that, it doesn't really
matter to Twisted. You might not be as happy with this later on if your
requirements change, but you can always worry about that later.

In particular, though, there's no reason or need to call reactor.stop()
in order to "switch" to your non-event driven code. Wherever you were
thinking of putting that call, just put your non-event driven code there
instead.

Jean-Paul

 

[Mike C. Fletcher]

On 05:55 am, (e-mail address removed) wrote: ....

Jean-Paul is obviously far more authoritative on the "twisted way" than
I am, so if he says you can just run your synchronous operation in-situ,
that's probably the way to go, but IIRC there's a
reactor.deferToThread() function which can run your synchronous code
"off to the side", while allowing the twisted code to continue to
process incoming operations. Thus you'd do something like:

def process( dataset ):
dl = [ remote_call( x ) for x in dataset]
dl = defer.DeferredList( dl )
def on_all_results( results ):
reactor.deferToThread( sync_process, (results,)).addCallback(
process )
return dl.addCallback( on_all_results )

(I'm typing all of that from the distance of a few years of memory
decay, so take it as "loosely this, with the proper function names and
the like"). Threads aren't really part of the "twisted way" in my
understanding, but they can be used if necessary AFAIK, and they will
let your application remain responsive to network events during the
processing.

HTH,
Mike

--
________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://www.vrplumber.com
http://blog.vrplumber.com

 

[exarkun]

On 05:55 am, (e-mail address removed) wrote: ...

Jean-Paul is obviously far more authoritative on the "twisted way" than
I am, so if he says you can just run your synchronous operation in-
situ,
that's probably the way to go, but IIRC there's a
reactor.deferToThread() function which can run your synchronous code
"off to the side", while allowing the twisted code to continue to
process incoming operations. Thus you'd do something like:

def process( dataset ):
dl = [ remote_call( x ) for x in dataset]
dl = defer.DeferredList( dl )
def on_all_results( results ):
reactor.deferToThread( sync_process, (results,)).addCallback(
process )
return dl.addCallback( on_all_results )

(I'm typing all of that from the distance of a few years of memory
decay, so take it as "loosely this, with the proper function names and
the like"). Threads aren't really part of the "twisted way" in my
understanding, but they can be used if necessary AFAIK, and they will
let your application remain responsive to network events during the
processing.

Yep, you're correct here Mike (except it's
`twisted.internet.defer.deferToThread` rather than
`twisted.internet.reactor.deferToThread`). If it is safe to call
`sync_process` in a thread, then this may be a good approach as well,
and it will free up the reactor to continue to respond to events
(assuming `sync_process` plays nicely - ie, is written in Python or is
an extension that releases the GIL, of course).

In my post, I was trying to highlight the idea that there's not really
anything special going on in a Twisted program. You can choose to block
if you wish, if the consequences (events go unserviced for a while) are
acceptable to you.

Jean-Paul